Automatic Numbering Sequences
CloverETL can generate number sequences that can be used for generating automatically incremented number sequences. Although simple, It has some very useful capabilities
|•||Automatically number record ids for each output record|
|•||Sequence numbers persist across graph executions. Clover stores the sequence number in an external file so that subsequent runs carry on where the number left off. The best way of visualizing this is to watch the short video below.|
|•||Sequence numbers are thread-safe. This means that if you are running graphs in parallel (requires server version) then you do not need to worry about the numbering getting out of sync.|
|•||Can be tuned (using the Cache parameter) to allocate large numbers of sequences in one go.|
One thing that is less than obvious is the use of the Cache property. If this is set to 1, then each time a Sequence number is required, it will have to go to the sequence file and generate the number before returning. For large amounts of data this can slow things down. If you set it to 1000, then it will only need to access the file when it runs out of numbers. If you are running a graph in parallel then the need for a reasonable caching value becomes more important. One thing to note, though, is that when a graph has finished running, any unused sequence numbers will remain allocated but not used (nor will they ever get used again).