Transforming Data | CloverETL - data integration software and open source ETL tool

CloverETL Overview

For more details and videos take a look at the Quick Start Guide →

Designer UI

Anatomy of a Graph

Read & Write Data

Transform & Modify

Merge & Join

Other Actions

Clover Script

Performance


	Reformat : this allows incoming data to be mapped to an output structure. Data can be transformed as it is mapped also.			Dedup : removes duplicates from a data stream. Uses various matching algorithms.
	Aggregate : this allows incoming data to be aggregated as with a SQL Group By command.			Filter : filter a data stream to "true" and "false" output ports or multiple ports based on user defined conditions.
	Sort : sorts data on one or more fields in a data stream.			Partitioner : splits your data stream into several output streams based on a user defined condition.

Transforming Data

Clover is outfitted with a large number of components that let you modify and route your data as it flows through your data transformation.

As well as these powerful components you can also use CTL, Clover's own scripting language that let's you perform hand crafted operations on each record at any point in your data transformation.

About Transformation Components

A graph is built by adding, configuring and chaining any number of components together. As data passes between the components you may need to alter the data at various points within the graph. Some transformation components allow such alterations to be performed on a record by record basis (Reformat, Filter, Partitioner). Others are configured to perform bulk operations on the entire data set that passes through it (Aggregate, XSLT, Dedup).

Commonly Used Transform Components

You will find that the Reformat command is a constant companion. This allows you to do do any type of field mapping of calculation on each record that passes through the component. Filtering and Sorting are also very commonly used actions and Clover provides highly configurable components. You may also find you used the Aggregation components regularly to perform any type of aggregtion you might perform with SQL (GROUP BY and aggregation functions) but you can also perform advanced aggreations that SQL cannot.

	XSLT : performs an XSLT transformation.			Copy : takes an incoming data stream and replicates into any number of identical output streams.
	Concatenate : concatenates identical data from each input port			Group Sort : highly optimized sort algorithm used when adding new sort fields to already sorted data.
	Intersection : takes two input streams A and B and outputs three streams based on one or more key fields (only in A, only in B, in both A and B)			Normalizer : breaks complex data from a single record into multiple output records of different formats
	Denormalize : generates one output record from multilple input records			Merge : takes sorted input data from multilple ports and merges based on key fields
	Gather : takes data from input ports as it is presented and merges into a single output

CTL - Scripting

Clover components will take care of the majority of your needs but every now and then you may have an application that requires something out of the ordinary. CTL scripting allows you to write simple or complex scripts that operate on each data record as it pass through the component with CTL added. Learn more →