Anatomy of a Clover Graph

From Community Edition to Cluster Edition, data transformations are configured using Graphs. These graphs are built using the Clover Designer. This section shows how a simple graph is put together.

Components

Components are dragged into the graph from the toolbox on the right hand side of the designer. There are components for reading, writing, transforming, modifying, joining and other system and Java level commands. Clover Designer →

Reading Data

All graphs need to get data from somewhere. A graph will always have at least one and often several Readers that can get data from a file system, http, ftp, web service, database, LDAP or other sources. Clover reader components are highly configurable and can read data in just about any format you throw at them. Readers & Writers →

Ports

Once data has been read in, it is passed to an Output Port. In a graph, an Edge connects the output port of one component to the Input Port of another component. Readers typically only use one Output Port but some Tranformation Components can filter or partition the data to multiple Output Ports based on conditions.

Edges and Metadata

An Edge, the line that connects component ports, carries data between components. The Edge has a so-called Metadata structure associated with it. Metadata is a simple record structure that can be manually defined or can be automatically generated from a database table, XSD data,  XLS or flat file.

Transform Components

Once the data has been read in, it often needs to be transformed by special Transformation Components. These can calculate new field values, sort, filter, aggregate and perform many other operations. Transform Components →

Joining

You will often pull data from two or more sources and you then may need to join these data streams based on common key fields. This can be identical or similar to a SQL JOIN statement. Clover offers a range of Joiners that perform different types of tunable joins. Join Components →

Writing Data

A graph will end with your transformed data being written to one or more output files or databases. Clover offers a wide range of components that allow generic or native drivers to be used for fast output.

Debugging Graphs

Another great feature of Clover ETL software is the way that you can debug a graph. You can enable debugging for any of the Edge lines connecting components. When you run the Graph, you can see how many records are processed and passed along the Edge. You can then right-click the Edge and view the data that was passed along the line. This is a great way of monitoring and testing your graph in the design phase.

Other Graph features

Clover lets you perform fast Lookups by setting up high-speed Lookup files and even ultra fast memory based Lookups. You can also pass parameters into your graphs by setting parameters manually. The Enterprise and Cluster Editions take this a step further and allow parameters to be passed in via an http call. There are also many other advanced features that let you fine tune graph operations.