CloverETL 4.0

Helping Developers Empower Business

Watch CloverETL 4.0 Overview Video

CloverETL 4.0 introduces subgraphs, a powerful way of creating user-defined components that let you hide complexity and design data transformations on a much higher level, using business components rather than low-level data operations. Also, the new features that support subgraphs are very useful on their own – automatic metadata propagation reduces the need to manage metadata in transformations, while the execution view panel gives you easier access to debug information for faster development.

Released Nov 12, 2014
  • Production
Recent Releases
4.8.0 Dec 13, 2017
4.7.1 Nov 28, 2017
4.7.0 Oct 10, 2017
4.6.1 July 19, 2017
4.5.0 Mar 14, 2017
4.4.1 Feb 14, 2016
4.3.1 Nov 10, 2016

See all releases

Try it yourself

Download Trial




  • Designer
  • Server
  • Major New Feature

Subgraphs

Reduce complexity, create business-level components, share your expertise. Create new components without programming. Wrap complex functionality into a simple interface.

Create new data sources and targets

Embed connections, metadata and other properties into subgraphs and share them with others.

Focus on business logic

Create a library of your business logic components, rather than low-level data processing functions.

Reduce complexity

Wrap logical parts of large transformations together into steps that you can easily drill into.

Share your expertise

Solve hard problems and share the solution with others. Everyone can contribute.



  • New Feature

Automatic Metadata Propagation

Metadata are now automatically
assigned to edges when possible.

Save time and effort

No need to assign metadata to every edge – they are determined based on the component and its surroundings.

Reuse transformations

Changes automatically propagate through complex transformations, making them easier to reuse.

Distribute with subgraphs

Metadata embedded into subgraphs are readily available in upstream transformations, with no linking needed.



  • New Feature

Execution View Panel

View execution history and visually access debugging data for subgraphs and jobflows.

Real-time tracking

Watch data in real time as it flows through your transformation or jobflow.

Access stored debug data

Just double-click any layer of subgraphs or nested jobflow, and inspect collected debug data.

Troubleshoot finished runs

Connect to jobs executed by the Server and visually inspect what data flowed through successfully, in order to find potential problem areas.



  • New Feature

Tableau TDE Support

Create Tableau Extract files (TDE), directly enabling rapid data preparation and development.




Detailed list of improvements and fixes

ETL Developers (This item is most useful for developers; it either brings new functionality for transforming data or brings optimizations)

Administrators (This item is an improvement or feature that will help setup, install, administer and manage the application)

Support (This item helps staff supporting the production operation to identify potential problems or avoid such)

4.0.2

4.0.1

4.0.0 (long list)

Subgraphs [ See All | Hide All ]

Subgraph support in Server runtime New CLO-218 4.0.0-M1
Server supports the execution of graphs containing subgraphs (graphs that can be used similarly to components).
Editing of subgraphs in Designer New CLO-218 4.0.0-M1
You can create and develop new subgraphs in Designer as .sgrf files.
Wrapping components into new subgraphs New CLO-2897 4.0.0-M1
You can select a bunch of components and easily create a new subgraph based on them.
Automatic metadata propagation New CLO-1475 4.0.0-M1
Many components now automatically propagate the correct metadata. This minimizes the need to manually select metadata. Subgraphs will also be able to provide you with correct metadata for their outputs or inputs.
Subgraph functionality extended to standalone Designer New CLO-4393 4.0.0-M2
Subgraphs can now be developed, used and executed in Designer - with the new Runtime, it's not necessary to use Server for subgraph execution.
Improved subgraph editor Improvement CLO-3630 4.0.0-M2
Editor of subgraphs was improved for ease of use and clarity.
Configure subgraphs as components New CLO-3548 4.0.0-M2
When using subgraphs, you can configure them as ordinary components - they have properties with configuration dialogs. Internally, the subgraphs utilize the extended model of graph parameters to define their configuration.
Explicit metadata propagation New CLO-3941 4.0.0-PROD
You can manually define the path of metadata propagation. This feature can be used to override the default metadata propagation behavior and to implement more complex use cases.
Open subgraphs on Ctrl + mouse click New CLO-4602 4.0.0-PROD
Use Ctrl + mouse click on a Subgraph component to quickly open the subgraph. This action works also for ExecuteGraph, ExecuteJobflow and RunGraph components and allows you to quickly navigate the hierarchy of executed jobs.
Updated CloverETL perspective Improvement CLO-4002 4.0.0-PROD
The CloverETL perspective layout is tweaked so that you can see all important runtime information at the same time.

Server Administration & UX [ See All | Hide All ]

Optimized Execution History tree for large numbers of jobs (10k+) Improvement Server CLO-2418 4.0.0-M1
Display of execution hierarchy is optimized for a huge number of executed jobs by presenting them in an easily-navigatable tree structure.
Added filtering of Executions History by sandbox Improvement Server CLO-3005 4.0.0-M1
Filter contents of Executions History also by sandbox.
Always show and expand nested jobs when searching for job by its run ID Improvement Server CLO-2550 4.0.0-M1
Automatically expand and show nested jobs when searching by run ID. This is useful for searching in large execution hierarchies.
Use syntax highlighting in Server Console Improvement Server CLO-2302 4.0.0-M1
Server Console uses syntax highlighting when displaying source files, XML files, etc.
Display warning on startup if Java memory is low Improvement Server CLO-881 4.0.0-M1
Server will display a warning if the configured memory available to Java is low.

Features & Improvements [ See All | Hide All ]

Added graph Cleanup action to remove unused elements New CLO-2963 4.0.0-M1
The Cleanup action will find elements of graph that are not used in it, e.g. old forgotten metadata, connections etc. This is a useful tool for keeping your graphs clear and tidy.
Added filter to Palette for searching for components New CLO-2803 4.0.0-M1
Find components in Palette by searching by their name, description, etc.
Added MongoDB components to Community Designer Improvement CLO-2822 4.0.0-M1
Added JSONExtract component to Community Designer Improvement CLO-2822 4.0.0-M1
Added support for SSL client certificate in HTTPConnector Improvement CLO-2598 4.0.0-M1
Added ability to export debug data to a CSV file Improvement CLO-1938 4.0.0-M1
Renamed EmailSender attribute "Use TLS" to "Use STARTTLS" Improvement CLO-2829 4.0.0-M1
Simplified New ETL Graph wizard Improvement CLO-1670 4.0.0-M1
Added escapeURL function for ordinary strings, not just complete URLs Improvement CLO-398 4.0.0-M1
Removed Marquee tool from palette Improvement CLO-3181 4.0.0-M1
Added filename to UniversalDataReader log after error Improvement CLO-1752 4.0.0-M1
Launch Service parameter names are derived from dictionary Improvement CLO-3912 4.0.0-M2
Automatically set parameter name based on selected dictionary entry name.
Added a new toolbar in the Edit Component dialog with common actions Improvement CLO-3754 4.0.0-M2
Actions for working with graph parameters, custom component properties, and other are now accessible in the Edit Component dialog via a new toolbar.
Jobs can have an execution label defined Improvement CLO-3604 4.0.0-M2
You can dynamically set an execution label for each job to better distinguish it. The label is also set automatically from the executing component name - e.g. from the name of Subgraph component.
Improved edge routing algorithm Improvement CLO-3429 4.0.0-M2
Improved the user interface of AddressDoctor component Improvement CLO-3699 4.0.0-M2
The AddressDoctor mapping dialog is simplified and easier to use.
Improved performance of CloverDataReader (removed unnecessary deserialization) Improvement CLO-2655 4.0.0-M2
Added new allocation type (run on all nodes) Improvement Cluster CLO-2510 4.0.0-M2
Simplifies scaling in Cluster - this new allocation type automatically uses newly added cluster nodes.
Highlight unused objects in Outline Improvement CLO-3704 4.0.0-M2
Outline shows objects in grey if they are not used in the graph. This makes it easy to find old unused metadata, connections, etc.
Better defaults for DataGenerator Improvement CLO-3214 4.0.0-M2
DataGenerator can be used without configuration. It generates one empty record by default.
Sleep component without connected ports Improvement CLO-4164 4.0.0-M2
Sleep component now also works without any connected port.
Better metadata editing using keyboard Improvement CLO-3888 4.0.0-M2
TableauWriter component New CLO-4096 4.0.0-PROD
A new TableauWriter component that can send data to Tableau for business analytics visualization.
Open a running or finished job in Designer New CLO-4564 4.0.0-PROD
Open a currently running or already finished Server job (graph, jobflow) in Designer. Designer will show the graph, tracking information and log updated live, the same as if the job was started from Designer. This can be used to monitor the state of a running job, or for post-mortem analysis of a failed job.
Updated CloverDataReader and CloverDataWriter components Improvement CLO-5060 4.0.0-PROD
CloverDataReader and CloverDataWriter components were updated to be consistent with other readers and writers (e.g. support partitioned writing, autofilling etc). Additionally, their compression was improved.
Loop support in ETL graph Improvement CLO-3939 4.0.0-PROD
Loop component can be used also in ordinary ETL graphs, not just jobflows. You can use it for example to orchestrate REST calls to web services that return data in chunks (or pages).
Loop component has new attribute "Token logging" Improvement CLO-4545 4.0.0-PROD
Token logging attribute controls the log level on which the Loop component reports token information.
SpreadSheetWriter can write into an existing sheet in streaming mode Improvement CLO-4028 4.0.0-PROD
Writing into a specific sheet is supported in streaming mode, so the other sheets won't be removed. Please note that the existing sheets must first be loaded into memory, so it's not possible to write huge data into different sheets.
Added and improved CTL functions Improvement CLO-3703 CLO-4384 CLO-4145 CLO-3838 4.0.0-PROD
Added or improved CTL functions for getting sandbox prefixed path, bit operations, initialization of maps and access to custom component properites.
Cleanup of graph context menu Improvement CLO-3158 4.0.0-PROD
Context menu in the graph editor is simplified to contain only the relevant items.
Add and remove custom component properties Improvement CLO-4170 4.0.0-PROD
Component configuration dialog allows you to easily add and remove custom properties of components. These properties can be used from CTL and Java transformations as additional means of configuration.
Link to component from console New CLO-4120 4.0.0-PROD
IDs of components are highlighted in the graph log. Clicking on the ID shows and selects the component in the graph.
Select parameter for a component attribute in a dialog Improvement CLO-3801 4.0.0-PROD
You can use a dialog to select a parameter as value of a component attribute.
Configurable JCE provider for encrypted Server configuration Improvement CLO-4764 4.0.0-PROD
3rd party providers of encryption algorithm for Server configuration can be selected, in a similar way as secure parameters. This allows you to use stronger encryption than provided by default in Java, e.g. BouncyCastle.
Server cron scheduling supports seconds Improvement CLO-4523 4.0.0-PROD
Cluster node syncing is now more robust for high latency networks Improvement CLO-4529 4.0.0-PROD
Improved asynchronous messaging between nodes to accommodate for higher latency networks
Possibility for the Execution History of Server Console to show only jobs that the user has access to in sandbox Improvement CLO-5317 4.0.0-PROD
Tasks History in Server Console filters on Status column Improvement CLO-4770 4.0.0-PROD

Compatibility [ See All | Hide All ]

CloverETL requires Java 7 Compatibility CLO-1055 4.0.0-M1
Upgraded versions of supported application containers of Server Compatibility CLO-3229 4.0.0-M1
Upgraded libraries that were in very old versions Compatibility CLO-2874 4.0.0-M1
Added support of MySQL 5.6 as Server configuration database Compatibility CLO-2718 4.0.0-M1
Sybase is no longer supported as Server configuration database Compatibility CLO-3188 4.0.0-M1
Deprecated components are now hidden Compatibility CLO-2974 4.0.0-M1
Removed old FactTableLoader wizard Compatibility CLO-2521 4.0.0-M1
CTL: fixed substring() inconsistency with 2 and 3 parameters Compatibility Fix CLO-3201 4.0.0-M1
CTL: split() now returns empty list if input is null Compatibility Fix CLO-3097 4.0.0-M1
CTL: indexOf() now accepts null values Compatibility Fix CLO-3094 4.0.0-M1
Fixed EmailReader propagating CC to all succeeding records Compatibility Fix CLO-2731 4.0.0-M1
Removed CTL compatibility check wizard Compatibility CLO-2894 4.0.0-M1
CloverETL Server supports Tomcat 7 Compatibility CLO-3734 4.0.0-M2
CloverETL Designer is bundled with Eclipse 4.3 Compatibility CLO-3752 4.0.0-M2
New Runtime for Designer Compatibility CLO-4392 4.0.0-M2
The new Runtime instance in Designer is shared among all executed graphs in one session - impacts configuration of JVM parameters, memory settings, etc.
XLSDataReader and XLSDataWriter are deprecated Compatibility CLO-2823 4.0.0-M2
XMLExtract: explicit mapping has priority over implicit automap by name Compatibility CLO-2029 4.0.0-M2
Accessing non-existent tracking field in jobflow output mapping fails Compatibility CLO-1872 4.0.0-M2
Subgraph debug input and output components are identified by position in graph Compatibility CLO-4470 4.0.0-M2
Incompatibility with 4.0.0-M1 subgraphs - debug input/output components are identified by position instead of the way they are connected. Designer automatically updates and fixes the subgraphs.
Additional graph parameters are set by Runtime (such as RUN_ID). Compatibility CLO-5129 4.0.0-PROD
Graph parameters with additional information about the execution are automatically set by the Runtime. For example the RUN_ID parameter.
Removed special ZIP support in CloverDataReader and CloverDataWriter Compatibility CLO-4447 4.0.0-PROD
CloverDataReader and CloverDataWriter had support for a special ZIP format. This is replaced by an improved compression format.
Database connection shared by multiple components in graph does not perform commit if the graph fails. Compatibility CLO-4878 4.0.0-PROD
If the connection instance is shared by multiple components (i.e. it's not in threadsafe mode) and the components are configured not to perform commit themselves, then a failing graph will not perform commit. This allows you to define a big transaction over the whole graph - if the graph fails, then it will perform rollback.
Fixed DeleteFiles recursive delete deleting the whole sandbox if fileURL was empty Compatibility CLO-5114 4.0.0-PROD
If DeleteFiles is configured for recursive delete and receives empty fileURL (as an attribute or from input), it deletes the contents of its sandbox.
Removed "Export metadata to XSD" wizard Compatibility CLO-2029 4.0.0-PROD
Removed "Export graphs" wizard Compatibility CLO-1872 4.0.0-PROD

New Designer Runtime [ See All | Hide All ]

New Runtime layer in Designer New CLO-4392 4.0.0-M2
Graphs executed within Designer are executed by a new CloverETL Runtime. The runtime builds upon CloverETL Engine and provides support for subgraphs, better execution tracking and more.
Improved Execution user interface New CLO-4466 4.0.0-M2
A new Execution View was introduced that shows the current execution status of a running graph and its sub jobs (e.g. subgraphs). You can access details such as the graph file, log, and tracking information by double clicking on a job.

Extended graph parameters [ See All | Hide All ]

Public graph parameters Improvement CLO-4396 4.0.0-M2
Selected graph parameters can be marked as public. For subgraphs, these parameters are presented as properties of the parent subgraph component. This allows for easy configuration of a subgraph, making it feel like a component.
Graph parameters have a type Improvement CLO-3555 4.0.0-M2
A new 'type' property of a graph parameter is used to choose correspoding user interface/dialog for editing the parameter - e.g. URL dialog, key selection etc.
CTL2 as parameter value Improvement CLO-3600 4.0.0-M2
CTL2 can be used to dynamically compute the value of graph parameters.
Editor for parameter files New CLO-2942 4.0.0-M2
Graph parameters editor can now be used to edit external parameter files.
Content assist for server parameters Improvement CLO-1968 4.0.0-M2
Content assist in CTL also proposes runtime parameters injected by Server.

Fixes [ See All | Hide All ]

Fixed sandboxes.home not being accessible during initial deploy, and Server not starting after restart Fix CLO-3100 4.0.0-M2
Fixed asterisk in Samba URL causing unexpected behavior Fix CLO-4062 4.0.0-M2
Fixed AddressDoctor configuration dialogs missing some options Fix CLO-3377 4.0.0-M2
Fixed updates to properties not reflected unless confirmed by clicking somewhere else (on OSX) Fix Mac OS CLO-3070 4.0.0-M2
Fixed job type icon sometimes not being displayed in Server Console Fix CLO-3067 4.0.0-M2
Fixed SortWithinGroups losing records Fix CLO-4812 4.0.0-PROD
Fixed exception when unregistering JDBC driver on JBoss Fix CLO-4807 4.0.0-PROD
Fixed CTL2 performance issues in locale and timezone manipulation Fix CLO-4478 4.0.0-PROD
Fixed EOF as delimiter not working for fixed length metadata Fix CLO-4462 4.0.0-PROD
Fixed MongoDBWriter handling of null values Fix CLO-4428 4.0.0-PROD
Fixed MongoDBWriter batch mode error handling Fix CLO-5030 4.0.0-PROD
Fixed RunGraph component not supporting running jobs in separate JVM Fix CLO-4212 4.0.0-PROD
RunGraph can run ETL graphs in separate JVM. However it does not support subgraphs in the separate JVM, as it does not start the whole Runtime. This is a compatibility feature.
Fixed CTL error reporting issues in ExtFilter Fix CLO-1514 4.0.0-PROD
Fixed DB Lookup not accepting key longer than 10 digits Fix CLO-4571 4.0.0-PROD
Fixed space preceding a * in wildcard mapping not reporting error in Designer or Runtime Fix CLO-4532 4.0.0-PROD
Fixed FTP disconnect error not correctly reported Fix CLO-4404 4.0.0-PROD
Fixed Server bundle not starting if path to its installation contained the space character Fix CLO-4046 4.0.0-PROD

Designer Runtime improvements [ See All | Hide All ]

Secure paramers support in local projects Improvement CLO-4525 4.0.0-PROD
Secure parameters that encrypt their value can be used also in ordinary Designer projects, not only in Server projects. Configuration of the master password is in Designer's preferences.
Set engine properties Improvement CLO-4459 4.0.0-PROD
It is possible to set engine configuration properties (defaultProperties file) from the Runtime's preferences.

Known Issues [ See All | Hide All ]

Mac OS X Mavericks (10.9) might give you a warning about running unsigned software. Compatibility Mac OS 4.0.0-PROD
This is a known issue with all Eclipse-based software. However, you can still safely use the application. Please read this post at apple.com or this one
ESET Antivirus might cause ETL Runtime not to work properly (Unexpected EOF) 4.0.0-PROD
The issue seems to be caused by ESET's protocol filtering feature. To fix this, either disable protocol filtering (Setup > Enter Advanced Setup > Web and email / Protocol Filtering > uncheck "Enable application protocol and filtering") or add an exception (Setup > Enter Advanced Setup > Web and email / Protocol Filtering > Excluded Applications; add exception for "javaw.exe" used by Designer (e.g. "C:\Program Files\CloverETL Designer\jdk1.7.0_51_64\bin\javaw.exe")