CloverETL 4.2.0

CloverETL 4.2 improves (not only) team productivity and brings performance boost for parallel processing.

We have redesigned the internals of how Designer interacts with Server so that you can now put Server remote projects under version control. For Server Corporate, we're bringing “data partitioning” – a feature formerly available only in Cluster. It lets you run multiple parallel instances of a component at once – giving you more punch for operations that can utilize multiple CPU cores or high latency operations, such as API calls.



About this release: This is a production release and we recommend upgrading from older versions to this one. Please check Upgrade Instructions for details.

Released Jun 9, 2016
  • Production
Recent Important Releases
4.2.0 Jun 9, 2016
4.1.4 Jun 1, 2016
4.1.3 Apr 6, 2016
4.0.8 Dec 8, 2015
4.1.0 Sep 10, 2015

See all releases

Try it yourself

DOWNLOAD


New features


Version control (Git, SVN, ...) now works with Server projects

Designer now mirrors remote Server projects to your local disk, allowing you to easily put the local copy under version control.

With this you can manage remote Server projects under version control directly. You no longer need to keep a separate versioned local copy of the Server sandbox.

Version control operations such as pull, push, commit, merge, etc. work directly with the files in the local copy of the project, so you can use any standard version control tool, including built-in Eclipse plugins for Git and SVN. Designer then transparently synchronizes all changes to the Server sandbox.


Remote Server files available locally

As an effect of the previous feature, you can open and save files originating from a Server sandbox with any tool you need.

CloverETL - remote files locally

Convert between local and Server projects

You can easily deploy a project to Server from your workspace by using Convert To Server Project shortcut. Conversely, you can disconnect a Server project to keep just the local copy or connect and sync it with another Server sandbox.

CloverETL - convert to server project

Data Partitioning: Run components in parallel

Corporate Server and Cluster only

Run parts of a transformation (individual components or sequences of steps) in parallel, optimizing workflows for multiple processor cores on a single Server box.

We've made Cluster capabilities available to single Server (available in Corporate only). You can now run multiple instances of a component to increase throughput of throttled Web Service APIs, to parallelize CPU intensive operations that would otherwise run only in a single thread, or simply partition and process data on multiple disks to maximize I/O performance.

CloverETL data partitioning

Toggle details on edge

Corporate Server and Cluster only

Press D to cycle between levels of detail when running data partitioning on a single Server (Corporate) or mulitple nodes (Cluster). You can check numbers of records being processed by and transfered between individual workers.

CloverETL toggle details on edge for cluster and data partitioning

Source editor Syntax highlighting

The Source tab in the main editor now supports syntax highlighting.

CloverETL - source syntax hightlight

Amazon S3 auto-select endpoint for region

Amazon recommends using correspoding endpoint URLs when accessing S3 buckets hosted in specific regions. You can now select the Amazon region and CloverETL will automatically generate the right URL.

CloverETL - Amazon S3 region auto-select

Draw edge from input

You can now draw new edge from an input port to create a new upstream component. Just click an input port, drag to a blank area and then use the Add Component dialog to create the component.

CloverETL - create edge from input port

Upgrade Instructions for CloverETL 4.2

To help you with upgrading to this version, we've prepared a simple checklist for you:

Before You Upgrade

  • Be sure to check "Compatibility" notes for ALL intermediary releases. We mark all changes that can potentially alter the function of your existing transformations by "Compatibility" label to make it easier for you. You can safely ignore most of them as we try hard to keep as much backwards compatibility as we can. There's a comprehensive list of all releases that will help you get the information quickly.
  • Upgrade Designer and Server together. We always release Designer and Server together under one version. It's highly recommmend to upgrade Server and Designer at the same time. Although using different version of Designer to connect to Server might work, it is not generally supported.
  • There are no incremental patches. We don't release incremental patches. Every upgrade is actually a full installation that will update whatever necessary in your workspaces, sandboxes and databases automatically as needed.
  • Don't forget to backup. Although none of the upgrade steps requires it, we recommend you back up your work. The upgrade will keep all your transformations, jobflows and configurations safe. However, as a good word of advice, it never hurts to have a backup.

Designer Upgrade

  • Download the latest version by logging into your customer account. If you no longer have access there, no problem. Click here to recover your password or contact our support.
  • Install new version of Designer. You can install Designer over your existing installation. The process will automatically clean up the old version. Don't worry, you will NOT lose your workspaces, graphs and transformations. However, if you installed some additional plugins to Designer (Eclipse plugins) you might need to reinstall them, Eclipse should automatically help you do that. When you start the application, point it to your existing workspace directory. With some major releases we may notify you about upgrading the workspace to the latest version. In such cases you won't be able to use the workspace with previous versions. Thus, be sure to upgrade all Designers if you're sharing the workspace.
  • Activate the product on first start. The license key changes with every major version (i.e. from 4.2 to 4.3) and we automatically renew the keys for you. Just go to the download area again and copy/paste the license key from there. If you can't find the latest keys, you will need to renew your product maintenance.

Server Upgrade

  • Download the latest version by logging into your customer account. You'll find Server in the same list as Designer downloads. If you no longer have access there, no problem. Click here to recover your password or contact our support.
  • Plan for downtime. Upgrading the Server requires downtime, so plan your upgrades in advance. If you're running multiple environments, upgrade non-production installation first and run all your checks first.
  • Follow step-by-step Server Upgrade Guide. We've prepared detailed instructions on how to properly shut down Server and install a new one. Server will upgrade its database and sandboxes from any previous version automatically.
  • Activate the product on the login screen of Server Console. The license key changes with every major version (i.e. from 4.2 to 4.3) and we automatically renew the keys for you. Just go to the download area again and copy/paste the license key from there.





Detailed list of improvements and fixes in 4.2


Productivity / Version Control for Server Projects [ See All | Hide All ]

Placeholders for large files in Designer New Feature CLO-7899 4.2.0-M1
Large files (over 10MB by default) are not automatically synchronized to the local copy. You can download the files manually or change the threshold in Preferences > CloverETL
Resolving Conflicts between local and server copy New Feature CLO-7836 4.2.0-M1
When Designer synchronizes with Server, you can manually resolve conflicts between incoming and local changes.
Ignore files in server project synchronization New Feature CLO-7604 4.2.0-M1
Unify various CloverETL project natures Improvement CLO-7741 4.2.0-M1
Convert local project to Server project (and back) New Feature CLO-8290 4.2.0-M1
In Designer, you can now easily convert a local project to a Server one and vice versa. No need to create new sandbox and then export/copy the files.
Handling of connection outages Improvement CLO-7613 CLO-8207 4.2.0-M1
When working with with Server projects in Designer, connection problems do not affect the synchronization process. Designer will queue the changes until you're back online. Also, Designer reconnects automatically.
Graph header (.grf/.jbf XML files) is now not modified when not needed Improvement CLO-8192 4.2.0-M1
This caused inconveniences when someone resaved a graph without making any changes, yet it appeared as modified in version control, because Designer would change the header in the XML source.

Platform Support [ See All | Hide All ]

Hadoop integration now supports versions CDH 4.3, 5.4 Improvement CLO-7254 4.2.0-M1
Dropped support for Eclipse 3.X Improvement CLO-6003 4.2.0-M1
We no longer support installations of Designer plugin into 3.x Eclipse environments

Packaging / Versions [ See All | Hide All ]

Designer is now bundled with Java 8 Improvement CLO-8558 4.2.0-M1
Designer now comes with Java sources in the bundled JDK Improvement CLO-7042 4.2.0-M1
When developing Java code/transformations, you have out-of-the-box access to full Java sources
Server bundle upgraded to Tomcat 8 Improvement CLO-7764 4.2.0-M1
The Server bundle download is now built on top of Tomcat 8. We don't expect any major changes to setup and configuration

Engine Improvements [ See All | Hide All ]

Amazon S3 support: Replaced JetS3t with Amazon SDK Improvement CLO-7170 4.2.0-M1
S3 connectivity now uses the latest native Amazon SDK
Validator email address rule has new option "Allow addresses with no TLD". Formerly, this was a default behaviour, now you have to explicitly turn it on. Compatibility New Features CLO-6916 4.2.0-M1
FlatFileReader and FlatFileWriter aliases for UniversalDataReader and UniversalDataWriter New Feature CLO-8266 4.2.0-M1
Improved temp file management in Spreadsheet components Improvement CLO-7118 4.2.0-M1
JSONWriter and XMLWriter: New attribute "Write raw value" enables you to take raw snippets of JSON/XML and send them to its output without escaping it. Improvement CLO-6786 4.2.0-M1
New second output port for unmatched data in ExtMergeJoin, ExtHashJoin Improvement CLO-6588 4.2.0-M1
You can now capture unmatched records in ExtMergeJoin and ExtHashJoin
New metadata templates for port writing outputs in XMLWriter, UniversalDataWriter, CloverDataWriter, SpreadsheetDataWriter, StructuredDataWriter, JSONWriter Improvement CLO-4728 4.2.0-M1
Predefined metadata templates on writers, when you want to write to ouput port. Connect output edge to the writer, right the edge and select New Metadata From Template

Designer Improvements [ See All | Hide All ]

Syntax highlighting in graph source editor Improvement CLO-6465 4.2.0-M1
Improved searching for dependencies of graph elements Improvement CLO-5588 4.2.0-M1
More precise algorithm that detects and highlights unused metadata and other graph elements in Outline
Removed legacy options from CloverETL Preferences page Improvement CLO-7962 4.2.0-M1
iif() and lookup() CTL functions were missing in function list in Transform and Expression editors Improvement CLO-7533 4.2.0-M1
Draging & dropping a job into jobflow now names the newly created component based on the name of the job Improvement CLO-6972 4.2.0-M1
For example, if you take TruncateTables.grf and drop it into a jobflow, it will now create an ExecuteGraph component named "TruncateTables"
Drawing an edge from input port: You can now click an input port, draw a new edge and then pick the source component form the Add Component dialog New Feature CLO-6050 4.2.0-M1
S3: Added region selection option to Edit URL dialog Improvement CLO-8588 4.2.0
To achieve the best performance, it is recommended to access S3 buckets via corresponding regional endpoint URLs. We added region selection to the Edit URL dialog so that CloverETL will automatically generate correct URLs based on region of your S3 bucket.

Server Improvements [ See All | Hide All ]

Data partitioning (run component in parallel) in CloverETL Server Corporate New Feature CLO-8874 4.2.0
You can now set components or portions of a graph to run in multiple parallel instances on a singe Server Corporate machine. This was formerly only available when running Cluster.
Users can now pick JNDI names from JNDI tree whenever JNDI configuration option is present New Feature CLO-7160 CLO-7159 4.2.0-M1
New option to create a heap dump directly from Server Console New Feature CLO-7151 4.2.0-M1
Heap (and thread) dump can be downloaded in Configuration > System Info > Diagnostics.

Fixes [ See All | Hide All ]

Shift+Space to insert a subgraph should use ${SUBGRAPH_DIR} instead of sandbox:// Fix CLO-7683 4.2.0-M1
SimpleGather is blocked in fast propagate mode Fix CLO-7211 4.2.0-M1
Opening file though File menu is throwing a null pointer exception Fix CLO-7297 4.2.0-M1
SpreadSheetWriter: mapping by order does not work with hyperlinks Fix CLO-7104 4.2.0-M1
Transform Editor does not open at all when metadata missing Fix CLO-3549 4.2.0-M1

Compatibility [ See All | Hide All ]

S3 connectivity: Replaced JetS3t with Amazon SDK Compatibility Improvement CLO-7170 4.2.0-M1
Designer as eclipse plugin: Dropped support of Eclipse 3.x Compatibility CLO-6003 4.2.0-M1
XMLExtract: wrong attribute value is parsed Compatibility Fix CLO-7126 4.2.0-M1
DBJoin should propagate metadata Compatibility Improvement CLO-8477 4.2.0-M1
Rollup - preExecute() is never called Compatibility Fix CLO-7202 4.2.0-M1
SpreadSheetWriter: mapping by order does not work with hyperlinks Compatibility Fix CLO-7104 4.2.0-M1
Sequences used in COMPILED CTL no longer skip values Compatibility Fix CLO-5762 4.2.0-M1
Validator email address rule has new option "Allow addresses with no TLD". Formerly, this was a default behaviour, now you have to explicitly turn it on. Compatibility Fix CLO-6916 4.2.0-M1
CloverETL now supports IBM InfoSphere MDM version 11.5 Compatibility CLO-8790 4.2.0
IBM InfoSphere MDM API libraries bundled with CloverETL have been updated to 11.5 - this is currently the only supported version.

Under the hood [ See All | Hide All ]

Metadata propagation algorithm moved from Designer to Server Improvement CLO-6437 4.2.0-M1
Allow more hyperlinks in graph run log Improvement CLO-7023 4.2.0-M1
Custom password storage for remembering Server project credentials Fix CLO-8806 4.2.0
When Designer connects to a remote Server sandbox and creates the local synchronized copy (default behaviour for 'shared' type sandbox), it remembers the password in an encrypted form in a proprietary storage format in workspace .metadata folder. We had to switch from standard Eclipse password store because of numerous internal issues. Please check the related issue link for details.


ETL Developers (This item is most useful for developers; it either brings new functionality for transforming data or brings optimizations)

Administrators (This item is an improvement or feature that will help setup, install, administer and manage the application)

Support (This item helps staff supporting the production operation to identify potential problems or avoid such)