CloverETL 4.5.0


CloverETL 4.5 adds another addition to the Salesforce connector family—the Salesforce Wave Writer. It also introduces enhancements to the Event Listeners UI in the Server Console and adds options for restarting jobs from the Executions History view with parameters.

This release does a bit of housekeeping too, removing now obsolete CTL1 support and backquote inline code. Please check the Compatibility section for more details.



About this release: This is a production release. We recommend upgrading to this version from all previous versions. Please check Installation/Upgrade and especially the Compatibility section for details.

Released Mar 14, 2017
  • Production
Recent Releases
4.7.0 Oct 10, 2017
4.6.1 July 19, 2017
4.5.0 Mar 14, 2017
4.4.1 Feb 14, 2016
4.3.1 Nov 10, 2016
4.2.1 Aug 18, 2016
4.1.4 Jun 1, 2016

See all releases

Try it yourself

TRY NOW


New features in 4.5


Salesforce Wave Connector

We're introducing new Salesforce Wave writer—a simple solution for loading data into Wave Analytics/BI platform. You can create new datasets and append or update/insert data into existing ones.

CloverETL - Salesforce Wave Writer Connector

Event Listeners overhaul

We've redesigned the user interface for Event Listeners in Server. It has been consolidated into a single module listing all listeners present in the system for a better overview.

CloverETL - New Event Listeners module

Restart job on Server; modify parameters if needed

You can now restart a job on a Server (useful for restarting failed executions) from the Executions History tab. You can also change the parameters of the job (graph or jobflow) before restarting it.

In the Executions History view in the Server Console, you have the option to either restart a job using the same parameters it was first executed with or manually change the parameters before restarting the job.

Imagine a file listener starting a job. It passes a file path to the job as a parameter. Restarting such a job requires knowing the exact parameters that the listener provided. With this update, you can easily reuse the starting parameters, modify them if you need to, and restart the job.

CloverETL - Restart job on Server

Removed obsolete CTL1 + additional compatibility changes

We are completely removing CTL1 code from CloverETL.

This obsolete predecessor of the current CTL2 (CTL2 was introduced in CloverETL 3.0 in 2010) has long been deprecated, so we've now moved to completely remove it from the product.

With this change, inline expressions in backquotes (e.g. "filename-`today()`.csv") are being removed as well, as this syntax caused numerous issues. Instead, use dynamic parameters for on-the-fly calculations.

We're doing some long-awaited housekeeping in this release. None of the changes should affect your everyday use of CloverETL. However, we recommend going over the Compatibility section of this release.


Installation/Upgrade Instructions for CloverETL 4.5

To help you install or upgrade to this version, we've prepared a simple checklist:

Before You Upgrade

  • Be sure to check the "Compatibility" notes for ALL intermediary releases. We mark all changes that can potentially alter the function of your existing transformations with a "Compatibility" label to make it easier for you. You can safely ignore most of them, as we try hard to keep as much backwards compatibility as we can. There's a comprehensive list of all releases that will help you get the information quickly.
  • Upgrade Designer and Server together. We always release Designer and Server together under a single version. It's highly recommended to upgrade Server and Designer at the same time. Although using different versions of Designer to connect to Server might work, it is not generally supported.
  • There are no incremental patches. We don't release incremental patches. Every upgrade is in fact a full installation that, if installed over older version, will automatically update whatever is necessary in your workspaces, sandboxes, and Server databases as needed.
  • Don't forget to backup. Although none of the above upgrade steps requires explicit backup, we recommend you always back up your work. The upgrade will keep all your transformations, jobflows, and configurations safe. However, as a good word of advice, it never hurts to have a backup.

Designer Upgrade

  • Download the latest version by logging into your customer account. If you no longer have access there, click here to recover your password or contact our CloverCARE Support.
  • Install the new version of Designer. You can install Designer over your existing installation. The process will automatically clean up the old version. Don't worry, you will NOT lose your workspaces, graphs, and transformations. However, if you installed some additional plugins to Designer (Eclipse plugins) you might need to reinstall them. Eclipse should automatically help you do that. When you start the application, point it to your existing workspace directory. With some major releases, we may notify you about upgrading the workspace to the latest version. In such cases, you won't be able to use the workspace with previous versions. Thus, be sure to upgrade all Designers if you're sharing the workspace.
  • Activate the product on first start. You will need a new key as we issue new license keys for every new major version (i.e. from 4.2 to 4.3). However, if you're on our maintenance program, we automatically renew the keys for you. Just go to the download area again and copy/paste the license keys from there. If you can't find the latest keys, please contact us to renew your product maintenance.

Server Upgrade

  • Download the latest version by logging into your customer account. You'll find Server in the same list as Designer downloads. If you no longer have access there, click here to recover your password or contact our CloverCARE Support.
  • Plan for downtime. Upgrading Server requires downtime, so plan your upgrades in advance. If you're running multiple environments, upgrade the non-production installation first and run all your checks there first.
  • Follow step-by-step Server Upgrade Guide. We've prepared detailed instructions on how to properly shut down Server and install a new one. Server will upgrade its database and sandboxes from any previous version automatically.
  • Activate the product on the login screen of Server Console. The license key changes with every major version (i.e. from 4.2 to 4.3) and we automatically renew the keys for you. Just go to the download area again and copy/paste the license key from there.




Detailed list of improvements and fixes in 4.5

Salesforce Connector

Salesforce Wave writer
New Feature CLO-9493 4.5.0-M1
Support for writing datasets into the Salesforce Wave analytics platform. Supports creating new data sets, as well as append, upsert and delete
Updated Salesforce connectors to use Salesforce API version 39
Compatibility CLO-10170 4.5.0-M2
Added batch size configuration mode to SalesforceWriter and SalesforceBulkWriter.
New Feature CLO-10474 CLO-9936 4.5.0-M2
Reducing the batch size can help avoid Apex CPU limit for slowly running Apex triggers
Added concurrency mode option to SalesforceBulkWriter. This option allows switching from parallel (default) to serial to avoid lock contentions when loading multiple batches.
New Feature CLO-9936 4.5.0-M2

MongoDB Support

Autentication Database can now be specified as part of a MongoDB connection.
New Feature CLO-5728 4.5.0-M1
See more details https://docs.mongodb.com/manual/core/security-users/#user-authentication-database
"Use SSL" new configuration option available on MongoDB connection
New Feature CLO-9579 4.5.0-M1
Bundled driver updated to version 3.4. Older versions are still supported.
Improvement CLO-10175 4.5.0-M1

File Event Listeners

Show 10 last performed file listener checks and their status
Improvement CLO-10025 4.5.0-M1
File listeners can now be set to ignore zero-length files
New Feature CLO-9845 4.5.0-M1

UI Improvements

“Enable with condition” dialog now allows creating new parameters directly.
Improvement CLO-9957 4.5.0-M1
This makes it easier to set up, withour prior definition of the required parameter
JSON and XML Extract mapping dialogs now validate metadata, no longer allowing you to create invalid metadata by accident.
Improvement CLO-7448 4.5.0-M1
Added consistency check on readers: Error edge only makes sense if Data policy is set to Controlled
Improvement CLO-9927 4.5.0-M1

Server

Restart job with same parameters
New Feature CLO-9499 CLO-9709 4.5.0-M1
Now you can restart a job with the same input parameters (or you can change them before running the job). This greatly simplifies restarting jobs with parameters provided by Listeners or other parameter-driven automation.
There is now just a single Event Listeners module that consolidates all types of Event Listeners (file, graph, jms, …) into one unified UI for a better overview. This is a user interface update only; all your listeners will continue working without change.
Improvement CLO-10068 4.5.0-M2
Expanding the Restart Job feature from M1, you can now change parameters of a job before restarting it from Executions History.
New Feature CLO-9710 4.5.0-M2

CTL

Removed support for CTL1 (deprecated as of CloverETL 3.0 with the introduction of the now standard CTL2). With this update we're completely removing CTL1 from the product. Any transformations written in CTL1 will no longer work after this update and will have to be rewritten to CTL2.
Compatibility CLO-6378 4.5.0-M2
Unfortunately, there's no automated conversion tool between CTL1 and CTL2. We recommend revisiting and rewriting any CTL1 code manually.
Inline CTL code in backquotes (e.g. "filename-`today()`.csv") is no longer supported. This feature has been completely removed with CTL1. Backquotes now resolve as ordinary characters and no longer produce warnings when used. This mainly affects MySQL query syntax which no longer requires escaping with backslashes (although that syntax is still valid and is backwards compatible). As a replacement for inline code in backquotes, use dynamic parameters.
Compatibility CLO-9579 4.5.0-M2

Components

JSONExtract and XMLExtract will now fail if there is a field referenced in the mapping that is not present in the output metadata.
Compatibility Improvement CLO-9907 4.5.0-M2
"ExtFilter" component renamed to "Filter". Existing graphs designed with ExtFilter will now display "Filter" when opened in 4.5. However they can still be used in older versions without problems.
Compatibility CLO-10300 4.5.0-M2

Designer

Runtime now starts immediately after Designer startup. (You can change this behavior under Window > Preferences > CloverETL > ETL Runtime: Start Runtime on Designer startup)
Improvement CLO-10320 4.5.0-M2

Other Improvements

Fixed confusing error message on UniversalDataReader when maximum number of errors was exceeded
Improvement CLO-10084 4.5.0-M1
Cluster: Auto resume counter is reset after 4 hours of continuous success checks
Improvement CLO-10071 4.5.0-M1
Performance improved for Titan FTP Server
Improvement CLO-8609 4.5.0-M1

Fixes

Fixed MSSQLDataWriter with sqlncli (bcp for linux) throwing "Unable to open BCP host data-file"
Fix CLO-9873 4.5.0-M1
HttpConnector uses CloverETL engine default charset instead of the system default one. This is a backwards incompatible change, as the two defaults can be different. You can set the charset directly on the component or use the now default configuration parameter Defaults.DataParser.DEFAULT_CHARSET_DECODER in defaultProperties.
Compatibility Fix CLO-10204 4.5.0-M2
Server configuration export/import: Fixed potential deadlock during configuration import when using MSSQL as Server database
Fix CLO-10549 4.5.0
Server configuration export/import: Uncommitted preview sometimes caused changes in current configuration, fixed.
Fix CLO-10536 4.5.0
UI improvements in Server / Tasks History—icons, resizable table, etc.
Improvement CLO-10494 4.5.0
Fixed: Metadata extraction fails when a MSSQL table name starts with a digit (e.g. "123table")
Fix CLO-10471 4.5.0
Fixed: Passwords containing "@" weren't properly masked in graph logs and Console output.
Fix CLO-10457 4.5.0
ExecuteScript no longer hangs on a process that cannot be killed within a given time limit. Instead, it now properly times out and finishes, but the background process remains active until it stops naturally or hangs there forever.
Fix CLO-10108 4.5.0
Fixed: Profiler Reporting Console login failures
Fix CLO-9808 4.5.0
Fixed: Metadata extracted from Salesforce are quoted by default, which isn't standard for other forms of metadata extraction
Fix CLO-10543 4.5.0

Compatibility

JSch library updated to 0.1.54 (S/FTP)
Compatibility CLO-6538 4.5.0-M1
MongoDB: Bundled driver updated to version 3.4. Older versions are still supported.
Compatibility CLO-10175 4.5.0-M1
Removed support for CTL1 (deprecated as of CloverETL 3.0 with the introduction of the now standard CTL2). With this update we're completely removing CTL1 from the product. Any transformations written in CTL1 will no longer work after this update and will have to be rewritten to CTL2.
Compatibility CLO-6378 4.5.0-M2
Unfortunately, there's no automated conversion tool between CTL1 and CTL2. We recommend revisiting and rewriting any CTL1 code manually.
Inline CTL code in backquotes (e.g. "filename-`today()`.csv") is no longer supported. This feature has been completely removed with CTL1. Backquotes now resolve as ordinary characters and no longer produce warnings when used. This mainly affects MySQL query syntax which no longer requires escaping with backslashes (although that syntax is still valid and is backwards compatible). As a replacement for inline code in backquotes, use dynamic parameters.
Compatibility CLO-9579 4.5.0-M2
"ExtFilter" component renamed to "Filter". Existing graphs designed with ExtFilter will now display "Filter" when opened in 4.5. However they can still be used in older versions without problems.
Compatibility CLO-10300 4.5.0-M2
Updated Salesforce connectors to use Salesforce API version 39
Compatibility CLO-10170 4.5.0-M2
HttpConnector uses CloverETL engine default charset instead of the system default one. This is a backwards incompatible change, as the two defaults can be different. You can set the charset directly on the component or use the now default configuration parameter Defaults.DataParser.DEFAULT_CHARSET_DECODER in defaultProperties.
Compatibility Fix CLO-10204 4.5.0-M2
JSONExtract and XMLExtract will now fail if there is a field referenced in the mapping that is not present in the output metadata.
Compatibility Improvement CLO-9907 4.5.0-M2
Dropped binary builds for Linux 32-bit
4.5.0


ETL Developers (This item is most useful for developers; it either brings new functionality for transforming data or brings optimizations)

Administrators (This item is an improvement or feature that will help setup, install, administer and manage the application)

Support (This item helps staff supporting the production operation to identify potential problems or avoid such)