CloverETL 4.4.1

CloverETL 4.4 expands Salesforce connectivity with new SalesforceReader and Writer components that bring support for complex SOQL queries (function calls, joins, subqueries) and binary attachments.

In this update, we also continued working on Server features—adding support for Remote File Listeners (the ability to watch FTP, SFTP, Samba, and S3 folders) and introducing Task Failure Listeners, an additional automation option for monitoring and recovery of Event listeners.


Don’t forget to pay attention to the Compatibility section. We’re changing some defaults that may potentially affect many core components. However, you shouldn’t see any negative impact, as the changes we’re making are aimed towards more predictable, and thus error-proof, behavior.


About this release: This is a production release. We recommend upgrading to this version from all previous versions. Please check Installation/Upgrade and especially the Compatibility section for details.

Released Feb 15, 2017
  • Production
Recent Releases
4.7.0 Oct 10, 2017
4.6.1 July 19, 2017
4.5.0 Mar 14, 2017
4.4.1 Feb 14, 2016
4.3.1 Nov 10, 2016
4.2.1 Aug 18, 2016
4.1.4 Jun 1, 2016

See all releases

Try it yourself

TRY NOW

Previous & deprecated releases:

4.4.0
4.4.0-M1
4.4.0-M2


New features in 4.4


Salesforce Connector—Complex Queries and Attachments

We've expanded our Salesforce connector family with SOAP-based SalesforceReader/Writer components. The SOAP API gives you support for complex SOQL queries—aggregation and multi-currency functions, joins, aliases, reading deleted records (for capturing incremental changes) and subqueries—as well as reading and uploading of attachments.

With these components you also save your API limits—each query or write takes only a single API call (per 2,000 records) and returns results faster compared to SalesforceBulkReader/Writer, which work best for large batches.

However, these new components are limited to low data volumes, typically up to thousands of records. That makes them great for lookups or ad hoc queries, but for large data sets, SalesforceBulkReader/Writer would be a much better option.

CloverETL - Salesforce Connectors Attachments, Complex Queries

File Listeners—Remote Folders and "File Finished" Check

Corporate Server and Cluster only

You can now watch "hot" remote folders—FTP, SFTP, Amazon S3 and Samba shares. Previously, only local folders were supported.

Also, you can now avoid jobs triggering on unfinished files (large files might appear in a folder before they finish copying/uploading). Do so by setting a polling interval that blocks the trigger until the file size has stabilized.

CloverETL - File Event Listeners

Task Failure Listeners

Corporate Server and Cluster only

Task Failure listeners allow you to configure reactions to failures of other listeners (e.g. a File Event Listener watching a "hot" folder starts failing because the folder becomes unavailable).

The reaction can be anything a normal listener can do, from sending emails (default) to running cleanup jobs, scripts, etc.

We've integrated these directly into Listeners modules (look for "Create Notification" button).

CloverETL - Task failure listeners

Amazon Redshift Driver is Now Built-in

You'll find the Amazon Redshift driver in the DB connection configuration dialog out-of-the-box now. Amazon recommends using this driver instead of the Postgres driver.

ParallelReader Supports S3

ParallelReader component now supports native parallel reading from Amazon S3.

Filter shortcut in Data Inspector

Right-click a value to quickly turn it into a filter in Data Inspector.

CloverETL - Data Inspector Filter shortcut

Omit XML declaration in XMLWriter to easily generate fragments

You can instruct XMLWriter to omit <?xml ... ?> declaration to write out fragments which you can later easily append or merge together.

By default the declaration is generated (set to "false"), maintaining backwards compatibility.

CloverETL - Omit XML declaration in XMLWriter

Compatibility

Please read carefully:

We've changed the default encoding for many components to UTF-8 (from ISO-8859-1).

This might affect your existing transformations (e.g. failing or records that are were okay before but are now invalid in UTF-8 definition).

We recommend reviewing the Compatibility section for this release. We're introducing some changes to defaults that could affect your production data (only if you are relying on some strangely behaved defaults).


Installation/Upgrade Instructions for CloverETL 4.4.x

To help you install or upgrade to this version, we've prepared a simple checklist:

Before You Upgrade

  • Be sure to check the "Compatibility" notes for ALL intermediary releases. We mark all changes that can potentially alter the function of your existing transformations with a "Compatibility" label to make it easier for you. You can safely ignore most of them, as we try hard to keep as much backwards compatibility as we can. There's a comprehensive list of all releases that will help you get the information quickly.
  • Upgrade Designer and Server together. We always release Designer and Server together under a single version. It's highly recommended to upgrade Server and Designer at the same time. Although using different versions of Designer to connect to Server might work, it is not generally supported.
  • There are no incremental patches. We don't release incremental patches. Every upgrade is in fact a full installation that, if installed over older version, will automatically update whatever is necessary in your workspaces, sandboxes, and Server databases as needed.
  • Don't forget to backup. Although none of the above upgrade steps requires explicit backup, we recommend you always back up your work. The upgrade will keep all your transformations, jobflows, and configurations safe. However, as a good word of advice, it never hurts to have a backup.

Designer Upgrade

  • Download the latest version by logging into your customer account. If you no longer have access there, click here to recover your password or contact our CloverCARE Support.
  • Install the new version of Designer. You can install Designer over your existing installation. The process will automatically clean up the old version. Don't worry, you will NOT lose your workspaces, graphs, and transformations. However, if you installed some additional plugins to Designer (Eclipse plugins) you might need to reinstall them. Eclipse should automatically help you do that. When you start the application, point it to your existing workspace directory. With some major releases, we may notify you about upgrading the workspace to the latest version. In such cases, you won't be able to use the workspace with previous versions. Thus, be sure to upgrade all Designers if you're sharing the workspace.
  • Activate the product on first start. You will need a new key as we issue new license keys for every new major version (i.e. from 4.2 to 4.3). However, if you're on our maintenance program, we automatically renew the keys for you. Just go to the download area again and copy/paste the license keys from there. If you can't find the latest keys, please contact us to renew your product maintenance.

Server Upgrade

  • Download the latest version by logging into your customer account. You'll find Server in the same list as Designer downloads. If you no longer have access there, click here to recover your password or contact our CloverCARE Support.
  • Plan for downtime. Upgrading Server requires downtime, so plan your upgrades in advance. If you're running multiple environments, upgrade the non-production installation first and run all your checks there first.
  • Follow step-by-step Server Upgrade Guide. We've prepared detailed instructions on how to properly shut down Server and install a new one. Server will upgrade its database and sandboxes from any previous version automatically.
  • Activate the product on the login screen of Server Console. The license key changes with every major version (i.e. from 4.2 to 4.3) and we automatically renew the keys for you. Just go to the download area again and copy/paste the license key from there.




Maintenance release 4.4.1

Fixes

Fixed: objectGUID attribute couldn't be correctly obtained from Active Directory
Fix CLO-10223 4.4.1
Fixed: When importing an external CTL file more than once, the warning "Multiple import: ignoring redundant occurrences" throwed Unexpected null value
Fix CLO-10313 4.4.1
Fixed: Metadata propagation fails for parametrized subgraph hierarchy
Fix CLO-10240 4.4.1
Fixed: Scheduling module on Server—Cron expression edit dialog doesn't store values and refreshes page unexpectedly
Fix CLO-10101 4.4.1
Fixed: Launch Services on Server—Incorrect Test Launch Service URL when switching to HTTPS
Fix CLO-10058 4.4.1
Fixed confusing message at the end of a truncated error message in log
Fix CLO-10034 4.4.1
Fixed: Multiple Redshift components can now run simultaneously. Detected as DBLookupTable not being possible to use by two components concurrently in a single phase
Fix CLO-9995 4.4.1
Fixed: LDAPWriter—byte on input throws NPE
Fix CLO-9760 4.4.1
Fixed: Added config properties are not visible with Oracle DB
Fix CLO-8507 4.4.1
Sequences are now serialized into the .grf XML file in alphabetical order which helps make the file more stable for version control.
Fix CLO-9843 4.4.1


All improvements and fixes in 4.4.0

Salesforce CRM Connector

New SalesforceReader component—Full SOQL support—functions, joins, subqueries. Saves API calls (1 call per query per 2,000 records), supports attachments. Based on SOAP API.
New Feature CLO-9171 4.4.0-M2
New SalesforceWriter—SOAP API based, also supports writing Attachments
New Feature CLO-9172 CLO-9492 4.4.0-M1
Advanced visual SOQL editor
Improvement CLO-9428 4.4.0-M1
SOQL syntax highlighting supported
Improvement CLO-9429 4.4.0-M1
Inline editing of SOQL queries in component editor
Improvement CLO-9328 4.4.0-M1
SalesforceReader can be set to read deleted records. Salesforce keeps deleted/archived records temporarily (~2 weeks) available for reading. This is quite useful for CDC, comparing two snapshots and finding deleted records.
New Feature CLO-9902 4.4.0
SalesforceWriter—non-existent file when uploading attachments is reported on reject port
Improvement CLO-9538 4.4.0-M1
SalesforceBulkWriter—warning if trying to use unsupported upload of attachments
Improvement CLO-9502 4.4.0-M1
SalesforceReader—support for multi-currency—convertCurrency()
New Feature CLO-9853 4.4.0
SalesforceReader—added support for polymorphic keys and relationships
New Feature CLO-9823 4.4.0
SalesforceReader—support function distance()
New Feature CLO-9819 4.4.0
SalesforceReader—added support for count() without parameter
4.4.0
SalesforceReader—subquery join type is now always set to LEFT OUTER JOIN
Compatibility CLO-9915 4.4.0

Task Failure Listeners

Introducing "Task Failure" listeners. A new way to trigger tasks (emails, job execution, script) when a task fails to execute (e.g. a file listener is unable to check a folder because the folder is no longer available)
New Feature CLO-9473 4.4.0-M1
New shortcut in Event Listeners for creating and managing failure notifications (Task Failure Listeners)
Improvement CLO-9659 4.4.0-M2
On Detail tab of a Task Failure Listener, you can now see a listing of all parameters available to the graph executed as the reaction to the failure (applies only if "Task" is set to "Start a graph")
Improvement CLO-9715 4.4.0-M2

Remote File Event Listeners

Added support for remote locations to File Event listeners: ability to watch FTP, SFTP, Samba and S3 folders
New Feature CLO-3934 4.4.0-M1

File Event Listeners

Improved URL entry in File Listeners (driven by the need for more complex URLs for the new Remote File Listeners). A new Paste URL option will automatically parse and populate the form fields (host, port, …). Validate Accessibility button can be used to verify that CloverETL can connect to the specified remote location.
Improvement CLO-9723 4.4.0-M2
File Listener can now wait for a file to complete uploading/copying by waiting until its size is stabilized and constant
New CLO-9758 4.4.0
Improved File Listener form layout to simplify configuration
Improvement CLO-9724 CLO-9476 4.4.0
Implicit and custom parameters are now displayed in detail of FileEventListener
Improvement CLO-9906 CLO-9905 4.4.0

Amazon Web Services

ParallelReader support for Amazon S3
New CLO-9500 4.4.0-M1
JDBC driver for Amazon Redshift is now bundled with CloverETL (Replaces Postgres for Redshift use)
New CLO-7143 4.4.0-M1

Cluster

Improved auto-suspend and re-join for nodes that temporarily go down
Improvement CLO-9651 CLO-9590 4.4.0

UI Improvements

Record counts on edges are now highlighted on selection for better visibility
Improvement CLO-9569 4.4.0-M2
Server Console: Scheduler Enable/Disable shortcut button in Scheduling tab
Improvement CLO-9954 4.4.0
Decimal field precision is now displayed in Transform editor metadata panels
Improvement CLO-9920 4.4.0

Other Improvements

XMLWriter can be instructed to omit XML declaration in its output for easier writing of XML fragments
New CLO-9465 4.4.0-M1
Debugging/CTL: Tooltips now work for records too (not only variables)
Improvement CLO-9412 4.4.0-M1
Inline CTL is now prettier in the graph XML file format, which is easier for human eyes to read
Improvement CLO-9244 4.4.0-M1
Data inspector: New filtering shortcuts
Improvement CLO-9201 4.4.0-M1
File URL dialog: Explains discrete/source/stream port writing or reading
Improvement CLO-8973 4.4.0-M1

Fixes

Fixed: Automatic metadata propagation makes Designer unresponsive
Fix CLO-9097 4.4.0-M1
Fixed: UniversalDataReader very slow when using wildcards in archives
Fix CLO-6584 4.4.0-M1
Fixed: Wildcard resolution in readers matches directories
Fix Compatibility CLO-9619 4.4.0-M1
Fixed: Designer performance drop for graph with many subgraphs
Fix Improvement CLO-9466 4.4.0-M1
Implicit metadata are not persisted any more
Fix Improvement CLO-9530 4.4.0-M1
Fixes in Cluster error handling (nodes not respecting forced suspend, ignoring heartbeat failures, deadlocks on network failures)
Fix CLO-9591 4.4.0-M2
Fixed: Server Temp Space setting not being used after restart
Fix CLO-9875 4.4.0
Fixed: EmailFilter creating multiple duplicate records
Fix CLO-9757 4.4.0
Fixed: Not able to link a connection from Component Editor dialog
Fix CLO-9911 4.4.0
Fixed: EmailFilter causing ConcurrentModificationException
Fix CLO-9872 4.4.0
Logs in Designer console and Server logs now supports UTF-8
Fix CLO-8349 4.4.0
Fixed: WebServiceClient causing NullPointerException when Response Mapping was used as a bound parameter in a subgraph
Fix CLO-9804 4.4.0-M2
Fixed: Value of Root context blank in JNDI Browser in Pivotal tc Server
Fix CLO-9829 4.4.0

Differences/Fixes since the Milestone Releases

SalesforceReader: subquery join type is now fixed to LEFT OUTER JOIN
Compatibility CLO-9915 4.4.0
UI improvements in File Event Listener tab on Server
Improvement CLO-9919 4.4.0
Fixed: SalesforceReaders—queries with ORDER or GROUP clauses didn't work
Fix CLO-9938 4.4.0
Fixed: SalesforceBulkWriter doesn't handle unprocessed records
Fix CLO-9935 4.4.0
Fixed: Internal Salesforce connection persists default hostname "login.salesforce.com"
Fix CLO-9933 4.4.0
Fixed: SalesforceReader doesn't return subquery results when letter case in query doesn't match
Fix CLO-9903 4.4.0
Unified SOQL parser between SalesforceReader and SalesforceBulkReader
Compatibility CLO-9890 4.4.0
Fixed: SalesforceReader—correctly supports string literals with escaped characters
Fix CLO-9827 4.4.0
Changed default task failure notification e-mail template
Improvement CLO-9854 4.4.0
Sort Salesforce object fields
Improvement CLO-9550 4.4.0-M1

Compatibility

Changed default encoding to UTF-8 (formerly was ISO-8851-1). Affects UniversalDataReader/Writer, FlatFileReader/Writer, etc. See related ticket to see the list of affected components
Compatibility CLO-9676 4.4.0-M2
SpreadsheetDataWriter no longer allows String field as a target when writing the resulting Excel (binary) data to an output port
Compatibility CLO-9726 4.4.0-M2
Unified handling of "decimal" field type in implicit vs. explicit mappings. Implicit mapping (applied when component's "Mapping" attribute is blank) formerly produced NULL values when source and destination decimal precisions didn't match. In other words, the component treated the two fields as of different types and didn't map them. As of 4.4, decimal fields are always implicitly mapped despite their potentially different precisions. This prevents losing data (i.e. fields aren't silently ignored in implicit mappings anymore) but can produce runtime errors (only) for values that are larger than available destination precision. (Work in progress and logic might change for production release)
Compatibility Fix CLO-9831 4.4.0-M2
Please note this applies to IMPLICIT mappings ONLY. If you've defined any explicit mapping ($out.decimalField = $in.0.sourceDecimal) or star mapping ($out.0.* = $in.0.*), the new behavior was already there—you would get a warning in the mapping dialog and you would get runtime errors for values that wouldn't fit.Affected components using the implicit mapping: SalesforceBulkWriter, SalesforceWriter, SalesforceReader, SalesforceBulkReader, EmailFilter, EmailSender, DBExecute, DBOutputTable, HttpConnector, TokenGather, Barrier, Profiler, Validator, MongoDBReader, MongoDBWriter, MongoDBExecute and jobflow components with mappings
Update bundled Java in Designer to 1.8.0_101
Compatibility CLO-9535 4.4.0-M1
SalesforceReader—subquery join type is now always set to LEFT OUTER JOIN
Compatibility CLO-9915 4.4.0

Known Issues

Salesforce components don't support compound field type (e.g. Address)
Known Issue CLO-9833 4.4.0
OpenJDK not officially supported for CloverETL Server. We've added a warning in Server Console when Server is running on OpenJDK.
Known Issue CLO-9959 4.4.0


ETL Developers (This item is most useful for developers; it either brings new functionality for transforming data or brings optimizations)

Administrators (This item is an improvement or feature that will help setup, install, administer and manage the application)

Support (This item helps staff supporting the production operation to identify potential problems or avoid such)