Customer Portal

CloverETL 4.8 adds numerous improvements to Data Services and introduces an assortment of minor useful improvements.

We focused on securing Data Services, adding management of SSL contexts for individual REST jobs and adding a "sensitive" option to parameters, preventing them from showing up in Server logs, Execution History, etc. We also simplified configuration of reading input data as byte or character streams.

Expanding the security theme, we're introducing a Server login "lockout" to prevent brute-force password guessing and a new option to disable HTTP Server API (and keep only HTTPS). Even more on the security front, there's now support for Kerberos in Hadoop connection (HDFS + YARN) and Hive JDBC connections.

New features in 4.8

Data Services—SSL Context Configuration

There's a new more accessible configuration for your Data Services running over SSL. You can deploy multiple contexts using different keys and options. And each Data Service can be either running in the default Server context or one of the custom ones.

Data Services—Sensitive parameters

Input parameters of a Data Service can be marked as sensitive. They work just like normal parameters with one exception— their values are masked with an asterisk in Server logs, Executions History, etc.

Data Services—Failure notifications

Alerts and notifications can be configured for each Data Service. You can choose ranges of HTTP codes that are reported as failure, as well as failure rates to alert you only in serious cases and not bother in occasional hiccups.

Data Services—Streaming Request Body

An improvement from last version, the binary (or string) body of a request can be directly read using an edge coming from Body of the input panel connected to a component with a "port:" File URL.

Server Security updates

New password lockout mechanism can be configured for the Server to prevent brute-force attacks. You can configure the number of attempts before the account gets locked and a timeout for next try.

To further secure the Server, you can now completely disable Server's HTTP API to eliminate potential API attacks.

Kerberos Supported in Hadoop Connection

Hadoop connections (HDFS and YARN) and Hive JDBC connection now support Kerberos authentication.

MongoDB—Improved Bulk Write and new Bulk Update

MongoDB 3.2 brought an improved bulk write operation and also introduced new bulk update function. We've updated CloverETL MongoDB implementation to support these new features.

Compatibility Notes

We dropped support for Apache Tomcat 6.x, added support for Tomcat 9.x and updated the Server Bundle to Tomcat 8.0.47.

To help you install or upgrade to this version, we've prepared a simple checklist:

Before You Upgrade

  • Be sure to check the "Compatibility" notes for ALL intermediary releases. We mark all changes that can potentially alter the function of your existing transformations with a "Compatibility" label. Typically, you can safely ignore most of them, as we try hard to keep as much backwards compatibility as possible. There's a comprehensive list of all releases that will help you get the information quickly.
  • Upgrade Designer and Server together. We always release Designer and Server together under a single version. It's highly recommended to upgrade Server and all Designers at the same time. Although using different versions of Designer to connect to Server might work, it is not generally supported.
  • There are no incremental patches. We don't release incremental patches. Every upgrade is in fact a full installation that, if installed over the older version, will automatically update whatever is necessary in your workspaces, sandboxes, and Server databases as needed, no user data will get overwritten.
  • Don't forget to backup. Although none of the above upgrade steps requires explicit backup, we recommend you always back up your work. The upgrade will keep all your transformations, jobflows, and configurations safe. However, as a good word of advice, it never hurts to have a backup.

Designer Upgrade

  • Download the latest version by logging into your customer account. If you lost your credentials or no longer have access there, click here to recover your password or contact our CloverCARE Support.
  • Install the new version of Designer. You can install Designer over your existing installation. The process will automatically clean up the old version. Don't worry, you will NOT lose your workspaces, graphs, and transformations. However, if you installed some additional plugins to Designer (Eclipse plugins) you might need to reinstall them. Eclipse should automatically help you do that. When you start the application, point it to your existing workspace directory. With some major releases, we may notify you about upgrading the workspace to the latest version. In such cases, you won't be able to use the workspace with previous versions—be sure to upgrade ALL Designers at once if you're sharing the workspace.
  • Activate the product on first start. You will need a new key as we issue new license keys for every new major version (e.g. from 4.6 to 4.7). If you're on our maintenance program, we automatically renew the keys for you. Just go to the download area again and copy/paste the license keys from there. If you can't find the latest keys, please contact us to renew your product maintenance.

Server Upgrade

  • Download the latest version by logging into your customer account. You'll find Server in the same list as Designer downloads. If you no longer have access there, click here to recover your password or contact our CloverCARE Support.
  • Plan for downtime. Upgrading Server requires downtime, so plan your upgrades in advance. If you're running multiple environments, upgrade the non-production installation first and run all your tests there first.
  • Follow our step-by-step Server Upgrade Guide. We've prepared detailed instructions on how to properly shut down Server and install a new one. Server will upgrade its database and sandboxes from any previous version automatically.
  • Activate the product on the login screen of Server Console. Server requires new license keys with every major version (e.g. from 4.6 to 4.7) and we automatically renew the keys for you. Just go to the download area again and copy/paste the license key from there.
Release Compatibility/Upgrade notes, Features & Fixes Published Download

Fixes

Fixed checkconfig (instant validation of graphs during development) overloading the Server for huge graphs
Fix CLO-13820 4.8.3
Fixed freezing Designer when developing large graphs
Fix CLO-13818 4.8.3
Fixed retrieving execution history via API with Data Service run records
Fix CLO-13751 4.8.3
Fixed node auto-rejoin in Cluster not working
Fix CLO-12464 4.8.3

Compatibility

Designer installed as an Eclipse plugin now requires Eclipse 4.5
Compatibility CLO-13819 4.8.3
July 16, 2018 Download CloverETL 4.8 4.8.3

Improvements

Added LZ4 compression for remote edges for better network performance
Improvement CLO-13157 4.8.2
S3: enabled the default credentials provider chain when the S3 URL does not contain secret key + access key
Improvement CLO-13093 4.8.2

Compatibility

Added support for Tomcat 8.5 & Pivotal tcServer 3.2.9
Compatibility CLO-12921 CLO-12918 4.8.2

Fixes

Fixed failing get() function when using DB lookup in MS SQL database. See more details in the linked issue.
Fix CLO-13006 4.8.2
Fixed corrupted URLs being passed to subgraphs when they had GZIP protocol in them.
Fix CLO-13153 4.8.2
Fixed SocketTimeoutException when using remote edges in a clustered graph. See more detail in the linked issue.
Fix CLO-11848 4.8.2
April 16, 2018 Download CloverETL 4.8 4.8.2

Data Services

Export Data Services Configuration now exports Alerts and Notification settings. Notice, Data Services exported from 4.8.1+ and imported into 4.8.0 will not import these additional settings.
New feature CLO-11546 4.8.1
When you modify "Alerts and Notification" settings in a way that resets the FAILING state of a Data Service to STABLE, the system will send a notification email (if configured) reflecting the Data Service is now STABLE (caused by manual reset).
New feature CLO-11660 4.8.1

Features & Improvements

Improved performance when starting jobs with lots of data in input parameters.
Improvement CLO-12691 4.8.1
WebServiceClient component now accepts SOAP responses in XOP (XML-binary Optimized Package). This fixes "No response message received for InOut MEP" error.
New feature CLO-12521 4.8.1

Fixes

Fixed subgraphs causing a deadlock when a parent connects a different number of ports to a subgraph than it is designed for. This can occur when you change the subgraph's port layout without modifying the parent graph in Designer. Before this fix, running the parent graph would cause an infinite deadlock, now it throws an error.
Fix CLO-12410 4.8.1
Fixed MongoDBWriter causing "Execution failed: Collection name not specified" error. Now the component uses the collection name from the first record (subsequent values are ignored).
Fix CLO-12309 4.8.1
Fixed CLOSE_WAIT lingering TCP connections after graph finishes in Clustered mode (e.g. on multiple nodes or with data partitioning)
Fix CLO-12302 4.8.1
Fixed Data Inspector not showing all records on edges while the graph is running in some cases (with just a few <1000 records)
Fix CLO-11146 4.8.1
February 25, 2018 Download CloverETL 4.8 4.8.1

Data Services

Read request body as stream using an edge connected to "Body" port on the Data Service REST job Input panel
Improvement CLO-11788 CLO-11981 4.8.0-M1
Manage multiple SSL contexts for Data Services, with different keys, URLs, etc.
New feature CLO-11380 4.8.0-M1
Mark REST job parameters as "sensitive" to hide them from Server logs, Executions History, etc.
New feature CLO-11780 4.8.0-M1
New filter widget in Data Services module for quickly finding a particular service
New feature CLO-11714 4.8.0-M1
Job test page can now be used send HTTP request payloads, testing services using a "body" parameter.
New CLO-12145 4.8.0
"Never mark endpoint as failing"—New option in Alerts and notification for ignoring a particular Data Service failures
New CLO-12077 4.8.0
HTTP code or range of codes can be used to identify what is and what's not considered a failure in Alerts and Notificatons
New CLO-12075 4.8.0
Executing a data service from its testing page or from Designer now stores a record in Executions History
Improvement CLO-12045 4.8.0
State and History tab now shows a cause of a failure
Improvement CLO-12040 4.8.0

Connectors

We now allow SELECT queries in DBExecute for specific purposes
New feature CLO-11969 4.8.0-M1
MS Access driver updated, fixing some connecting issues ("TABLES: user lacks privileges or object not found")
Compatibility Improvement CLO-12162 4.8.0
MongoDBWriter: Added batch number option to Ouput and Error mappings
New CLO-12059 4.8.0
Connections to S3 now support proxy using File URL syntax for proxies
New CLO-10185 4.8.0
MongoDBWriter: Adding support for improved Bulk Write and Bulk Update in MongoDB 3.2
New feature CLO-11637 4.8.0-M1

Server Security

User lock-out can be configured after predefined number of failed attempts to log into Server Console to prevent brute-force attacks.
New feature CLO-11925 4.8.0-M1
Server's HTTP API can now be disabled to allow only HTTPS connection and prevents abuse through plain HTTP.
New feature CLO-11887 4.8.0-M1
Data Services job configured with POST method fails when explicitly calling getRequestBody() CTL function inside the job (e.g. in Mapping of GetJobInput component). This problem only appears when when data is POSTed from a web form that causes the receiving Data Service job to parse the POST body as parameters, making the body no longer available. In this case, use getRequestParameters() function instead.
Known Issue CLO-11309 4.8.0-M1

Server UI

Improved performance of Executions History view for large amounts of records
Improvement CLO-10347 4.8.0
Added a convenient search field to Users and Groups administration modules
New CLO-12091 4.8.0

Hadoop Security

Support for Kerberos authentication in Hadoop HDFS, YARN connections and Hive JDBC connection
New feature CLO-11837 4.8.0-M1

Fixes

Data Services: An error is thrown when multiple inputs are connected and response format is set to "CSV"
Compatibility Fix CLO-11936 4.8.0-M1
Fixed Server Console showing the plain values of secure parameters under certain conditions
Fix CLO-11931 4.8.0-M1
Fixed REST Job failing if there were both enabled and disabled components connected to the Output panel
Fix CLO-11399 4.8.0
Fixed event listener manual task execution params not being passed
Fix CLO-12110 4.8.0

Compatibility

Dropped support for Server on Apache Tomcat 6.x
Compatibility New CLO-11922 4.8.0-M1
Add support for Apache Tomcat 9
Compatibility New CLO-11924 4.8.0-M1
Updated Server All-in-one bundle with Tomcat 8.0.47
Compatibility New CLO-11923 4.8.0-M1
Data Services: An error is thrown when multiple inputs are connected and response format is set to "CSV"
Compatibility Fix CLO-11936 4.8.0-M1
Updated underlying smbj library (from 0.3.0 to 0.5.1)
Compatibility Improvement CLO-12123 4.8.0
MS Access driver updated, fixing some connecting issues ("TABLES: user lacks privileges or object not found")
Compatibility Improvement CLO-12162 4.8.0
December 11, 2017 Download CloverETL 4.8 4.8.0
  • For Developers (Improvements most useful for developers bringing new functionality or optimizations in data transformation and orchestration)
  • For Administrators (Improvements or features that will help setup, install, administer and manage the platform)
  • For Support (Helps staff supporting the production environment to identify and escalate potential problems or avoid such)
  • For Security (Improvements and changes relevant to security focused staff – sys admins and developers alike)