CloverETL 3.5.0-M2

Everything you need to keep your data integration healthy

We'd like to share more of our progress with you. Lately we've been working on something that will be a big part of our next production version of CloverETL, set for release at the end of the year.

In this CloverETL milestone, we're offering users a test-drive of a new package – CloverETL Data Quality. With CloverETL Data Quality, we've bundled time-proven tools and practices for data quality into a complete package that you can use to profile and check the quality of, and validate your data with clear visibility into each step as with well documented rules and outputs.

As usual, this release also brings additional tweaks and features. We've finalized a few of the improvement tracks that were started in the first milestone of 3.5 release. You'll find you have better insight into Server activity with improved Monitoring and simplified product deployment, such as simple license management, a bundled Server download package, and more granular import/export configuration options to migrate between Server instances.


Released Oct 31, 2013
  • Milestone
Recent Releases
4.7.0 Oct 10, 2017
4.6.1 July 19, 2017
4.5.0 Mar 14, 2017
4.4.1 Feb 14, 2016
4.3.1 Nov 10, 2016
4.2.1 Aug 18, 2016
4.1.4 Jun 1, 2016

See all releases

Interested in trying it out? Learn how to get started or sign in to download.

Release 3.5.0-M2 Contents

  • Major New Feature

CloverETL Data Quality

Integrated toolset for Data Quality

Discover Potential Issues

The goal of the CloverETL Data Quality package is to meet the need to understand and monitor the quality of your data. Data Quality starts with discovery – an ability to get quick insight into potential quality issues with a dataset. To see patterns, distribution of missing fields, incorrect data types, and more. This insight helps you decide how to further handle the dataset. Do you have really unclean data, calling for a rigid workflow with manual correction steps? Or can an occasional error here and there be fixed automatically?

Put Data Quality To Work

Once you have identified potential issues, the tools in the CloverETL Data Quality package will help you analyze and validate data in the transformations. With the newly introduced Validator, you can visually define data quality rules and apply them within your workflow easily. Adding the power of jobflows to the mix, you can fully embrace a powerful data quality strategy using CloverETL.

Please note the CloverETL Data Quality package will ultimately replace our Data Profiler extension. With the Data Profiler as part of the package, users will benefit from more comprehensive Data Quality tools.

Components in the Data Quality package:

  • Major New Feature

Data Quality; Introducing Validator

Visual rule-based data filtering. Replace coding with visual business rules designer.
  • New Feature

Encrypted Parameters

Store passwords securely; meet security policies
  • New Feature

JSONExtract

Visual mapping and streamed reading for JSON
  • New Feature
  • Server

Server Status Monitoring

Instant awareness of Server status

We are enhancing the Monitoring feature of the Server to give you better visibility into the current state of the Server. It helps you analyze what's happening in order to better optimize and safeguard against potential performance issues before they happen.

The redesigned dashboard brings a new performance chart that puts memory consumption and running jobs into a single perspective in real time.

  • Feature
  • Server

Configuration Migration

Enables automation for migration between Server instances
  • Improvement
  • Server

Server Usability Improvements

Things that make your life easier

Additional Features & Fixes

Features

  • Designer
  • New feature
External CTL files are now supported – Introducing standalone editor for CTL files. Full validation and context is available except component related information like edges connected and metadata declarations. (CLO-1731)
  • Designer
  • Improvement
CTL editor has better context-assist (data types, local variables) (CLO-2199)
  • Designer
  • Improvement
Improved graph export to HTML - components are interlinked, graph image is an active image map with links
  • Engine
  • CTL2
  • New
New function round(number, precision) (CLO-1812)
  • Engine
  • New
EXASolution connectivity (CLO-1591)
  • Engine
  • New
HTTPConnector has a new option "Stream input file" for better performance (CLO-1937)
  • Engine
  • New
HTTPConnector has new mapping field "requestContentByte" and "contentByte", adding byte support on top of the existing string variants. (CLO-1032)
  • Engine
  • New
New attribute "Sorted input" on writers that support partitioning by key. This optimizes performance as files for keys in order can be closed immediately after writing. (CLO-812)
  • Designer
  • New
Duplicate external metadata (creates an internal copy) (CLO-1114)

Fixes

  • Engine
  • fix
Fix-length file parsing fixed when metadata contains autofilling fields as last fields
  • Engine
  • fix
JavaBeanReader performance improved by using "nodeName" direct access rather than XPath
  • Engine
  • fix
SpreadsheetDataReader mapping dialog was not opening for corrupt style information in Excel files. (java.lang.IndexOutOfBoundsException fixed)
  • Engine
  • fix
Fixed file operations paths: relative vs. absolute path for remote location – see attached issue for explanation (CLO-1509)
  • Engine
  • fix
Function removeNonPrintable() no longer removes non-ASCII accented characters (e.g. chars ä, ö, etc.) (CLO-1814)
  • Engine
  • fix
Fixed JSONReader: fields containing underscores were not populated in implicit mapping (CLO-1015)
  • Designer
  • fix
HTML graph export – fixed performance and star ($out.0.* = $in.0.*) mapping redering (CLO-1449)
  • Designer
  • fix
Removed limitation for "maximum cache size" in the lookup dialog (CLO-1037)
  • Server
  • fix
Fixed Server not being able to process a task (in a listener or scheduled) without "clustering" enabled (CLO-1658)

Known Issues & Compatibility

  • Jobflows
  • Server
  • Compatibility
CloverDataReader/Writer data files not compatible between version 3.4 and 3.5 when these components are used in a mix of jobflows and transformation graphs. Read comments in related issue. (CLO-1382)
  • Engine
  • CTL2
  • Compatibility
Functions ceil() and floor() return decimal instead of number for parameters of type decimal (CLO-2005)
  • Engine
  • CTL2
  • Compatibility
Some conversion functions return NULL instead of throwing an exception (CLO-1586)
  • Engine
  • CTL2
  • Compatibility
Some date functions return NULL instead of throwing an exception (CLO-1584)
  • Engine
  • CTL2
  • Compatibility
Null 'case' values in switch() are now allowed (CLO-737)
  • Engine
  • CTL2
  • Compatibility
Function get() for lookups now always returns NULL for keys not found or NULL key, exception if unknown field is requested. Before, there was a difference between compiled and interpreted mode - returning null or exception respectively. (CLO-1582)
  • Engine
  • Compatibility
Fixlen data reading changed in regards to automatic trimming. Read the related issue. (CLO-1405)
  • Engine
  • Compatibility
Built-in MySQL JDBC driver updated to version 5.1.26 (formerly 5.1.22). The new version is optimized for MySQL 5.6. (CLO-1886)
  • Engine
  • Compatibility
FastSort defaults have been changed. Maximum open files is now 1000 by default, Number of sorting threads is 1. (CLO-1775)
  • Designer
  • Installation
Old version of Designer on Windows is automatically uninstalled when new version is installed (only versions 3.5.0-M2 and newer are uninstalled) (CLO-1028)