CloverETL Release Notes
Version 3.2.1
released January 05, 2012
CloverETL Designer
Fix of View Data dialog error when data contained NULL value/s
WebServiceClient name space support fixed
See CloverETL 3.2.1 at Jira Issue Tracker for full list of changes
CloverETL Server
Updated Quartz scheduler to 1.6.5 (see CLS-606)
See CloverETL 3.2.1 at Jira Issue Tracker for full list of changes
Version 3.2
released December 19, 2011
CloverETL Designer
Dynamic Edge Buffers
Transformation Editor Enhancements
Field Labels
Designer Activation
CloverETL Server
LDAP User Management
Cluster Licencing
Version 3.1.2
released October 13, 2011
CloverETL Designer
XMLWriter minor tweaks
- Schema with local imports can now be properly used
- Mapping generation from XSD no longer prefills element types as values
- XSD root selection combo default selection changed
- Drop of a field onto element and binding tweaked to be more user-friendly
- Root element can contain a binding to a port as long as only one record per file is set. A warning is reported otherwise.
Customization options for OEM partners
Changed file URLs form to be Java compliant
- formerly file:c:/example is now correctly file:/c:/example
XMLExtract mapping editor fixed bug: preserve mapping of items mapped by ../ notation
ComplexDataReader GUI fixes - Mac OS X crashes fixed
CloverETL Server
Customization options for OEM partners
Engine plugins can be easily added to the server via configuration option
- see http://server-demo-ec2.cloveretl.com/clover/docs/osgi.html
- Engine plugins library isolation, fixes clashing custom libraries with Server default libraries
MySQL DB installation script - replaced TYPE with ENGINE keyword for newer MySQL versions
Class loader issue on JBoss fixed
Version 3.1
released June 16, 2011
CloverETL Designer
New component ComplexDataReader (commercial only)
- for reading structured and unstructured flat files
- similar to MultiLevelReader, yet more powerful and user-friendly
Redesigned component XMLWriter (EXT_XML_WRITER)
- brand new mapping (incompatible)
- old XMLWriter still available as a deprecated component
- supports disk caching for large amounts of data
XML components visual redesign
- brand new XMLWriter interface with visual and source editors
XMLExtract improvements
- automatic data denormalization
- metadata creation out of XMLExtract dialog
- ability to add ports and assign metadata
- annotations
- improved error reporting
AddressDoctor component (commercial only)
Lotus Notes Domino Reader/Writer/metadata extract (commercial only)
New component EmailReader (commercial only)
Graph editor visual improvements
- handy description bar
- component autosize for component name
- improved edge handling
DnD in graph editor improved
- drag'n'drop features for dragging files to graph, outline, etc.
New CTL2 functions - xml2json() and json2xml()
Support for WebDAV and Amazon S3 protocols
DBInputTable supports reading SQL queries from input port
Java class wizard intergration into Transform Editor
View data is not modal any more
- multiple View data windows can be open at the same time
Graph zoom and scroll information is preserved
Cobol CopyBook format metadata extract wizard
Primitive sequence support (no file required)
Alt-drag note does not work as a container drag
Auto-mapping in Transform Editor is case-insensitive
Binary parsers for integer and long, double (little/big endian, packed decimal)
New component Pivot
New component MetaPivot
PervasiveDB support
Firebird is has now built-in support
Trash multiple input ports
Trash can simulate a writer (Performance/Validate records modes)
New component DataSampler
New attribute for Denormalizer - group size
Performance of CTL random* functions improved
Performance of CTL String.indexOf() improved
Fixed handling of paths with spaces
ParallelReader improvements
Performance of Concatenate improved
Truezip is now used for handling zip files (append, excel)
CloverETL Server
WebDAV API
Web GUI: manual task execution
- tasks now can be executed manually instead just by the scheduler
Support for JBoss 5.1
Digest HTTP Authentication
SOAP WS API added to the manual
"graph version" attribute added to the executions history
Launch services requests are logged to the "[tmp]/cloverlogs/launch" by default
Fix: NPE is not thrown when the specified launch service is missing
debug_mode config parameter fixed
better tracking data persistence
Version 3.0.2
released March 30, 2011
CloverETL Designer
CTL2: Lists and maps of records are now possible.
Now you can use list of records and maps of records in your code. Example: // lists MyDataRecord[] myList; MyDataRecord a; MyDataRecord b; myList[2] = a; b = myList[2]; // maps map[string, MyDataRecord] myMap; myMap["hello"] = a; (For maps, only values can be records)
XMLExtract now distinguishes between null and empty strings
When reading XML files with XML Extract, you can encouter the need to know whether an element or attribute is present in the source
or just has an empty value.
XMLExtract now sets either "null" value whenever the element/attribute is omitted or empty value ("")
when the element/attribute is present but without a value
Aggregate: avg() and sum() aggregate functions now perform integer overflow check
Now you can easily see whether the aggration function produce the integer overflow and thus you're not getting the right results.
Fix: DataIntersection correctly reports unsorted input now
DataIntersection now stops immediately when it sees the input is not sorted properly ensuring data integrity.
UniversalDataReader & Writer now support "Quote character" attribute for specifying
Quoting support in readers/writers has been around for long time but now you can even specify the quoting character. In previous CloverETL versions there has been built-in quote defined as either `'` or `"` (quote or double quote). Now we present more fine-grained setting where the quote character can be defined.
CTL2: byte and cbyte data types can now be compared with == or != interchangeably
CTL1 and CTL2: Huge performance improvement in index_of() function
New DataSampling example project in engine examples
Updated built-in MySQL JDBC driver to mysql-connector-java-5.1.15
EmailSender: MessageBody can now be multiline
Extract metadata from database can now handle unknown JDBC data types
Fix in DBOutputTable when working with byte values in batch mode
Fixed Partition transform conversion to Java
DBFReader explicit "strict" data policy setting is now correct
XLSWriter now creates empty files correctly
Fixed Denormalizer issue when loading Denormalize transform specified by class name
Fixed DBExecute not working when calling stored procedure and connection defined by JNDI
Fixed SystemExecute running Windows PowerShell scripts
Fixed NPE when linking a lookup table
CloverETL Server
Upgraded facelets lib from 1.1.14 to jsf-facelets-1.1.15.B1.jar because of the JBoss 6.0.0
Removed some jars from the WAR file for JBoss
Added element into persistence.xml (because of JBoss)
Version 3.0.1
released November 15, 2010
CloverETL Designer
Default locale can be specified in defaultProperties
Fixed opening of the fact table loader wizard
Fixed XML mapping dialog breaking the mapping with skipped elements
Updated Oracle built-in driver class
Fixed annoying delays and hang-ups when working with SQL in Oracle
Components in palette are now sorted in alphabetical order
Fixed Transform Dialog issues when working with faulty lookups
Filter in SQL editor fixed
Import metadata from XSD fixed for names not conforming to Clover conventions
Tracking view incorrect CPU load fixed
Regular expression tester fix for incorrect regexps
error ports of UniversalDataReader and LDAP Reader can now have also byte/cbyte fields instead of just strings
byte and cbyte types in CTL2 are now fully compatible
fixed XMLExtract failing on opening input files in multiple phases
recursive parameter definition is now properly detected and warned in the log
CTL2 "for" cycle consuming memory fixed in CTL2 interpreted mode
ByteBufferUtils.encodeLength(ByteBuffer, int) fixed for 5-byte lengths
CloverETL Server
fixed NPE in HTTP API exec history #5061
added JDBC info to web GUI monitoring section
fix of schedules installation and initialization
Changes in tracking data: "nodeName" now contains node name and new attribute "nodeId" contains node ID. In addition "totalRecords" is of long type now.
all listener sections in web GUI are grouped in one "event listener" section group
Version 3
released August 24, 2010
CloverETL Designer
Brand new transformation language – CTL2
- Introducing next generation of CTL (CloverETL Transformation Languge)
- Typed languge, close to Java, can be transparently converted to Java
- up to 30% speed increase
Dictionary support in the Designer
- Dictionary can be declared in the Designer
New format for boolean data field - e.g. /man|m/female|f/
Fact Table Loader wizard
New attribute 'quotedStrings' on DataWriter component - all written data can be quoted
CSV compliant flat file reading/writing
HTTP authentication support in HTTPConnector component
Extracting metadata from DBF files
Bundled examples with easy setup
Improved edge drawing for complex graphs
Default locale can be specified globally for the environment
FTP wildcards support added
Dictionary data types support enhanced to all CloverETL data types
Optimized resource allocation in CloverETL Server/Designer
Hundreds of minor functional and usability improvements
new Dictinary types: integer,long,number,decimal,boolean,date,byte
new RunGraph component attribute "paramsToPass"
dedicated graph property name "PROJECT_DIR" (absolute path to project home directory) was substituted by attribute at GraphRuntimeContext.contextURL
- new command line attribute -contexturl <path> should be used instead of -P:PROJECT_DIR="<path>"
equalNulls attribute of Merge component was removed - value of this attribute was fixed on 'true' value
list of components in tracking log (in console) are topologically sorted
new attributes for SystemExecute component - workersTimeout, charset
new attribute 'Collator sensitivity' on DataRecordMetadata
Joda date formatter is now default formatter (java formatter is still available with prefix "java:" in format string)
CloverETL Server
Distributed graph execution
A graph can now run on multiple nodes in a cluster
Introducing Local and distributed sandboxes
File event listener enhanced file name pattern
JMS message listeners
Groovy event listeners
Refactor of send JMS msg task; jms API is removed from the WAR file
license parameter for sandboxes management
graph parameters SANDBOX_CODE_PROPERTY_KEY, SANDBOX_ROOT_PROPERTY_KEY, GRAPH_FILE_PROPERTY_KEY renamed to proper names SANDBOX_CODE, SANDBOX_ROOT, GRAPH_FILE
sandbox file is downloadable on url /clover/downloadFile?sandbox=[sandboxCode]\&file=[filePathInSandbox] (requires BASIC HTTP Authentication)
OSGi framework embedded in the server, so CloverETL Server APIs (i.e. some specific WebServices) may be implemented as OSGi bundles
HTTP API operations executions_history and graph_status may return detail status data in XML
fix of passing event parameters to the graph (see description of back-port in the section REL 2-9-5 for more details)
Executions enqueuing enabled/disabled switch
Archivator reliability improved
HTTP API executions_history operation - ordering parameters
Task Archivator has new option - "Include dictionary files" - all dictionary temporary files older than specified timestamp will be removed
swaref standard for binary data transmission in our webservice interface was substituted by mtom standard
- backward compatibility was broken
tracking data are stored in table run_record in binary serialized format for new records; tracking_* tables keep in DB and contain tracking data for older records
verbose mode is off by default
Version 2.9.5
released August 23, 2010
CloverETL Designer
Added support for custom extension point to add extra VM arguments to graph run configuration
CloverETL Server
passing parameters between graphs using "graph event listener" feature; back-port of issue #4907 from the branch 3.0;
- Older versions passed all source graph parameters which overrided parameters in executed graph.
- Since version 2-9-5 server passes only parameters explicitly specified in the "execute graph" task. Please see the section "Execute Graph Task" in server manual for details.
It's possible to enable old-fashion behaviour by server config property "graph.pass_event_params_to_graph_in_old_style".
Version 2.9.4
released July 22, 2010
CloverETL Server
Added support for IBM Java
Version 2.9.3
released June 03, 2010
CloverETL Designer
Special characters in external transformations are no longer expanded
Fix of #4252 RunGraph component - GraphNotFound on the server
Fixed incorrect resolving of special characters in transforms specified by URL
Reset fix in InfobrightDataWriter
CloverETL Server
fix of concurrent graphs which end with error status
Version 2.9.2
released April 22, 2010
CloverETL Designer
Multiple external elements (metadata, connections, etc.) can now be linked by multiselect in file dialog
SQL-2-Clover and vice versa type conversions optimized
SQLite support improved
added record type in Variables tab of Transform Editor
added "parent" and "root" buttons to file dialog
double-dash comments in SQL are now possible
Invalid metadata in extract from flat file can now be save on user request
Example projects using DB migrated to Derby - easy setup for newbies
CloverDataReader is now able to read without index file
Fixed Oracle-specific crash with specific driver-level Java access
Fixed slow DB extract metadata for large table
CTL editor fixed - rarely break CTL code syntax
Mac specific GUI fixes (font size, shortcut keys)
FastSort NPE fix with non-nullable record and no default value
fixed XLSWriter long sheet names issue
fixed occasional NPE in DataIntersection
fixed DB2DataWriter not finishing in pipe mode
fixed ExtSort sorting composed key with nulls incorrectly
fixed fields delimited by "\n" - doubling backslash
fixed Reading timestamps from XLS sheets
CloverETL Server
Added support for IBM WebSphere 7
Upgrade of Spring to 2.5.6
Upgrade of RichFaces to 3.3.2
Changed namespaces and names of WebServices API - causes incompatibility with older versions
fix: Server time label
fix: Web GUI task log filtering
fix: NPE in JMX API killGraph method
fix: HTTP API execution status
graph parameters SANDBOX_CODE_PROPERTY_KEY, SANDBOX_ROOT_PROPERTY_KEY, GRAPH_FILE_PROPERTY_KEY renamed to proper names SANDBOX_CODE, SANDBOX_ROOT, GRAPH_FILE
fix: NPE when launch service's graph has component without ports
Version 2.9.1
released February 19, 2010
CloverETL Designer
File dialog improvements
Browsing of Server projects on separate tab
improved SFTP performance
Linking of external parameter files fixed
Occasional freeze on Mac OS fixed
JMS Writer - Fix of exception propagation during sending of JMS message
fixed HashPartition function giving negative port numbers occasionally
Version 2.9
released January 29, 2010
CloverETL Designer
New component for connecting to Infobright database
New component for connecting to Web Services (WebServiceClient)
XLS reading - formatted value can be obtained as a String type field if needed
Tracking - in final tracking summary average Rec/s and KB/s is displayed
LDAPWriter - possibility of changing/disabling multiseparator
CTL transformation has now PROJECT_DIR available
new AbstractPartitionFunction for Java writen PartitionFunction
Parameter files can be located outside project
Completely redesigned File URL dialog for better user experience
Parsing of NULL values from flat file by specifying custom "null" keyword
Improved User's Guide
Quick Start Guide
Aspell lookup threshold setting fine tuning
Eclipse branding for bundle version
Path to JAR in JMS connection definition now supports parameters
Export: image path in HTML file fixed
missing column names when browsing Oracle database
external metadata from outside project can be now used
PostgreSqlDataWriter and OracleDataWriter - NullPointerException fix
Reading of zipped MS Excel files
CloverETL Server
support for Jetty servlet container
fix HTTP API operation graph_status complains 'RunRecord is null'
Version 2.8.1
released October 21, 2009
CloverETL Designer
DBInputTable - incremental file is created automatically when properly configured
AspellLookupTable can now return edit distance
DBOutputTable has new attribute "Atomic SQL query" for ensuring that SQL query rollbacks on error
SQL query in DBExecute can have comments inside
Bundled jTDS for direct support for Microsoft SQL Server and Sybase
Support for SQLite database browsing
Drivers for Microsoft SQL Server and Sybase are bundled with Designers and readily available
Dynamic default value hints display current values based on graph settings
JDBC driver can be specified without loading the JAR - thus allowing the JAR to be on classpath
Unified use of Clover file URL dialog
Minor usability fixes in Server integration (Designer Pro)
Parameter file linking in Server project
Fixed occasional exceptions when loading external lookup tables
Fixed CPU info tracking for multithreaded components
Fixed special characters resolving in some URL attributes
XMLWriter can be created from Java
Minor fixes in InformixDataWriter
CloverETL Server
fix of tracking graph persistence
fix of thread synchronization during graph abortion
some spelling changes in web GUI
Version 2.8
released August 21, 2009
CloverETL Designer
CloverETL Server Integration - connect to Server (Designer Pro only)
- CloverETL Designer
Unified versioning of CloverETL Engine and Designer (starting with 2.8.0)
Method DataRecord2JmsMsg.createLastMsg(DataRecord) is deprecated. Use the same method without parameters instead.
org.jetel.component.jms.JmsMsg2DataRecordProperties can process javax.jms.BytesMessage
bodyField attribute of JmsReader/JmsWriter have default value
interfaces DataRecord2JmsMsg and JmsMsg2DataRecord now have 2 new methods: setGraph(TransformationGraph) and getGraph()
context(thread) class-loader changed to current(class) class-loader
new run option: -skipcheckconfig
optimized UniversalDataReader
Trash now can print out its ID for debugging purposes
unified null and empty field values in sorting, Dedup, etc.
Designer has two versions: Designer and Designer Pro
CloverETL Server
fixed JMX notifications in cluster
cluster node mutually diagnostics
JNDI DB DataSource usable for server DB connection
simple HTTP API operation graph_run accepts params which are passed to executed graph as additional properties
launch services API optionaly passes URL parameters to graph as additional properties
refactor of Graph Config properties + additional parameters