history.txt

实现了一个基本的ETL

components can now easily be added through modifying this file and specifying component's
ID and class name.
+when defining graph layout and parameters for components in form of XML file, component parameters
previously accepted only as XML nodes' attributes can be now defined in form of "child" nodes too.
This is useful for long definitions or multiline parameters.


Changes:
+information messages & debugging messages are now output through
Jakarta's CommonsLogging means - they are not sent directly to console any more.
+fixed DelimitedDataFormatter - to properly handle OneRecordPerLine policy on different
OS (CR&LF issue on Windows, Unix)
+fixed skipLeadingBlanks at FixLenDataParser
+refactored interface org.jetel.data.Number to org.jetel.data.Numeric - Number collided with
java.lang.Number class
+Reformat component now supports multiple output ports connected. Appropriate transformation
class has to be provided by user.
+Dedup component now offers selecting unique only data records - those records for which
there were no duplicates found.


REL 1-1-6
---------
New features:
+Sequences - definition of interface and implementation of simple sequence with
persistent value.
+Parameters defined (parameter files referenced) from within graph's XML definition 
+Long datatype - represents signed 64bit integer
+Number interface - interface with methods to get/set/compare int,double,long values of
data fields representing numeric value
+XML definition of graph's layout can now contain definition of properties (parameters).
They may be defined directly in graph or loaded-in from external files - see JavaDoc
for TransformationGraphXMLReaderWriter class.

Changes:
+DBInputTable now allows to specify fetchSize when reading data from database
+Filter now accepts (copies to output port 0) all records for which the filterExpression evaluates TRUE
(the same behaviour as ExtFilter)
+DBOutputTable - if maxErrors parameter is set to -1, all SQL errors are ignored.
+DataRecord now implements Comparable interface. However, compareTo() method can be used
only to compare records based on the same metadata object
+Removed package org.jetel.test from clover's engine and put into clover examples package
+Removed clover org.jetel.gui - long abandoned code
+runGraph() method is now covered by LGPL license

REL 1-1-5
---------
New features:
+DBInputTable,DBOutputTable,DBExecute now support loading SQL queries
from external file - use "url" attribute. File can be accessed locally or
through FTP or HTTP protocol - whatever is specified in "url".
+RecordTransform interface now includes "finished()" method. Called
when particular Node (e.g. Reformat, MergeJoin,..)finishes transforming data.
+DataRecordTransform class has been updated to reflect changes in RecordTransform
interface.

REL 1-1-5pre
----------
New features:
+Partition component - partition data using one of following algorithms -
(Round-Robin,Hash,Range)
+DataIntersect component - joins two data flows (A & B) and outputs records only
in A to port 0, records both in A&B to port 1 and records in B only to port 2.
+extra XML attributes defined for components performing reformats (Reformat, HasJoin, MergeJoin)
are now passed into transformation function as properties - can be used for parameterizing
transformation functions


Changes:
+enhanced MERGE_JOIN component to support full outer join.
+introduced code which detect potential deadlocks between components in graph and changes
direct-edge to buffered-edge.
+transformation graph now registers all DataRecord metadata defined (by their IDs) and offers
later retrieval of such metadata - good when using lookups which require metadata.
+SQLDataParser supports (again) setting of fetch size, also improved reporting of possible
parsing errors
+changed definition of LookupTable interface, also updated SimpleLookupTable & DBLookupTable
+changed RecordTransform interface
+DataRecordTransform helper class has been updated to reflect changes in RecordTransform interface

REL 1-1-4
------------
The same functionality as 1-1-4pre1.

REL 1-1-4pre1
-------------
New features:
+ new ExtSort component has been added. It performs external sorting (on-disk sorting)
if input data file is too big to fit into available memory.
Changes:
+ added/extended toString() methods to DataRecord,DataRecordMetadata
+ changed size specifier when (de)serializing StringDataField from
int to short -> saves two bytes on each serialized field which speeds up
processing. However, poses the limit that string can be max 32K chars long.
+ added duplicate() method to DataRecord and all field classes. It
makes duplicating (making copies) of existing records easier and much faster
(it doesn't perform several checkings which are not necessary in this context).
Creates new object.
+ added copyFrom() method to DataRecord and all field classes. Enables faster
copying of content of one record into another. Does not create new object.


REL 1-1-3a
----------
Changes:
+ BufferedEdge has been rolled back to version 1.1.1 since the version
from rel 1.1.3 turned out to be buggy.
+ DirectEdge has been reworked and tuned. It should be also better tested now.

REL 1-1-3
---------
New features:
+ extended metadata of locale support. Data & Numeric fields
now can have locale specified. It can be used alone or combined
with format string to specify how the value is going to be formated
when printed as String or what is the expected format of the data in text form
(for parsing).
+ DBOutputTable sends records rejected by DB to output port 0 (if connected). This
can be used for capturing rejected records. Does not work well in Batch insert/update
mode.
+ Metadata describing data flows can now be directly embedded into graph definition
file. 
+ When specifying where to look for file containing metadata in XML 
form, fileURL can now reference HTTP or FTP server (previously only FILE was accepted). 
This allows for centralized metadata stored on central server.
+ Metadata can now be extracted directly from database based on SQL query. This
allows for dynamic unloading of database tables without the need to create
metadata definition file upfront.
+ ExtFilter's expression language has been extended by datediff() function.

Changes:
+Updates to fatal error reporting of org.jetel.main.runGraph utility. Unless verbose
mode is specified, only error message is displayed without stack trace info.

REL 1-1-2
---------
New features:
+ extended ExtFilter - org.jetel.interpreter has been
extended, it now supports arithmetic operations and
has several built-in functions.

Changes:
+ added several new "macros" to org.jetel.data.GetVal

REL 1-1-1
---------
New features:
+ added DBF data/database reader. It comes with DBFAnalyze utility
which reads in DBF table and produces Clover metadata describing the
record structure. See examples (graphDBFRead) for illustration how it works.
+ created ExtFilter component which somehow resembles "old" Filter
with several enhancements (supports logical operators AND,OR,NOT,
braces, etc.)
+ created org.jetel.interpreter package - currently, code
supporting ExtFilter is there. This should become base
for Clover's internal transformation language interpreter.
+ for all data fields types, own hashCode() method implemented.

Changes:
+ All components which work on stream of records now contain yield() call (within
their main processing loop. This should improve thread scheduling and consequently
help improve data flow throughout the graph.
+ introduced checking of name validity for Nodes (all components), Edges,
Records, Fields. Allowed characters are [A-Za-z0-9_]. This is due to
planned future enhancements. Up to this version, was no particular constraint.
+ rearranged/refactored packages & classes. All data parsers (for textual data)
are now int org.jetel.data.parser. All data formatters (for textual data) are in
org.jetel.data.formatter. Also lookups have been moved to separate package -
org.jetel.data.lookup.
+ Extended interface definition RecordTransform to support variety of transformation
components. Use DataRecordTransform (in org.jetel.component package) class as a base
class for your custom transformations.
It allows you to omit implementation of many methods from RecordTransform interface
which are not used/called most of the time but have to be provided.
+ created new package org.jetel.data.lookup - base for consolidating
lookup tables support (old lookup tables have been put there). New
interface LookupTable has been created.
+ updated compareTo() methods for all data objects (NumericDataField,
StringDataField, etc.) to support comparison with various object types.
+ FixLenDataParser2 has been updated to support line separators (when fix-len
record is delimited by new-line character(s))
+ components FixLenDataReaderNIO,FixLenDataWriterNIO have been enhanced
to allow specifying of which character(s) is used to separate
record on line - if required.


REL 1-1-0
---------
New features:
+ change of license from GPL to LGPL
+ enhanced DBOutputTable to allow for hand-written SQL/DML statements - this
should give 100% flexibility when populating target DB table
+ improved CopySQL primitives to cope with basic datatype conversions - integer->decimal,
date->datetime, etc. - it regards to Clover-DB interaction
+ graph layouts (component attributes) can now reference globally defined
properties (through ${...property name...} ). These references are resolved
when materializing components/metadata.
+ added handling of NULLs when copying data to/from Database 
for fields (String, Date, Integer, Numeric);
+ for components requiring transformation classes to be specified (e.g. Reformat,Joins..),
the possibility to embed the Java source into XML text defining graph layout is possible. The source
code gets automatically compiled. When the graph is executed multiple times, it is
checked whether the source code changed and if not, compilation is skipped.

Bugs corrected:
+ fixed bug in CopyDate class (CopySQLData) when copying/setting DATE filed - thanks to Khelfa Mohamed


REL 1-0-8
---------
New features:
+ updated DBOutputTable component to support selective source2target fields
mapping - attribute cloverFields added.
+ updated DBOutputTable to support batch updates
+ updated Sort component to support lexicographics sorting & normal sorting
+ created new component - HashJoin.
+ created new class HashKey (used for building hash tables of DataRecords)
+ created new class RecordComparator to be used for sorting DataRecords
+ added Java Properties for DataRecordMetadata and DataFieldMetadata
+ new component CheckForeignKey added - contributed by Stewart Jeacocke
+ altered Filter component to send rejected records to second output port
  (if connected)
+ updated EdgeDebuger to support debugging limited number of records, filtering and sampling

Bugs corrected:
+ fixed problem in DataRecordMetadataXMLReaderWriter - part writing XML was wrongly
outputting format string for defaultValue attribute.