You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 44 Next »

 

Version 1.0.0-Beta

Version 1.0.0-Beta of Apache NiFi is huge release that includes a brand new UI, framework level improvements and much more! Note that this release is geared towards testers and developers outside the project, and may not be stable in terms of features or functionality.

There are many changes in the 1.0.0-Beta release but the Application level highlights include:

  • UI Refresh
    • The entire UI has been redone to have a cleaner, more modern look
    • UI has also been reworked to promote a better User Expierence
  • Zero master clustering
    • There is no longer a NiFi Cluster Manager (NCM)
    • Cluster will now auto elect a Cluster Coordinator to oversee the cluster
  • Multi-tenant authorization and internal authorization/policy management
    • There is no longer just the limited Administrator, Data Flow Manager, etc. roles
    • Policies are fully customizable for each component and at the controller level as well
  • Deterministic template export
    • Templates export is now deterministic and thus can now be version controlled

A full list of issues that were resolved can be found at: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12338066

Version 0.7.0

Version 0.7.0 of Apache NiFi adds several valuable Processors, a new developer focused doc, and continued improvement to usability and stability.

Release Date: July 12, 2016

Highlights of the 0.7.0 release include:

  • New Application Features
    • NiFi now supports exposing key/value pairs set in "nifi.properties" to Expression Language.
  • Improvements to Existing Capabilities
    • Added the ability to bulk using Index or Update to PutElasticSearch.
    • Allow concurrent execution of ExecuteScript.
    • Expression language now has a random number function.
  • New Extensions
    • New processors to Get data from SNMP agents and execute SNMP set requests.
    • Added new processor to Put to Slack.
    • New processors to Get and Put data to an MQTT broker.
    • Added processor to listen for messages using the Lumberjack protocol.
    • New ExtractMediaMetadata processor to extract metadata from various file types using Apache Tika.
    • New processors to Delete, Put and Get AWS DynamoDB.
    • Added processor to poll notification events provided by HDFS Inotify interface.
    • Add ReportingTask for sending Provenance events over Site-To-Site.
  • Important Stability Improvements
    • Fixed instance where archiver was not respecting nifi.content.repository.archive.max.usage.percentage.
    • Resolved issue where incoming connection could be deleted while processor was still running.
    • Fixed issue where FileSystemRepository could leave open file handles to data that had already been archived and/or deleted.
  • New Documentation
    • A new NiFi in Depth document was created to give an in-depth, developer level, look at NiFi and its design decisions.

A full list of issues that were resolved can be found at https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12335078

Version 0.6.1

Version 0.6.1 of Apache NiFi focuses on addressing defects.

Release Date: April 18, 2016

Highlights of the 0.6.1 release include:

  • Improvement to existing capabilities
    • Exposed property to control fetch size when interacting with databases in the 'QueryDatabaseTable' processor.  Previously it just defaulted to one and would be innefficient.
  • Defects addressed
    • Corrected an issue where high-rate bulletin generation caused more than the last five bulletins to be retained which wasted memory.
    • Corrected logic in GetKafka that resulted in leaking connections to Zookeeper ultimately leading to node instability.
    • Corrected PutKafka logic where incorrect message demarcation could result in data loss during transfer.
    • Simplified validation logic for Hadoop and HBase processors that led to excessive and expensive validation calls that caused sluggish UI behavior.
    • Resolved an edge condition which could cause the content archival to not expire data properly resulting in full content repositories and stopping the flow.

A full list of issues that were resolved can be found at https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12335496

Version 0.6.0

Version 0.6.0 of Apache NiFi provides several new features and addresses multiple bugs or improvements.

Release Date: March 26, 2016

Highlights of the 0.6.0 release include:

  • New Application Features
    • NiFi's REST API can now support Kerberos Authentication while running in an Oracle JVM.
  • Improvements to Existing Capabilities
    • ListenRELP and ListenSyslog now alert when the internal queue is full.  This means data receipt exceeds consumption rates as configured and data loss might occur so it is good to alert the user.
    • ListHDFS will now retain simple and finite state rather than an enumeration of all files seen with the same timestamp.
    • ListenSyslog, ListenRELP, ListenUDP have vastly improved performance and simpler configuration.
    • Updated Spark Receiver to support Apache Spark 1.6
    • Interaction with HBase now supports Kerberos enabled Apache HBase.
    • Fixed an issue in ExecuteSQL that caused SchemaParse failures against multiple databases.
  • New Extensions
    • New processor 'QueryDatabaseTable' monitors and tracks timestamp of latest record retrieved to support simple change capture cases.
    • Added a new processor that can be bootstrapped by a Spring Application Context.  Could enable someone to run Apache Camel routes or Spring Integrations flows in NIFi for example.
    • Added new processor to Put data with Amazon Kinesis Firehose
    • New processor to put data to Amazon Web Services Lambda service.
    • New processors to Put data to and Get data from Splunk.
    • New processors to Get and Put data to Apache Cassandra
  • Important Stability Improvements
    • Resolved corner case that could lead to corruption of flow file repository during shutdown.
    • Resolved a case that lead NiFi to hold open too many files when using run durations longer than 0 ms.
    • Resolved flow.xml.gz ordering issue that led to upgrade instability for clusters.

A full list of issues that were resolved can be found at https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12334372

Known Issues with this release:

  • NIFI-1701 - Getting issue details... STATUS :PutKafka error handling and logic far better but poor stream demarcation can lead to data loss.

Version 0.5.1

Version 0.5.1 of Apache NiFi addresses several bugs and issues.

Release Date: February 26, 2016

Highlights of the 0.5.1 release includes:

  • Highlights of Bugs addressed
    • Close a case that could lead to data loss: In the event that all flow files assigned to a given resource claim were swapped out, and the archive had to reclaim old space or was not engaged, and NiFi was restarted resource claims could be removed resulting in data loss.
    • LDAP based authorization blocking valid API calls: When using LDAP based authentication there were times when the access token wasn't being propagated which resulted in some valid API calls being rejected such as when downloading content or templates.
    • Improved error messaging: Several processors include improved error messages
    • Hadoop processor bug fixed: Issues with multiple PutHDFS processors fixed and client library dependency problems
    • Clustering restart and upgrade reliability: A long-standing issue which caused problems when a cluster was restarted or upgraded was fixed.
       

A full list of issues that were resolved can be found at https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12334887

Version 0.5.0

Apache NiFi 0.5.0 includes several exciting new capabilities to include new processors, exciting new developer tools for building and testing processors. Stability and performance continue to be a priority and a long list of bugs were identified and resolved!

Release Date: February 16, 2016

Highlights of the 0.5.0 release includes:

  • New Application Features
    • Data inspection: We've greatly enhanced the ability to interact with data and inspect it as it is flowing through NiFi. 
  • Things to make NiFi Development Better and Easier
    • State Management: As many developers are aware, keeping state in a Processor was often "up to you".  This extension to the framework addresses this by adding state management as a core feature. In addition, many existing processors which kept state were modified to take advantage of this new capability. 
    • Testing Improvements: We've improved the performance of many unit tests,  added support for groovy for unit tests, the ability to test in unit tests, as well as better support for integration testing 
  • Improvements to Existing Capabilities
    • RELP Support for Syslog: In addition to adding support for RELP (Reliable Event Logging Protocol) for transporting syslog messages, emphasis was put on improving the framework to enable even more extensions in future releases. 
    • S3 Improvements: We've pushed past the 5G max upload limitation of the previous PutS3Object processor by adding Multipart Upload support. Additionally, we added a couple authorization capabilities, like expression language support for keys, and an exciting new controller service created initially for fetching from buckets access across accounts.
    • Hive Support: The Kite bundle now supports sending data directly into Hive tables
    • Encryption: A broad set of enhancements were made to improve encryption and decryption of content and associated user documentation.
  • New Extensions
    • Script Execution: We've dramatically enhanced the ability to operate on data flowing through NiFi by adding support for launching scripts in a flow. We added a broad set of scripting languages - you can launch JRuby, Groovy, JavaScript, Lua, or Jython scripts with access to FlowFile data, which will allow much more dynamism and quick reaction capability in your data flows.
    • Support for sending Riemann Events: NiFi now supports sending events to the Riemann event stream processing system. 
    • Elasticsearch: Connectivity to the popular search engine was added to this release
    • Avro Schema Inference: Support was added allowing the inference of an Avro schema from JSON and CSV data flowing through NiFi.
    • AMQP support: NiFi now supports sending and receiving from AMQP (Advanced Message Queueing Protocol) based messaging systems

A full list of issues that were resolved can be found at https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12334158

Known Issues with this release:

  • NIFI-1497 - Getting issue details... STATUS : An issue with Access Tokens may cause Viewing Content and Custom UIs to fail
  • NIFI-1527 - Getting issue details... STATUS : An issue that under certain conditions can result in data loss
  • NIFI-1694 - Getting issue details... STATUS : Users upgrading from NiFi ≤ 0.4.1 may encounter errors in PGP key-based encryption/decryption with EncryptContent processors if they have provided an individual key file for the public keyring file or secret keyring file property. A valid keyring file must be provided, and the full userID "Name (Comment) <Email>" form should be provided. 

Version 0.4.1

Version 0.4.1 of Apache NiFi is an incremental release addressing several bugs and providing a few minor improvements over the 0.4.0 release.

Release Date: December 22, 2015

Highlights of 0.4.1 release include:

  • Bugs addressed
    • Site-To-Site: If the remote system was applying back-pressure and the sending system attempted to stop the connection the sending system flow configuration could hang.  If one of the nodes in a cluster being delivered to went off-line it could cause site-to-site to stop delivering to the other nodes in certain conditions.  The automatic account request mechanism was broken.
    • Flow File Ordering: Ordering based on timestamp of flow file entry has been augmented with a sequence counter to provide better ordering precision.
    • Run duration: Only shown now when the processor supports the batching mechanism of run duration.

A full list of issues that were resolved can be found at https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12334375

Version 0.4.0

Version 0.4.0 of Apache NiFi is a provides substantial improvements in functionality and usability, as well as providing some stability and performance improvements and bug fixes.

Release Date: December 11, 2015

Highlights of 0.4.0 release include:

  • New Application Features
    • Multiple Authentication Mechanisms: NiFi now supports multiple Authentication Mechanisms! No longer is NiFi tied to being either non-secure or security based on two-way SSL but now can provide
      User Authentication via LDAP. This was a significant undertaking, but has paved the way to far more easily provide new Authentication Mechanisms. Future releases
      will include additional mechanisms, such as Kerberos.
    • Drop FlowFiles from Queue: Users are now able to right-click on a Connection and drop the FlowFiles in the queue, rather than relying on FlowFile Expiration to remove unwanted FlowFiles.
  • Usability Improvements
    • Explicit Processor Connectivity: Processors that do not expect incoming data will no longer allow incoming Connections. Attempting to draw a connection to a "Source Processor" will show the
      connection line as a red, dotted line, and will not allow the Connection to be made. Likewise, Processors that require input to perform work will be invalid until they have an incoming Connection.
      Some Processors may accept incoming data or run without any incoming data. These Processors will be valid regardless of whether or not they have incoming connections.
    • Getting Started Guide: Getting Started Guide is added to the 'help' screen of the application. This guide provides an introduction to NiFi terms, introduces the key concepts of NiFi and
      discusses how to work with FlowFiles and their Attributes. This is similar in concept to the User Guide but is far less verbose and explains concepts at a higher level.
    • Provenance Fetch Event: A new Provenance Event Type (FETCH) was added. This Event Type is used to indicate that an existing FlowFile's contents were modified as a result of obtaining
      data from an external resource. This is in contrast to a RECEIVE event, which is used to indicate that a FlowFile entered the system as a result of obtaining data from an external resource.
  • New Extensions
    • RouteText: Allows user to easily establish queries against textual data. Each line of a FlowFile is matched against the specified rules and routed according to the rules (potentially many lines of text
      are included in each output FlowFile). Also supports grouping of textual data so that FlowFiles that are output do not contain text from two different groups.
    • TailFile: Allows user to "tail" a file, consuming data from the end of the file as it is written by another process. This is typically used to consume data from log files as it is written. Processor will pick up
      where it left off, even if NiFi is restarted and log files roll over.
    • ListenSyslog / PutSyslog: Listens for incoming Syslog events over UDP or TCP and sends events to Syslog.
    • ListFile / FetchFile: Performs a listing of files in a given directory and fetches those files. These processors differ from GetFile in that the ListFile processor keeps state about files that have been consumed,
      so that the file can be ingested only once without deleting the source file. Additionally, if the directory being monitored exists in a mounted volume, the state can be shared across the cluster so that a new
      primary node can pick up where the previous processor left off, and the listing can also be shared across the cluster, distributing the work of pulling in and processing the files.
    • ListSFTP / FetchSFTP: Performs a listing of files on an SFTP server and fetches those files. Similarly to ListFile / FetchFile, state can be distributed across the cluster so that multiple nodes can perform the
      work in parallel.
    • DeleteS3Object: Removes an Object from Amazon S3.
    • PutHBaseCell / GetHBase / PutHBaseJSON: Allows users to put the contents of a FlowFile to HBase and listen for changes to an HBase table, automatically pulling in the rows that are added/updated.
    • GetAzureEventHub / PutAzureEventHub: Send the contents of FlowFiles as Events to Microsoft Azure Event Hub or listen for incoming Events on an Event Hub and create FlowFiles for those Events.
    • GetCouchbaseKey / PutCouchbaseKey: Send the contents of FlowFiles to Couchbase or fetch the contents of a record from Couchbase.
    • AttributesToJSON: Easily form a JSON document (as the contents of a FlowFile or as an Attribute) from a user-defined set of FlowFile Attributes.
    • SplitAvro: Splits a FlowFile that consists of many Avro records into individual FlowFiles, each containing a smaller number of Avro records.
    • ExtractAvroMetadata: Extracts the metadata from the header of an Avro file and adds the metadata to the FlowFile as a set of Attributes.
    • Image Content Viewer: When users look at the details of a Provenance Event, in the Content tab, if the View button is clicked, and the contents of the FlowFile are an image, that image will be rendered in
      the UI, rather than indicating that no viewer is available for this content type.

A full list of issues that were resolved can be found at https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12333070

Version 0.3.0

Release Date: September 18, 2015

Highlights of 0.3.0 release include

  • Performance improvements in handling large volumes of small files.
  • Performance improvements in Provenance repositories.
  • Added Reporting Task for ApacheTM AmbariTM.
  • Improved stability of nifi bootstrap.
  • Added Processors for working with images.
  • Support for interacting with Kerberos enabled Hadoop clusters
  • Added additional Avro capabilities - merging datafiles & converting to json
  • Added Processors for performing INSERT, UPDATE, DELETE statements against relational databases
  • Added Processors that run ApacheTM FlumeTM sources and sinks.
  • Added Processors to integrate with Amazon Web Services (S3, SNS, and SQS).
  • Archival of content is now enabled by default.

A full list of issues that were resolved can be found at https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12329653

NiFi NAR Maven Plugin Version 1.1.0

Release Date: August 23, 2015

Highlights of 1.10 release include

  • Created a new plugin that allows developers to obtain a listing of transitive dependencies that are provided through a NAR dependency.

A full list of issues that were resolved can be found at https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12333301

Version 0.2.1

Release Date: July 26, 2015

Highlights of 0.2.1 release include

  • Removed all appropriate references to Incubator as NiFi is now a TLP.
  • Fixed a bug that appeared in 0.2.0 that caused new users to a NiFi instance not to be prompted to create an account.

A full list of issues that were resolved can be found at https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12333089

 

  • No labels