Apache Eagle should have a option to use ELK storage as a cod storage ( & longer retention), This also makes it possible to create any ELK based query to visualize data that is not provided by Apache Eagle or it can be integrated into dynamic charting library if and when Apache Eagle provides that functionality.

 

currently producer ingest or pull data and put it into kafka topic, which gets converted into JSON or other structured format to be processed further by app and alerts engine which also has a publisher to publish alert at various channel (email, db, kafka, etc... ).

 

This proposal should include all posible integration point where ELK can be leveraged as a cold storage.

  1. Raw logs data to ELK that is ingested into kafka.
    1. create elk app, that takes the topic info and format to indexify and put it into given ELK.
    2. create elk bolt to process all the data that alert app gets ( although it puts dependency onto alert app )
  2. alerts to ELK :( Taken from proposal to add ELK publisher is already made here :  EAGLE-958 - Getting issue details... STATUS
  3. aggregated metrics to ELK ?

 

 



  • No labels