Page tree
Skip to end of metadata
Go to start of metadata

Parent page for all content related to Traffic Logs and Analytics.  Traffic Logs is a system designed to collect and aggregate server logs from various ATS/ATC servers to be used for monitoring and reporting.  The primary challenge is scaling for ATS request/response logging at exceptionally high rates on the order of millions of requests per second.   The current intent is to build this out of existing open source platforms such as Kafka, Heka, and ELK. 

 

Links

TC-220 - Getting issue details... STATUS

Architecture

Core Feature Set

Create an open source system to gather ATS request / response logs so that end users can download raw logs.

1. Must be built from all open source components.
2. Must be able to ingest 1MM records a second on a reasonable amount of hardware and be horizontally scalable to add additional capacity.
3. Must allow storage of raw logs for configurable period of time between 30-90 days.
4. Must be able to process raw logs from ATS and split based service by assessing configuration of service.
5. Must be able to split logs into files of no more than 2 hours or 500k logs whichever occurs first. These should be starter configs and should be configurable on a per service basis.
6. Log files must be compressed to conserve space.
7. Log files should be accessible via a web server interface.
8. Log files should be accessible via an API request.

 

Additional Feature Set Ideas

  • Log indexing for search and charting
  • Log aggregation tables for rapid charting via portal
  • Log monitoring and alerting
  • Allow splitting logs within a service via configurable regex matching 

 

  • No labels

2 Comments

  1. Ryan Durfey What is traffic logs and why do we have a page for it?

  2. Traffic Logs is a system designed to collect and aggregate server logs from various ATS/ATC servers to be used for monitoring and reporting.  The primary challenge is scaling for ATS request/response logging at exceptionally high rates.   The current intent is to build this out of existing open source platforms such as Kafka, Heka, and ELK.   It is just getting off the ground at this time.  I will push some info out as soon as it is available.