Parent page for all content related to Traffic Logs and Analytics. Traffic Logs is a system designed to collect and aggregate server logs from various ATS/ATC servers to be used for monitoring and reporting. The primary challenge is scaling for ATS request/response logging at exceptionally high rates on the order of millions of requests per second. The current intent is to build this out of existing open source platforms such as Kafka, Heka, and ELK.
Links
| Jira | ||||||
|---|---|---|---|---|---|---|
|
Architecture
Core Feature Set
Create an open source system to gather ATS request / response logs so that end users can download raw logs.
1. Must be built from all open source components.
2. Must be able to ingest 1MM records a second on a reasonable amount of hardware and be horizontally scalable to add additional capacity.
3. Must allow storage of raw logs for configurable period of time between 30-90 days.
4. Must be able to process raw logs from ATS and split based service by assessing configuration of service.
5. Must be able to split logs into files of no more than 2 hours or 500k logs whichever occurs first. These should be starter configs and should be configurable on a per service basis.
6. Log files must be compressed to conserve space.
7. Log files should be accessible via a web server interface.
8. Log files should be accessible via an API request.
Additional Feature Set Ideas
- Log indexing for search and charting
- Log aggregation tables for rapid charting via portal
- Log monitoring and alerting
- Allow splitting logs within a service via configurable regex matching
...