Abstract

Hunter is a command-line tool, written in Python, that detects statistically significant changes in time-series data stored either in databases or CSV files. It is designed to be easily integrated into build pipelines and provide automated performance analysis that can decide whether code should be deployed to production.

Proposal

Background

Hunter performs statistical analysis of performance test results stored in CSV files or databases like Graphite, PostgreSQL, or BigQuery. It finds change-points and notifies about possible performance regressions.

A typical use-case of hunter is as follows:

A set of performance tests is scheduled periodically.
The resulting metrics of the test runs are stored as a time series in a database (Graphite, PostgreSQL, BigQuery) or appended to CSV files.
Hunter is launched by a Jenkins/Cron job (or an operator) to analyze the recorded metrics regularly.
Hunter notifies about significant changes in recorded metrics by outputting text reports or sending Slack notifications.

Hunter is capable of finding even small, but systematic shifts in metric values, despite noise in data. It adapts automatically to the level of noise in data and tries not to notify about changes that can happen by random. Unlike in threshold-based performance monitoring systems, there is no need to set up fixed warning threshold levels manually for each recorded metric.

The level of accepted probability of false-positives, as well as the minimal accepted magnitude of changes are tunable. Hunter is also capable of comparing the level of performance recorded in two different periods of time – which is useful for e.g. validating the performance of the release candidate vs the previous release of your product.

We aim to share Hunter with the ASF and use the strengths Apache brings in sustainability as a community of developers and users continues to grow.

Rationale

Performance regression detection is critical to maintaining high-quality software, especially in large-scale, distributed, and performance-sensitive systems. Existing projects often rely on manual or threshold-based analysis, which does not handle noisy results very well. Hunter addresses the problem by using statistical significance tests, such as Student’s t-test, to detect anomalies in benchmark results. It provides a fast, automated, accurate, and easy-to-integrate solution for detecting performance regressions in benchmark results.

By bringing Hunter to the ASF Incubator, we leverage the foundation’s strong community and collaborative development model to enhance the tool’s capabilities and adoption. ASF’s principles of community-driven development and open governance align perfectly with our vision for Hunter. Hunter has the potential to become a valuable asset within the ASF ecosystem, improving the performance quality of other Apache projects.

This is the right time. Adoption of changepoint detection (CPD) is growing and an ASF-backed project will help with that and foster a community where people can benefit from CPD.

Initial Goals

Transfer the existing codebase and documentation to Apache-hosted infrastructure.
Adapt to the Apache development process and infrastructure, transitioning our code review, build, and testing workflows to the ASF context.
Expand and diversify the Hunter community.
Integrate with other ASF projects.

Current Status

Meritocracy

Our goal with this incubator proposal is to build a diverse developer community around Hunter, following the Apache meritocracy model. Since Hunter was open-sourced in 2021, several companies have adopted it including Confluent and Nyrkiö, gaining active contributors from these organizations.

We are committed to fostering an environment that supports meritocracy, believing that the community and project will thrive by following the Apache Way.

Community

The Hunter community can currently be described as the below listed 6 core developers, plus a handful of power users that know the core team well and could potentially become contributors and committers in the future. Finally there are additional users of Hunter that use the tool daily but rarely interact with the core team.

The core contributors have presented and authored articles on their use of Hunter at several performance related industry conferences.

The core contributors share a common ancestry in that they all worked at Datastax at the time Hunter was created, however today a majority have other employers.

Core Developers

The core contributors are a diverse group of both unaffiliated developers and those hailing from small to large companies. They are scattered geographically, and some are highly experienced industry as well as open source developers. Though their backgrounds may be diverse, the contributors are united in their belief in community driven software development.

More detailed information on the core developers and contributors in general can be found under the section on homogeneous developers.

Alignment

Hunter is licensed under the Apache Software License 2.0. The copyright owner of a majority of the code and the sponsor of the first release of Hunter is Datastax, who has a long history with the Apache Foundation and therefore handing over the project to the ASF is a well understood, low friction process.

A major dependency is the pip module called simply "signal_processing_algorithms". It is also licensed under the ASL 2.0. It was open sourced by MongoDB in 2020 and remains under active development and is used in production. (However, MongoDB and this component is not part of this incubation proposal, rather will remain an external dependency.)

Known Risks

Project Name

…todo…
PODLINGNAMESEARCH-228 - Getting issue details... STATUS

Orphaned products

Hunter has been deployed at DataStax since 2020 and at Confluent since 2022 supporting multiple systems. Nyrkiö is a Finnish startup that in 2024 launched "Hunter-as-a-Service". Through Nyrkiö, Hunter is being used by several more companies, including Dremio and Tigerbeetle.

The core math library was open sourced by MongoDB in 2020 and is still used in their CI pipeline. This library is also used at several other technology companies, including Netflix.

Inexperience with Open Source

The core team has extensive experience contributing to open-source projects.

Henrik is a long time open source advocate, author of a book and namesake blog on open source community and business models. He volunteered on the OSI license-review mailing list for 2 decades. He is best known for his career with multiple open source databases, including recently Apache Cassandra. But he has also contributed to Drupal 7, impress.js, and a Linux webcam device driver - making him truly a full stack developer.

Sean is the first engineer to deploy Hunter in production to test Datastax Enterprise and later Apache Cassandra. He is currently the maintainer of the datastax-labs/hunter repository, which will become the code base for the Hunter Apache Incubator.

Matt is most known for his career as a Linux kernel subsystem maintainer, performance expert and contributor to the perf utility. He recently led a team of engineers who open sourced and contributed to Apache Cassandra 5.0 significant performance improvements in the area of indexing structures and compaction algorithms.

Piotr is the original author of Hunter. In his day job at Datastax he contributes to Apache Cassandra in the area of secondary indexing and query optimization.

Alex is an Apache Cassandra committer and a contributor to Apache Kafka, Apache Flink, and Apache Paimon. He introduced Hunter at Confluent to ensure performance quality of Kafka Streams.

Homogenous Developers

The 6 core developers are from 3 different companies: Nyrkiö, DataStax, Confluent. No company employs more than 2 developers.

Nyrkiö is a small startup aiming to bring Hunter into mainstream use by offering it as a web service and providing support and consulting services. Datastax and Confluent are mature technology companies using Hunter in the performance testing of their own products. In addition, two developers formerly worked at MongoDB, where the algorithm implementation used by Hunter was first developed.

Reliance on Salaried Developers

Hunter development is anticipated to take place during both salaried and volunteer hours. While most of the initial committers are compensated by their employers for their contributions, they all share a deep passion for the project. We are confident and hopeful that by fostering a strong community around Hunter, the project will thrive even if salaried developers are no longer able to contribute. Four of the proposed committers have continued to stay with the project and/or topic of change detection after they left the employer where they first used or created this technology, and ultimately introduced it at their next jobs.

Relationships with Other Apache Products

Confluent uses Hunter to analyze benchmark results of Kafka Streams, part of Apache Kafka project. The Confluent team supporting Apache Flink is planning to start using Hunter to analyze benchmark results of Apache Flink. DataStax uses Hunter to analyze benchmark results of Apache Cassandra. Dremio is using Hunter through Nyrkiö, to catch performance regressions in their version of Apache Iceberg.

Apache Brand

The primary motivation for submitting Hunter to the ASF is to build a diverse and robust community while ensuring long-term development stability. We aim to encourage a wide range of organizations to adopt and contribute to Hunter without concerns about ownership or licensing, or patch reviews and merges being dependent on a single company's employees, or even a single person. Additionally, being part of the Apache Foundation will facilitate the integration of Hunter into the QA pipelines of other Apache projects, thereby enhancing their performance quality.

Documentation

Hunter end user documentation is currently in a long README file (400+ lines) and some additional examples. Developer documentation is in the CONTRIBUTING file.

The project doesn't have a web page. We hope a project website on ASF web properties will improve visibility of the project itself and project documentation.

Initial Source

The source is located on GitHub in the following repositories:

https://github.com/datastax-labs/hunter - original repo and still the formal upstream
https://github.com/Gerrrr/hunter - Confluent fork, actively merging upstream
https://github.com/nyrkio/hunter - Nyrkiö fork, waiting to merge to the incubator repository once available

Source and Intellectual Property Submission Plan

Hunter’s initial source is licensed under the Apache License, Version 2.0. https://github.com/datastax-labs/hunter/blob/master/LICENSE.txt

All source code is copyrighted to 'DataStax', which will be re-assigned to the ASF after the SGA from DataStax is received.

External Dependencies

[Apache 2.0]
google-api-core 2.19.2
google-auth 2.34.0
google-cloud-bigquery 3.25.0
google-cloud-core 2.4.1
google-crc32c 1.5.0
google-resumable-media 2.7.2
googleapis-common-protos 1.65.0
grpcio 1.66.1
grpcio-status 1.66.1
importlib-metadata 7.1.0
importlib-resources 6.4.0
proto-plus 1.24.0
regex 2024.5.15
requests 2.32.3
rsa 4.9
signal-processing-algorithms 1.3.5

[MIT]
asn1crypto 1.5.1
attrs 23.2.0
autoflake 1.7.8
backports.tarfile 1.2.0
cachetools 5.5.0
charset-normalizer 3.3.2
expandvars 0.6.5
flake8 4.0.1
inflect 7.3.1
iniconfig 2.0.0
isort 5.13.2
jaraco.collections 5.1.0
jaraco.context 5.3.0
jaraco.functools 4.0.1
jaraco.text 3.12.1
mccabe 0.6.1
more-itertools 8.14.0
pip 24.2
platformdirs 4.2.2
pluggy 1.5.0
py 1.11.0
pycodestyle 2.8.0
pyflakes 2.4.0
pystache 0.6.5
pytest 6.2.5
pytz 2021.1
ruamel.yaml 0.17.21
ruamel.yaml.clib 0.2.8
ruff 0.6.9
scramp 1.4.5
setuptools 75.1.0
six 1.16.0
slack-sdk 3.28.0
tabulate 0.8.10
toml 0.10.2
tomli 2.0.1
tox 3.28.0
typeguard 4.3.0
tzlocal 5.2
urllib3 2.2.1
validators 0.18.2
virtualenv 20.26.2
wheel 0.44.0
zipp 3.19.2

[BSD-3-Clause]
dateparser 1.2.0
idna 3.7
numpy 1.24.0
pg8000 1.31.2
protobuf 5.28.0
scipy 1.9.3

[BSD-2-Clause]
decorator 5.1.1
pyasn1 0.6.0
pyasn1-modules 0.4.0

[MIT or Apache-2.0]
structlog 19.2.0

[BSD-2-Clause or Apache-2.0]
packaging 24.1

[BSD-3-Clause or Apache-2.0]
python-dateutil 2.9.0.post0

[Unlicense]
filelock 3.15.1

[MPL-2.0]
certifi 2024.6.2

[PSF-2.0]
distlib 0.3.8
typing-extensions 3.10.0.2

Cryptography

Hunter contains no cryptographic algorithms.

Required Resources

Mailing Lists

dev@hunter: for development discussions
private@hunter: for PPMC discussions
commits@hunter: for code changes

Git Repositories

The Hunter team is experienced in git and requests to transfer GitHub repositories(list in Initial Source) to Apache.

All GitHub notifications will be archived on ASF mailing lists.

Issue Tracking

The community would like to continue using GitHub Issues.