Motivation

For now, Ignite has not build-in profiling tool for user's operations and internal processes. Such a tool will be able to collect performance statistics and create a human-readable report. It will help to analyze workload and to tune configuration and applications.

Example of similar tools in other products: AWR [1] [2] [3] (Oracle) ; pgbadger [4], pgmetrics [5], powa [6] (PostgresSQL).

Description

We should provide a way to execute cluster profiling. Consider the following scenario:

  • Enable profiling mode.
  • Executes some arbitrary workload.
  • Collects profiling info.
  • Run the tool that will create the Report contains statistics of workload.

Performance report

The performance report will be in a human-readable format (html page) and should contain:

  • Ignite and plugins versions, topology changes, profiling start/end time
  • Queries (SQL, scan, ..) timings, resources:
    • Queries that took up the most time
    • Slowest queries
    • Most frequent queries
    • Failing queries
    • Queries count by type
    • Queries that took up the most CPU/IO/Disk
  • User tasks timings, resources
    • Jobs of slowest tasks
  • Caches and cache operations statistics:
    • Get/Put/Remove
    • Transactions
    • Invoke
    • Lock
    • create/destroy caches
  • Workload by nodes
    • CPU/IO/Disk resources
  • Checkpoints statistics
  • WAL statistics
  • PME statistics

Additional investigation required to gather following statistics:

  • Query parse time
  • Lock waiting time
  • User time
  • Messages process timings

Proposed Changes

The Ignite will log some additional internal performance statistics to profiling files. The format is like WAL logging.

One disk-writer thread and off-heap memory buffer will be used to minimize affect on performance. Maximum file size and buffer size can be configured on start.

The new extension performance-statistics-ext module will be introduced. It will contain the tool to build the report: build-report.sh(bat). The JSON format is used to store aggregated statistics and next draw in the report.

The report is based on the bootstrap library and can be viewed in a browser offline.

Management

1) JMX: 

PerformanceStatisticsMBean
  • void start() // Start collecting performance statistics in the cluster.
  • void stop() // Stop collecting performance statistics in the cluster.
  • boolean enabled() // True if collecting performance statistics enabled.

2) Control.sh utility. Functionality is like JMX.

3) System properties:

  • IGNITE_PERF_STAT_FILE_MAX_SIZE - Performance statistics maximum file size in bytes. Performance statistics will be stopped when the size exceeded.
  • IGNITE_PERF_STAT_BUFFER_SIZE - Performance statistics offheap buffer size in bytes.
  • IGNITE_PERF_STAT_FLUSH_SIZE - Performance statistics minimal batch size to flush in bytes.
  • IGNITE_PERF_STAT_CACHED_STRINGS_THRESHOLD - Performance statistics maximum cached strings threshold. String caching will stop on threshold excess.

Risks and Assumptions

Enabled profiling mode will cause performance degradation.

Discussion Links

Dev-list discussion.

Report example





Reference Links


  1. https://docs.oracle.com/cd/E11882_01/server.112/e41573/autostat.htm#PFGRF94176
  2. http://www.dba-oracle.com/t_sample_awr_report.htm
  3. http://expertoracle.com/2018/02/06/performance-tuning-basics-15-awr-report-analysis/
  4. https://github.com/darold/pgbadger
  5. https://pgmetrics.io/docs/index.html#example
  6. https://powa.readthedocs.io/en/latest/

Tickets

IGNITE-12666 - Getting issue details... STATUS






  • No labels

3 Comments

  1. Alexey Goncharuk

    Nikita Amelchev I took a look at the sample reports and noticed that only the number of operations is put on the timeline. Do you think it may be possible to collect histograms for each n-seconds interval so that we can build a heatmap for operations durations?

  2. Alexey Goncharuk

    Nikita Amelchev Also, can you also add the following details to the IEP?

    •  How the profiling data is collected? Is the data kept in memory (on heap, off heap?)
    •  or is it being dumped to disk, if yes, what is the format?
    • Did you consider using JFR extensions for dumping the profiling events?
    •  What is expected performance impact when this feature is enabled? Will it have configurable profiles like JFR does?
    • Why do we need a separate command-line tool? Does the tool need to have a cluster online to work? Should we add this to control.sh?
    • If the data is written in binary format, can I build a report later (similar to perf tool?)
    • What is the CLI for the tool (flags, input parameters, security permissions)

  3. Nikita Amelchev

    Alexey Goncharuk Sorry for delaying,

    > Do you think it may be possible to collect histograms for each n-seconds interval so that we can build a heatmap for operations durations?
    Yes, it's possible. All required statistics are collecting and can be properly aggregated and drawn at the report. I think I can add this view during review or at nearest time after.


    > How the profiling data is collected? Is the data kept in memory (on heap, off heap?) or is it being dumped to disk, if yes, what is the format?
    I have updated IEP. Yes, it's collected to profiling files on the disk. Offheap buffer is used bewfore flushing to the disk. The mechanics is like WAL logging.

    > Did you consider using JFR extensions for dumping the profiling events?
    I think this method may not take into account the effect of small queries.

    > What is expected performance impact when this feature is enabled? Will it have configurable profiles like JFR does?
    Locally I get less than 5% impact. I'll benchmark on real cluster soon. Offheap buffer size can be configured. Performance impact can't be configured.

    > Why do we need a separate command-line tool? Does the tool need to have a cluster online to work? Should we add this to control.sh?

    > If the data is written in binary format, can I build a report later (similar to perf tool?)

    The tool does not need a cluster. Profiling files can be parsed locally out of a cluster. It not assume to run a grid client node. This is why I use the separate script.

    > What is the CLI for the tool (flags, input parameters, security permissions)

    I have updated PR. See Profiling management section.

Apache Ignite