Page tree
Skip to end of metadata
Go to start of metadata

Loading test data

Run all tests:

Run just front-end tests

Run just back-end tests

Run most end-to-end tests

Run custom cluster tests

Run stress test

Things to look for in Test Failures

  1. Look for Minidump crashes. These look like: ./Impala/logs_static/logs/ee_tests/minidumps/impalad/71bae77a-07d2-4b6e-0885eca5-adc7ee36.dmp. Because some tests run in parallel (especially in the "ee" test suite), if one of the Impala daemons crashed, subsequent test results are meaningless. Start with the crash. If you see a crash dump, it's possible that there's a stack trace in the logs:
    # ag is a recursive grep tool; see https://github.com/ggreer/the_silver_searcher
    $ag 'Check failure stack trace:'
    Impala/logs_static/logs/ee_tests/impalad.ip-172-31-24-168.ubuntu.log.ERROR.20171023-180534.38031
    101743:*** Check failure stack trace: ***
    101746:*** Check failure stack trace: ***
    Impala/logs_static/logs/custom_cluster_tests/impalad.ip-172-31-24-168.ubuntu.log.ERROR.20171023-193700.44118
    12:*** Check failure stack trace: ***
    Impala/logs_static/logs/custom_cluster_tests/impalad.ip-172-31-24-168.ubuntu.log.ERROR.20171023-193658.43993
    12:*** Check failure stack trace: ***
    Impala/logs_static/logs/custom_cluster_tests/impalad.ip-172-31-24-168.ubuntu.log.ERROR.20171023-193659.44062
    12:*** Check failure stack trace: ***

     

  2. Look for "FAILED" (case insensitive) in the log output of the Jenkins job:

    $cat console | egrep -o -i 'FAILED +[^ ]+ +[^ ]+'
    FAILED query_test/test_udfs.py::TestUdfExecution::test_ir_functions[exec_option: {'disable_codegen_rows_threshold':
    FAILED query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_large_rows[exec_option: {'debug_action':
    FAILED query_test/test_udfs.py::TestUdfExecution::test_native_functions[exec_option: {'disable_codegen_rows_threshold':
    failed to connect:
    failed in 12443.43
    failed in 4077.50
  3. The tests produce xunit XML files; you can look for failures in these like so:
    # Search for and extract 'failure message="..."' in all XML files; uses
    # a multiline RE.
    $grep -Pzir -o 'failure message=\"[^"]*' $(find . -name '*.xml') | tr '\0' ' ' | head
    ./Impala/logs_static/logs/ee_tests/results/TEST-impala-parallel.xml:failure message="query_test/test_nested_types.py:70: in test_subplan
    self.run_test_case('QueryTest/nested-types-subplan', vector)
    common/impala_test_suite.py:391: in run_test_case
    result = self.__execute_query(target_impalad_client, query, user=user)
    common/impala_test_suite.py:600: in __execute_query
    return impalad_client.execute(query, user=user)
    common/impala_connection.py:160: in execute
    return self.__beeswax_client.execute(sql_stmt, user=user)
    beeswax/impala_beeswax.py:173: in execute
    handle = self.__execute_query(query_string.strip(), user=user)

What is the difference between impala-py.test and test/run-tests.py?


py.test (https://docs.pytest.org/en/latest/) is a framework for writing tests in Python. It's got a bunch of features. impala-py.test is a small wrapper that invokes py.test in the right environment (i.e., from the "virtualenv") and with LD_LIBRARY_PATH updated. When you're using impala-py.test, you're using py.test's mechanisms. Meanwhile, test/run-tests.py is a wrapper that invokes py.test a few times. It uses environment variables for control. Then it executes any "serial" tests, and then it executes some stress tests, and then, it instructs py.test to execute the "parallel" tests (those not marked serial or stress) with a bunch of parallelism. If you plotted your CPU graph while this was happening, you'd see something like. You can tell where the parallel testing happens about 3/4 of the way through. To make things even more complicated, there's bin/run-all-tests.sh, which invokes run-tests.py (but also "mvn test" and the C++ tests), and buildall.sh, which invokes run-all-tests.sh.

 

  • No labels

3 Comments

  1.  

    Tim Armstrong, in the stress test, how would I get for myself a tpch_60_parquet database?

    1. There isn't a super-easy way that I'm aware of - you can piece something together using tpch dbgen and ./bin/load-data.py. I updated the instructions to refer to a DB that is available by default.

       

      1. ./bin/single_node_perf_run.py now supports loading larger scales of TPC-H