Page History

...

Hive build jobs include Hive PreCommit Testing, and scheduled builds of various branches with various options.flags.  They are run on the Hive Ptest2 Infrastructure, which is a EC2 cluster currently sponsored by Cloudera, which runs each build sequentially, but splits the testing across all the resources of the cluster to achieve much higher throughput.

...

1. Jenkins: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/
   • This is the admin for Hive builds.  In the dashboard, user can see state of build queuesqueue, test results of run jobspast builds, etc.  
   • Jobs Builds are submitted either manually in the dashboard UI, automatically by patch upload (See Hive PreCommit Testing), or by schedule (branch builds).
2. Hive PTest WebServer
   • Exposes a REST API to take test request from the Jenkins jobsbuilds.  Each request specifies a profile property-file, which contains a set of properties such as what branch to build, what tests to run.
   • On a test request, it creates :
     • Creates EC2 slaves, which are spot-instances.
     Makes SSH and EC2 calls to distribute the testing to the EC2 slaves
     • Reads the specified property-file and compiles source code using it.
     • Distributes the compiled code across the EC2 slaves, makes SSH calls to run the tests remotely in parallel, and gather the results.
      If
   1. • If no further test request come in 30 minutes, the slaves are shutdown.

Jenkins Configuration/Debug

...

Hive PTest Configuration/Debug

• Test Logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/  (/data/hive-ptest/logs/)
• Profile properties files: /usr/local/hiveptest/etc/public/
• Webservice logs: /home/hiveptest/apache-tomcat-7.0.41/logs/
• Working Directory: /data/hive-ptest/working/ (where libs, source, etc are stored)
• Scripts: /usr/local/hiveptest/bin/
• Relevant JIRAS: HIVE-4675 and HIVE-4739

...

In /usr/local/hiveptest/public/ you can copy existing profile property-file, and update it to point to the new branch property.  Then in Jenkins, copy an existing jenkins job, but change the name of the profile property-file branch build, and in the build config modify the profile argument in the Hive Ptest2 Webserver call to point to the new oneprofile.

How can I read or make changes to Hive PTest Infra code?

Code is located in hive git repo.  It is under /hive/testutils/ptest2.  As the Hive Ptest2 Webserver is a running process, if changes are made, it needs to restartbe restarted.

How do I stop, start, restart the Hive PTest2 server, or have it use the latest test infra code?

...