Bigtop Packaging

One of the longer term goals is to have Bigtop packages serve as as reference to eventually get Hadoop introduced into most Linux distributions.

Currently, this is problematic owning to the lack of maven3 native packaging for many major Linux distributions. However, Bigtop makefiles and other artifacts, create an alterntive build root, which then uses native rpm or .deb packaging tools to create packages.

We strive to follow best practices with both rpm and Debian based distributions.
If you discover packaging or installation issues, please file a ticket https://issues.apache.org/jira/browse/BIGTOP

Getting started with bigtop packaging.

1) You will have to package both deb and rpm.  Starting with one or the other is the best.

2) Here are some steps you can follow to bring a new bigtop package in.

• determine where your source is, and add it to bigtop.bom
• update the bigtop-packages/src/common/<your-package> folder to have your component, and the do-component-build for it (which usually just builds a jar).  Why is there a "common" directory? Simply because deb and rpm packaging share some tasks (like do-component-build, which just usually runs a mvn or gradle command), and so we keep a common install directory which they can both leverage for packaging.
• If you need to apply a patch to the upstream source (use the utmost caution, in general Bigtop doesn't patch upstreams):
  • create a patch file (in the patch -p1 syntax) and place it in the bigtop-packages/src/common/<your-package> common folder. Name should be patch*.diff
  • for RPM Support place a #BIGTOP_PATCH_FILES token into the section with the Sources and  #BIGTOP_PATCH_COMMANDS between %setup and %build
  • for DEB the patch is applied by default.
  • multiple patch files are applied in the glob sorting order
• (for RPM) now add a .spec file into bigtop-packages/src/<your-package>/... into the appropriate directory (i.e. bigtop-packages/src/rpm/tachyon/SPECS/tachyon.spec).  Obviously, your tachyon.spec file will use whats in common/ in a RPM specific way, to install the RPM package.
• (for Debian) same as above ^ but of course, follow the debuild debian idioms: 
  
  • create a rules file using do-component-build
  • create a compat file, please use version 9 for new packages

  • create a control file, please use for new packages:   Build-Depends: debhelper (>= 9) and Standards-Version: 3.9.4 

• Test it with gradle <your-package>-deb on Ubuntu/Debian or <your-package>-rpm for the others.
  
  • For DEB: Have a look at the lintian errors and warnings, please try to minimize the numbers. 
• Finally add a smoke test! This is as easy as adding a new groovy file to bitop-tests/smoke-tests/<your-package>/TestThisStuff.groovy, following conventions that others have created. 

3) As always, we will improve on the directions above, but this should help to get you started. .

Helpful docs on bigtop packaging (will expand this table over time). 

Description	URL
Romans talk on RPMs vs Tarballs and BigTop Packaging
Walkthrough of RPM spec for hadoop in BigTop	http://jayunit100.blogspot.com/2014/04/how-bigtop-packages-hadoop.html
RPM Macros	https://rpm-packaging-guide.github.io/#rpm-macros https://docs.fedoraproject.org/en-US/packaging-guidelines/RPMMacros/
RPM %Files	https://rpm-packaging-guide.github.io/#files
Debian Policies	https://www.debian.org/doc/debian-policy/