Introduction
The website build pipeline consists of Makefiles, Sphinx, a variety of Sphinx plugins, a Sphinx website theme (and associated client-side modifications), API-specific documentation generation tools such as doxygen, roxygen, scaladocs and lein, shell scripts, Python scripts, Jenkins CI, a GitHub repo for the website artifacts, and Apache infrastructure to host the website.
...
Note: Ideally, this injection process would not happen. It would be handled as part of the website's overall layout as dynamically loaded content. However, the website's current design expects this injection process, and cleaning this up requires a website redesign.
Building the full site with your fork
The previous example building a single version was a precursor to the more complicated process of building the full site and its many versions.
The scripts used by CI and that can be used yourself for building dev version of the site are in build_version_doc and are still build_all_version.sh and update_all_version.sh.
These scripts call make docs which in turn calls make html from the docs directory. So, they inherit the properties described in the settings file. The version tags get passed down from the shell script into the docs Makefile invocation, which then is read in the Sphinx plugin mxdoc.py, which reads the settings file, which is what then calls all of the other docs generation tools.
Building Each Version and Optionally Using Your Fork
Build the content of the 1.2.0 branch in the main project repo to the 1.2.1 folder:
...
./build_all_version.sh "sphinx_error_reduction;v1.2.0;v1.1.0" "master;1.2.1;1.1.0" https://github.com/aaronmarkham/incubator-mxnet.git
Updating Each Version and Optionally Using Your Fork
Assuming you build 1.2.1, 1.1.0, 1.0.0, and master, you need to inject the versions dropdown to reflect these options for each page on the site:
...
Troubleshooting: When running locally in your fork, you may run into errors trying to use the upstream branches. They may not exist in your local repo. You can get around this by using tags instead. Just drop the "v" and use the plain version number. Or you can manually fetch each one that you intend to use so that you have the appropriate branch locally when you run these scripts.
Review the intermediate outputs and the final outputs
Both are really useful and can save you time later.
Intermediate build outputs
The intermediate outputs would the result of each version build. Navigate to docs/build_version_doc/<fork-username>. For my fork this would be docs/build_version_doc/aaronmarkham and inside I would see folders for master, 1.2.1, and 1.1.0 after running each earlier example. The contents of the master folder would have come from my sphinx_error_reduction branch. The contents of the other versions would be as follows: the 1.2.1 folder would have the content from the v1.2.0 branch, and the other two (1.1.0 and 1.0.0) would have the content from their respective branches. Renaming is useful with the branch names don't exactly match what's in them.
Final "updated" site outputs
The updated site outputs can be found by navigating to docs/build_version_doc/VersionedWeb. Note the structure has a version of the website as the root, then there is a versions folder that has each named version inside. Also note the .htaccess file. This is one of the artifacts that is copied during the update step and always placed in the root. It handles the redirects. Another artifact is the README.md file.
A full site example
Now that you have seen a few examples, let's take a look at a sample script the deploys the full site on a dev server. Let's call this fawkes_fly.sh.
...
cd ~/incubator-mxnet/docs/build_version_doc/./build_all_version.sh "1v1.4.x;v1.3.x;v1.2.0;1v1.1.0;1v1.0.0;0v0.12.0;0v0.11.0;sphinx_error_reductionmaster" "1.4.0;1.3.1;1.2.1;1.1.0;1.0.0;0.12.1;0.11.0;master" https://github.com/aaronmarkham/incubator-mxnet.git
./update_all_version.sh "1.4.0;1.3.1;1.2.1;1.1.0;1.0.0;0.12.1;0.11.0;master" master http://34.201.8.176/
sudo rm -rf /var/www/html/*
sudo cp -a ~/incubator-mxnet/docs/build_version/_doc/VersionedWeb/* /var/www/html/
After you have run this, you should be able to open your web browser and visit your site by its IP. Note it now has a versions dropdown.
Build and serve multiple features at once
A handy feature described in the previous example is using your own branch and fork to test your branch's latest updates. Instead of using "master" as the name for the custom branch, use the custom branch name, or whatever you want to appear on the site, so that people can use the dropdown to switch to that feature's view. In this way, you could deploy a site will a variety of branches to do A/B comparisons.
If you have already built various versions and you don't want to recreate them, then don't do the delete step in the earlier example. Retain the website. Each version is in the website's /version folder. Also, each version that was built and pre-update is in the docs/build_version_doc/<fork-username> folder. You can also run the one-off site build using make html. Copy your newly created versions of the site into aptly named subfolders within the the docs/build_version_doc/<fork-username> folder. Then rerun update_all_version.sh with the versions you want to appear in the dropdown list. Take that output and serve it, or even make a new directory in /var/www/html/versions and copy it in there, so you can run many flavors of the site at will.
Troubleshooting
- Sometimes the submodules get updated in master and you have to specifically update them yourself. If you see a build error that crashes on mshadow or another submodule's build you likely need to run the following command in your MXNet repo:
...
- You might get nagged about a file that is going to get overwritten with a checkout when you've made local changes in your branch, but are trying to run a test build from the upstream branch. At this point it is better to set the build_all_version script to use your fork, so you can test it out.
Next up
Website Updating Guide (pre v1.6)