During installation of CloudStack we deploy database and during upgrade database upgrade becomes a very serious issue. Let's talk about the present database deployment and the issues:
- We've two different routes to deploydb, using different approach for developers and for sysadmins (they would use cloudstack-setup-database)
- Until 4.1, upgrades are done by mgmt server, this has issues when there is a clustered mgmt server, i.e. multiple mgmt servers sharing the same database, we do upgrade by:
a. Turn down one mgmt server, upgrade it, start it. This mgmt server checks in db the cloudstack version, upgrades the db
b. Sysadmins meanwhile upgrades all the other mgmt server
c. When all mgmt servers are upgraded, they ask the cluster manager to run the cleanup. The whole process is automated, an error can introduce whole sets of issues.
We need an external tool which empowers a sysadmin and is also used by the developers. The process should allow a sysadmin to decide an upgrade or deployment path of database. Hence the motivation of DatabaseCreator, a tool which should solve these issues.
How will DatabaseCreator help:
- Would give a single tool used by both developer and sysadmins. Agony of the sysadmins would be much better understood by developers. Sysadmins would still use cloudstack-setup-database which would be
wrapper around DatabaseCreator
- Upgrade would be driven by sysadmin, giving them explicit power to backup database, deploy etc. For example in case of multiple (clustered) mgmt server, the upgrade would be like:
a. Upgrade database using DatabaseCreator, this step do a backup and would just upgrade the db such that it's backward compatible to the old database and old mgmt servers can still run on it. When this succeeds, system can proceed. Transaction update would provide a way to rollback.
b. Sysadmins would stop, upgrade and start mgmt server one by one.
c. Sysadmins would run the final cleanup operation, this would do a final round of cleaning up. If everything goes well, they just did a rolling CloudStack upgrade with little to no downtime (no downtime as at least one mgmt server was running)
To implement this, we first implemented a rolling upgrade mechanism by which the base sql schema was reverted to the one in 4.0 version. By sticking to a base version schema, developers would implement only upgrade paths from 4.0 version to version X. To do this, we did the following for 4.1 version:
- We stick to major db version 4.0 and create-schema is not changed.
- We move out any 4.1 changes to schema-40to410.sql which is the upgrade sql file and we implement a java class which handles the upgrade path.
Starting 4.2, we have following this done:
- Refactor out code that does upgrading in mgmt server. This means mgmt server will no longer upgrade.
- Separate out DatabaseCreator from cloud-server to a new artifact in tools/dbcreator.
- Fix the tool so it supports the operations as discussed above
- Replace the current pythonic implementation of cloud-setup-databases which wraps around DatabaseCreator.
- WE NEVER CHANGE schema files such as create-schema.sql
- WE TEST AND IMPLEMENT UPGRADE PATHS from version a to version b
- WE CLEANUP PROPERLY via our upgrade sql files and upgrade classes
The following is the 4.1 DatabaseCreator's usage: (this will change slightly for 4.2)
DatabaseCreator creates the database schema by removing the previous schema, creating the schema, and running through the database updaters.
Usage: DatabaseCreator [options] [db.properties file] [schema.sql files] [database upgrade class]
Options:
--database=a,b comma separate databases to initialize, use the db name in db.properties defined as db.xyz.host, xyz should be passed
--rootpassword=password, by default it will try with an empty password
--dry or -d, this would not run any process, just does a dry run
--verbose or -v to print running sql commands, by default it won't print them
--help or -h for help