An experiment has been conducted to show a possible split of the main NetBeans repository into several smaller parts.
In the experiment, the sources were re-organized into the following structure:
clusters
- platform
- harness
- ide
- java
- <etc>
- ide
- nbi
- nbbuild
In this structure, there Would be a top-level repository (consisting of ide, nbi and nbbuild) and then a repository per cluster.
The code that does the splitting is here:
https://bitbucket.org/jlahoda/split-nb-repo
this is the actualy splitting code:
https://bitbucket.org/jlahoda/split-nb-repo/src/55c9cf672644d61d69cf7a81b8c9e8622d8d988a/src/split/nb/repo/SplitNbRepo.java?at=default&fileviewer=file-view-default
And this patch needs to be applied after splitting:
https://bitbucket.org/jlahoda/split-nb-repo/src/55c9cf672644d61d69cf7a81b8c9e8622d8d988a/adjustments.patch?at=default&fileviewer=file-view-default
After this, a full NetBeans build should be possible, and testing of java.hints should run.
To try this yourself, you'll need:
- NetBeans 8.2 installation, in directory $NETBEANS_HOME
- a clean checkout of NetBeans main repository, tip: b7050e57a3c2, in directory $NETBEANS_REPO
- the target directory (empty), in $NETBEANS_TARGET
Then:
- clone the above repository, cd into it
build the code like this:
ant -Dvar.netbeans.home=$NETBEANS_HOME jar
run the produced jar like this:
java -cp $NETBEANS_HOME/platform/lib/org-openide-util.jar:$NETBEANS_HOME/ide/modules/org-netbeans-modules-project-ant.jar:$NETBEANS_HOME/platform/core/org-openide-filesystems.jar:$NETBEANS_HOME/platform/lib/org-openide-util-lookup.jar:$NETBEANS_HOME/ide/modules/org-netbeans-modules-projectapi.jar:$NETBEANS_HOME/ide/modules/org-netbeans-modules-projectapi-nb.jar:dist/split-nb-repo.jar split.nb.repo.SplitNbRepo $NETBEANS_REPO $NETBEANS_TARGET
apply the adjustments patch:
cd $NETBEANS_TARGET
patch -p1 -i $SPLIT_NB_REPO/adjustments.patch
Please note that this was only an experiment: many things would need fixing/improvements (see an incomplete list below).
Notes on the experiment:
- does not touch modules that are not in the standard distribution or contrib
- if all modules (including contrib) would be re-organized under clusters, paths inside the scripts could be simplified (i.e. clusters/platform/openide.util could become platform/openide.util inside the scripts)
- in the experiment, there is still a single global nbbuild/cluster.properties - would be better to split and place parts into individual repositories
- the directories under "clusters" don't necessarily need to map 1-1 to clusters: a single repository may contain code for multiple clusters. Like maybe ide and extide => ide, cnd and cndext => cnd, or even platform and harness => platform
- developers working on a cluster would need to checkout at least repositories for all clusters their cluster requires, transitivelly. Improvements can be considered, possibly using:
http://wiki.netbeans.org/DevFaqOrphanedNetBeansOrgModules - the experiment will create an unversioned copy of the sources - that would need to be changed to keep history, of course
- the main intent here is to experiment with a way to keep repository sizes smaller while keeping history. One of the risks here is that having (too many) repositories is also not convenient. So, some consolidation of clusters into repositories seems to be in order. As a data point, some other big project is proposing to move from multiple smaller repositories to a single bigger repository (i.e. closer to the current NetBeans state):
http://mail.openjdk.java.net/pipermail/jdk9-dev/2016-October/004987.html
2 Comments
Wade Chandler
On the JDK JEP they mentioned thoughts on core being one repo, but rejected that in favor of the whole, still, I think NB versus the JDK is quite different in some key ways with regard to scope:
Jan Lahoda
All I was trying to say is that having too many repositories is also troublesome. jdk9/dev forest on OpenJDK has in total 8 repositories (top-level, corba, hotspot, jaxp, jaxws, jdk, langtools, nashorn), and tasks like keeping them all up-to-date, or configuring a CI server are quite cumbersome.
Basically, as having a too big repository may cause trouble, having too many smaller repositories also may cause trouble.
For the NetBeans case, I think that if we choose to split the current repo, we probably should not create a repository per cluster. But rather coalescence reasonably related clusters into a single repository. Like ide, extide and maybe even dlight could be one repository, possibly platform and harness (maybe even have platform, harness and nbbuild in one repository), cnd and cndext could belong to one repo, some of the Java technologies could be also share repository (maybe java, javafx, profiler, apisupport? could be joined and javacard and mobility could be joined), etc.