The Clojure package was designed to be able to open the deep learning library to the Clojure ecosystem and build bridges for future development and innovation for the community. Since Clojure is a JVM language, it leverages the great work of the Scala package through interop with Java.
The main goal of the architecture was to build on the Scala package. The Clojure api reflects this design choice by closing following the Scala version while customizing it for a more idiomatic experience for Clojure users.
The building of the package on Scala does not preclude a migration to a pure Clojure/JNI migration in the future if desired. That effort can be done in an incremental fashion with the existing test suite as a refactoring guide.
The Scala package generates the bulk of the NDarray and Symbol functions through Scala macros from the C API. It does this in an automatic fashion during the runtime. These macros then generate Java Classes which can be exposed through reflection to Clojure.
The Clojure package uses a code generator process with reflection that looks at the Java NDarray and Symbol classes and produces Clojure files from it. The rationale behind this is to provide the user with actual files so that the functions can be browsed and inspected as well as documentation can be generated with them.
The process is automatic and at compile time - the generated files are not checked into git, but reside in org/apache/clojure-mxnet/gen and are packaged into the jar artifact.
The bulk of the NDArray and Symbol clojure functions are generated, but there are still some handcrafted ones as well. This is achieved by having a main namespace import the generated functions from a separate file.
The rest of the Clojure package namespaces are hand coded using the Scala/Java classes via interop.
The namespace of org.apache.clojure-mxnet was chosen to be different than org.apache.mxnet because of the close coupling and problems with interop with the Scala generated Java classes.
Different namespaces:
The organization of the code follows the Scala layout to a large extent, with the generator code under dev:
├── dev
│ └── generator.clj
└── org
└── apache
└── clojure_mxnet
├── base.clj
├── callback.clj
├── context.clj
├── dtype.clj
├── eval_metric.clj
├── executor.clj
├── gen
│ ├── ndarray.clj
│ └── symbol.clj
├── initializer.clj
├── io.clj
├── kvstore.clj
├── kvstore_server.clj
├── lr_scheduler.clj
├── module.clj
├── monitor.clj
├── ndarray.clj
├── optimizer.clj
├── profiler.clj
├── random.clj
├── shape.clj
├── symbol.clj
├── util.clj
└── visualization.clj
The unit tests for the Clojure package were ports of the Scala unit tests to ensure compatibility. They are cpu only and can be run with lein test
Although the goal of the Clojure package is parity with the Scala version, some items have been deferred:
The Build process uses Leiningen (https://leiningen.org/), which is the major Clojure build tool. It is maven based so it has compatibility with current project's artifacts and setup. It also has a standalone version that can be simply added to the repo and used without having to modify anything on the build servers.
The Build has a project dependency on the appropriate Scala jar for the operating system.
To build the Clojure project and get the resulting jars, use the command lein install to install in maven locally. or lein jar that will just create the jars in the target directory.
The jar artifacts can then be deployed to a repository. The process here needs to be determined.
Documentation can always be improved, but the Clojure project has a good start in the form of docstrings, clojure.spec, api generation, and examples
An effort has been made to add a descriptive docstring to every major api function.
Clojure.spec, which is an optional typing tool, has been added to the Module api to help users get feedback on the form of the parameters for the functions.
The use of this can be expanded in the project more generally in the future.
API generation can be done on the project with the plugin https://github.com/weavejester/codox by executing lein codox.
Many of the Scala examples have been ported over to the Clojure package as well as a Tutorial REPL driven example which can be used for web page tutorials.
Here is a list of current examples: