Problem

Many of the current MXNet-Scala APIs are derived from the MXNet C++ backend, they are extracted using Scala Macros. These APIs are not visible when used on an IDE since they are generated only at Compile time.  The  parameters to these APIs are Key value arguments which forgoes type-safety with no documentation available, User has no idea on what APIs are available or what parameters these APIs take, his best bet would be to look at Python API documentation on mxnet.apache.org and try out a similar approach in Scala. The current API looks something like the below.

Symbol.Activation(name = "relu2")()(Map("data" -> fc2, "act_type" -> "relu"))

Proposed Approach

In this project, we will extract all the public APIs available from the C++ backend and generate a Scala trait for each module with the all APIs and the corresponding parameters. The concrete implementation of the module will extend the trait, the implementation comes from the existing Macro.

There will also be a new implementation on the Scala APIs which will be more user-friendly as shown below.

Symbol.Activation(Some(fc1), "relu", Some("relu2"))

Advantage for this API:

• All arguments will be well documented for user to check.
• Argument list will be fixed which avoid wrong args input comparing to the previous Scala API.
• Less input from user required.

For example,  for NDArray APIs we will generate a file NDArrayBase.scala with NDArrayBase trait with all the API signature. NDArray class will extend this trait to provide the implementation using existing Macro or modified macro that uses the generated signature.

Alternative Approach considered

Create a functionBuilder and use a keyword build() to execute the function.

function_nameBuilder(requiredargs...)
	.arg1(arg1)
	.arg2(arg2)
	.arg3(arg3)
	.build()
 
class functionBuilder(requiredArg : String, map : mutable.Map[String, Any] = mutable.Map()) {

  def addArg1(arg1 : String) : functionBuilder = {
    map("arg1") = arg1
    new functionBuilder(requiredArg, map)
  }

  def addArg2(arg2 : Int) : functionBuilder = {
    map("arg2") = arg2
    new functionBuilder(requiredArg, map)
  }

  def build() : String = {
    "Build in progress"
  }
  
}

Advantage for this API:

• Frustration free on many optional parameters
• Clearer view to see how one function constructed
• Better Java Support (if we have many arguments in a function that can be optional) 

Technical Challenges

There are four major difficulties we need to solve:

• Extracting C++ APIs information and convert them into correct format in Scala.
• Generating functions with arguments and docs
• Handling different types of input and generating corresponding code

Here is an example on what it would look like for a single function:

/**
  * Applies an activation function element-wise to the input.
  * 
  * The following activation functions are supported:
  * 
  * - `relu`: Rectified Linear Unit, :math:`y = max(x, 0)`
  * - `sigmoid`: :math:`y = \frac{1}{1 + exp(-x)}`
  * - `tanh`: Hyperbolic tangent, :math:`y = \frac{exp(x) - exp(-x)}{exp(x) + exp(-x)}`
  * - `softrelu`: Soft ReLU, or SoftPlus, :math:`y = log(1 + exp(x))`
  * - `softsign`: :math:`y = \frac{x}{1 + abs(x)}`
  * 
  * 
  * 
  * Defined in src/operator/nn/activation.cc:L150
  * @param data   The input array.
  * @param act_type   Activation function to be applied.
* @return Symbol
*/
def Activation(data : Option[Symbol], act_type : String, name : Option[String], attr : Option[Map[String, String]]) : Symbol= {

    val map : mutable.Map[String, Any] = mutable.Map()
    if (!data.isEmpty) map("data") = data.get
    map("act_type") = act_type
    val currName = name.getOrElse("")
    val currAttr = attr.getOrElse(Map())
    createSymbolGeneral("Activation", currName, currAttr, Seq(), map.toMap)

}

We are also facing some senario in our design:

A function in MXNet called FullyConnected, it contains 6 variables. If the 5th variable (no_bias) set to be true, then user don't need to pass in the 2nd and third variable. If that one set to be false, then user are required to pass in these variable. So it is nearly impossible to generate different override methods base on users' need. We have hundreds of methods like this and it will take a very long time. We hope if we can get this information from C++ APIs but unfortunately we don't have them in hand. In that case, a solution for this may just set all args to be None so user can choose to pass in or not. And Python API do the same thing https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.FullyConnected. In this case, we choose to set all optional args to None/Null by default in order to let user pass minimum input to the example.

Open Questions

Milestones

How do we generate the code

Code generation is essentially an important part in the MXNet installation section. There are currently two approaches: 

1. Generate with write file.
2. Generate with Macros

The second option are currently used inside of the MXNet Scala to generate the old APIs. However, more workload will be applied to new API as the arguments and implementation of the code are dynamic. As an example, the old API would be fixed into the format as shown below:

Symbol.<function_name>(name : String = null, attr : Map[String, String] = null)(args : Seq[Symbol] = null)(kwargs : Map[String, Any]) : Symbol = {
	createSymbolGeneral(name, attr, args, kwargs)
}

The new API would look like:

Symbol.<function_name>(name : Option[String] = None, attr : Option[Map[String, String] = None, arg1 : Symbol, arg2 : MXFloat ...) : Symbol = {
	val map : mutable.Map[String, Any] = Mutable.Map()
	map.add("name", name)
	map.add("attr", attr)
	map.add("arg1_name", arg1)
	...
	functionGen(map.toMap)
}

We will need an additional function to handle these arguments called functionGen. It requires more complicated Tree-architecture design and a new Macros to implement.

package scalaMig

object Test extends TestBase {
  
  def Activation(name : String = null)() = {
    println("Hello")
  }

  def main(args : Array[String]) = {
    Activation("hi")
  }


}

class TestBase {
  def Activation(data : Option[Int] = None) : Unit = {
    println("Hello World")
  }
}

Glossary

References