You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 19 Next »


MXNet-Scala is a Scala binding for the MXNet Deep Learning framework. It currently supports Training and Inference

MXNet-Scala Modules

ModulePackage GeneratedDependenciesDescription
  1. assembly
  1. mxnet-full-parent_2.11
  2. mxnet-full_2.11-{platform}
  1. mxnet-core_2.11
  2. libmxnet-scala-{platform}

2. core

mxnet-core_2.11

  1. mxnet-init_2.11
  2. mxnet-macros_2.11

3. macros

mxnet-macros_2.11

  1. mxnet-init_2.11
  2. libmxnet-init-scala-{platform}

4. init

mxnet-init_2.11

 

5. init-native

  1. mxnet-scala-init-native-parent
  2. libmxnet-init-scala-osx-x86_64

  3. libmxnet-init-scala-linux-x86_64

mxnet-init_2.11


6. native

  1. mxnet-parent_2.11
  2. libmxnet-scala-{platform}

 mxnet-core_2.11


7. infer

mxnet-infer_2.11

  1. mxnet-core_2.11
  2. libmxnet-scala-{platform}

8. spark

mxnet-spark_2.11

  1. mxnet-core_2.11
  2. libmxnet-scala-{platform}
  3. spark-mllib_2.11

9. examples

mxnet-examples_2.11

  1. mxnet-core_2.11
  2. mxnet-infer_2.11

Publishing Scala Packages to Nexus/Maven

Currently 3 Packages from the above generated are published to Maven. `mxnet_full_2.11-{platform}`.

The following flags are used to build the MXNet Backend(derived from mxnet pip compiler flags)


FLAGSMXNet-Scala OSX

MXNet-Scala Linux CPU

MXNet-Scala Linux GPU
USE_BLAS
0 
USE_LAPACK11 1
USE_OPENCV11 (opencv 3.4.1) 1
USE_S311 1
USE_DIST_KVSTORE01 1
USE_OPENMP 11
USE_CUDA

1

USE_CUDNN

 01
USE_NCCL 0

1

USE_MKLDNN
00
Java version

$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

ubuntu@ip:~$ java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (Zulu 8.20.0.5-linux64) (build 1.8.0_121-b15)
OpenJDK 64-Bit Server VM (Zulu 8.20.0.5-linux64) (build 25.121-b15, mixed mode)


Building on OS-X

Setup GPG Keys https://www.apache.org/dev/openpgp.html#key-gen-install-latest-gnupg 

Export/Import GPG Keys

gpg --export-secret-keys -a xxx@apache.org > ~/Downloads/keys.asc
gpg --import /tmp/keys.asc
#list the keys to make sure they are there.
gpg --list-keys
#you will need this to enter your gpg-key password
sudo apt install gnupg-agent

Build MXNet backend on OSX

Clone MXNet repo and checkout the release branch

export YOURFORK=whatever
export RELEASE_VERISON=1.2.0
git clone https://github.com/$YOURFORK$/incubator-mxnet
git remote add upstream https://github.com/apache/incubator-mxnet
git fetch upstream $RELEASE_VERSION && git checkout -b $RELEASE_VERSION --track upstream/$RELEASE_VERSION

Install necessary libraries

brew install lapack
brew install openblas

Build MXNet Backend

#This will build the MXNet backend with the tested configuration for publishing MXNet-Scala to Maven.
scala-package/dev/compile-mxnet-backend.sh osx-x86_64-cpu ./

Maven Publish

Follow Maven Publish Link https://www.apache.org/dev/publishing-maven-artifacts.html to setup repo settings, etc., before you proceed

Maven Publish Process
#if you don't set this it will fail with the error "gpg: signing failed: Inappropriate ioctl for device"
export GPG_TTY=$(tty)

#make sure everything is working as expected
make scalarelease-dryrun
# Releases to Apache Snapshot
make scaladeploy 
# Verify that the package has been uploaded in https://repository.apache.org/content/repositories/snapshots/
# Preparing the release will create the new tag, automatically checking in on your behalf.(We have disabled CheckIn)
make scalarelease-prepare
# this will upload to the staging repository and make it ready for closing
make scalarelease-perform

Build MXNet backend on Linux-CPU

To build for linux-cpu platform, we use the AWS Deep Learning Base AMI that comes built with all the necessary 

libraries such as opencv/cudnn etc,.

Install Maven on Ubuntu

sudo apt-get install -y maven
Debugging Info

git push --force --delete ./ refs/heads/mxnet-parent_2.11-1.2.0
git tag -d mxnet-parent_2.11-1.2.0

If you face an issue because of gpg such as  ```Enter passphrase: gpg: gpg-agent is not available in this session```

sudo apt-get install gnupg2
sudo mv /usr/bin/gpg /usr/bin/gpg.1
sudo ln -s /usr/bin/gpg2 /usr/bin/gpg

gpg: signing failed: Inappropriate ioctl for device

```export GPG_TTY=$(tty)```

Note: compiling with USE_SIGNAL_HANDLER throws a segmentation fault on both OSX and Linux platforms

Publishing to Maven

export GPG_TTY=$(tty) make scalapkg make scalarelease-dryrun
  • No labels