Versions Compared

Old Version 28

changes.mady.by.user Naveen Swamy

Saved on

compared with

New Version 29

changes.mady.by.user Naveen Swamy

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

ModulePackage GeneratedDependenciesDescription
  1. assembly
  1. mxnet-full-parent_2.11
  2. mxnet-full_2.11-{platform}
  1. mxnet-core_2.11
  2. libmxnet-scala-{platform}

2. core

mxnet-core_2.11

  1. mxnet-init_2.11
  2. mxnet-macros_2.11

3. macros

mxnet-macros_2.11

  1. mxnet-init_2.11
  2. libmxnet-init-scala-{platform}

4. init

mxnet-init_2.11


5. init-native

  1. mxnet-scala-init-native-parent
  2. libmxnet-init-scala-osx-x86_64

  3. libmxnet-init-scala-linux-x86_64

mxnet-init_2.11


6. native

  1. mxnet-parent_2.11
  2. libmxnet-scala-{platform}

 mxnet-core_2.11


7. infer

mxnet-infer_2.11

  1. mxnet-core_2.11
  2. libmxnet-scala-{platform}

8. spark

mxnet-spark_2.11

  1. mxnet-core_2.11
  2. libmxnet-scala-{platform}
  3. spark-mllib_2.11

9. examples

mxnet-examples_2.11

  1. mxnet-core_2.11
  2. mxnet-infer_2.11

Publishing Scala Packages to Nexus/Maven

Currently 3 Packages from the above generated are published to Maven. `mxnet_full_2.11-{platform}`.

Compile Flags Used to build the Scala package

The following flags are used to build the MXNet Backend(derived from mxnet pip compiler flags)

...

MXNet-Scala Linux CPU

...

USE_CUDNN

...

1

...

$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

...

ubuntu@ip:~$ java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (Zulu 8.20.0.5-linux64) (build 1.8.0_121-b15)
OpenJDK 64-Bit Server VM (Zulu 8.20.0.5-linux64) (build 25.121-b15, mixed mode)



Building on OS-X

Setup GPG Keys https://www.apache.org/dev/openpgp.html#key-gen-install-latest-gnupg

Clone MXNet repo and checkout the release branch

Code Block
languagebash
git clone https://github.com/$YOURFORK/incubator-mxnet ~/mxnet-$RELEASEVERSION
cd ~/mxnet-$RELEASEVERSION
git remote add upstream https://github.com/apache/incubator-mxnet
git fetch upstream v1.2.0 && git checkout -b $RELEASEVERISON --track upstream/v1.2.0
git submodule update --init --recursive

Install lapack, openblas, opencv packages

Code Block
languagebash
#try opencv3 next time you release
brew install lapack openblas opencv@2

Build MXNet backend on OSX

Code Block
languagebash
#This will build the MXNet backend with the tested configuration for publishing MXNet-Scala to Maven.
cd ~/mxnet-$RELEASEVERSION
scala-package/dev/compile-mxnet-backend.sh osx-x86_64-cpu

Maven Password Encryption for use in release

Now setup your maven create a master password and paste the encrypted the  more info Password Encryption

Code Block
languagebash
touch ~/.m2/settings-security.xml &&
mvn --encrypt-master-password

Paste the encrypted master-password you get from above in `~/.m2/settings-security.xml`

Code Block
languagexml
<settingsSecurity>
  <master>ENCRYPTEDMASTERPASSWORD</master>
</settingsSecurity>

Encrypt your password for `apache` login

Code Block
languagebash
cp /usr/share/maven/conf/settings.xml ~/.m2/ && mvn --encrypt-password

Paste the below with your apache username and encrypted apache password updated under `<servers>`tag, this will used to publish to apache snapshot and staging in `~/.m2/settings.xml`.

Code Block
languagexml
    <!-- To publish a snapshot of some part of Maven -->
    <server>
      <id>apache.snapshots.https</id>
      <username>APACHEUSERNAME</username>
      <password>APACHEPASSWORDENCRYPTED</password>
    </server>
    <!-- To stage a release of some part of Maven -->
    <server>
      <id>apache.releases.https</id>
      <username>APACHEUSERNAME</username>
      <password>APACHEPASSWORDENCRYPTED</password>
    </server>

Maven Publish

Follow Maven Publish Link https://www.apache.org/dev/publishing-maven-artifacts.html to setup repo settings, etc., before you proceed

Code Block
titleMaven Publish Process
#if you don't set this it will fail with the error "gpg: signing failed: Inappropriate ioctl for device"
export GPG_TTY=$(tty)
#build scalapkg
make scalapkg
#make sure everything is working as expected
make scalarelease-dryrun
# Releases to Apache Snapshot
make scaladeploy. This will upload to the snapshot repo
# Verify that the package has been uploaded in https://repository.apache.org/content/repositories/snapshots/
# Preparing the release will create the new tag, automatically checking in on your behalf.(We have disabled CheckIn)
make scalarelease-prepare
# this will upload to the staging repository and make it for testing
make scalarelease-perform

You have to close the repo for it to become available on the Staging Repo, find details here

Once you have tested the staged package, you can promote the package and it will become available on maven central, find details here

Build Linux-CPU

To build for linux-cpu platform, we use the AWS Deep Learning Base AMI that comes built with all the necessary 

libraries such as opencv/cudnn etc,.

Export/Import GPG Keys

First you need to export the gpg keys you created on Mac and import to the linux host

Install Maven, gnupg, gnupg2 & gnupg-agent

Code Block
languagebash
sudo apt-get install -y maven gnupg gnupg2 gnupg-agent && 
#soft link gnupg2 to gnupg, Update the below when you find a better solution
sudo mv /usr/bin/gpg /usr/bin/gpg.1 && 
sudo ln -s /usr/bin/gpg2 /usr/bin/gpg

Export and ship gpg keys you created on the Mac

Code Block
languagebash
gpg --export-secret-keys -a xxx@apache.org > ~/Downloads/keys.asc
scp ~/Downloads/keys.asc ubuntu@host:~/keys.asc

Import and verify they look fine

Code Block
languagebash
gpg --import ~/keys.asc
#note the key id
gpg --list-keys

Update the gnupg-agent cachettl, it takes a long time to build on linux and linux-gpu instances and will time out during release if you don't baby sit the build

Code Block
languagebash
mkdir -p ~/.gnupg && touch ~/.gnupg/gpg-agent.conf
#set to 4 hours 
echo "default-cache-ttl 14400" > ~/.gnupg/gpg-agent.conf && 
echo "max-cache-ttl 14400" >> ~/.gnupg/gpg-agent.conf
#key id you go when you ran gpg --listkeys
echo "default-key KEYIDFROMABOVE" >> ~/.gnupg/gpg-agent.conf

Start gpg-agent to use the above

Code Block
languagebash
eval $(gpg-agent --daemon)

Build Backend

Follow steps to clone the repo and checkout the correct release branch

Code Block
languagebash
cd ~/mxnet-$RELEASEVERSION
./scala-package/dev/compile-mxnet-backend.sh linux-x86_64-gpu ./

Follow steps from OSX

Build on GPU

Follow all steps for Linux-CPU

Update the nccl location approriately in ~/incubator-mxnet/scala-package/dev/compile-backend.sh

```echo "ADD_LDFLAGS += -L/lib/nccl/cuda-9.0/lib```

Code Block
languagebash
cd ~/mxnet-$RELEASEVERSION
./scala-package/dev/compile-mxnet-backend.sh linux-x86_64-gpu ./

Using Packages in Staging Repository

Update your pom file to include the repository info like below:

Code Block
languagexml
titleStaging Repo
  <repository>
    <id>snapshot</id>
    <url>https://repository.apache.org/content/repositories/staging</url>
  </repository>
</repositories>

<dependencies>
  <dependency>
    <groupId>org.apache.mxnet</groupId>
    <artifactId>mxnet-full_2.11-osx-x86_64-cpu</artifactId>
    <version>1.2.0</version>
  </dependency>
</dependencies

Maven Publish Debugging 

If you find that maven release-prepare fails with tag already exists
Code Block
languagebash
git push --force --delete ./ refs/heads/mxnet-parent_2.11-1.2.0
git tag -d mxnet-parent_2.11-1.2.0
If you face an issue because of gpg such as  ```Enter passphrase: gpg: gpg-agent is not available in this session```
Code Block
sudo apt-get install gnupg2
sudo mv /usr/bin/gpg /usr/bin/gpg.1
sudo ln -s /usr/bin/gpg2 /usr/bin/gpg
gpg: signing failed: Inappropriate ioctl for device

```export GPG_TTY=$(tty)```

...

Release Process