This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Page tree
Skip to end of metadata
Go to start of metadata

Note: Following instructions are applicable for publishing Scala and Java package simultaneously as both bindings are inside the same package and built together.

Publishing Scala Packages to Nexus/Maven

Currently 3 Packages are published to Maven. `mxnet_full_2.11-{platform}`.

Compile Flags Used to build the Scala/Java package

The following flags are used to build the MXNet Backend(derived from mxnet pip compiler flags)


FLAGSMXNet-Scala OSX

MXNet-Scala Linux CPU

MXNet-Scala Linux GPU

USE_BLAS
01
USE_LAPACK111
USE_OPENCV1(OpenCV 3.4.1)1 (opencv 3.4.1)

1

(Last published opencv 3.4.1)

USE_S3111
USE_DIST_KVSTORE011
USE_OPENMP111
USE_CUDA

1

USE_CUDNN

00

1

(Last published with (CUDA 9.2)

USE_NCCL0

0

1

(Last published with CUDNN 7.0.5)

USE_MKLDNN
00
Java version

$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

ubuntu@ip:~$ java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (Zulu 8.20.0.5-linux64) (build 1.8.0_121-b15)
OpenJDK 64-Bit Server VM (Zulu 8.20.0.5-linux64) (build 25.121-b15, mixed mode)

ubuntu@ip:~$ java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (Zulu 8.20.0.5-linux64) (build 1.8.0_121-b15)
OpenJDK 64-Bit Server VM (Zulu 8.20.0.5-linux64) (build 25.121-b15, mixed mode)

Building on OS-X

Setup GPG Keys https://www.apache.org/dev/openpgp.html#key-gen-install-latest-gnupg

Clone MXNet repo and checkout the release branch

git clone https://github.com/$YOURFORK/incubator-mxnet ~/mxnet-$RELEASEVERSION
cd ~/mxnet-$RELEASEVERSION
git remote add upstream https://github.com/apache/incubator-mxnet
git fetch upstream v1.2.0 && git checkout -b $RELEASEVERISON --track upstream/v1.2.0
git submodule update --init --recursive

Install lapack, openblas, opencv packages

#try opencv3 next time you release
brew install lapack openblas opencv

Build MXNet backend on OSX

#This will build the MXNet backend with the tested configuration for publishing MXNet-Scala to Maven.
cd ~/mxnet-$RELEASEVERSION
scala-package/dev/compile-mxnet-backend.sh osx-x86_64-cpu ./

Maven Password Encryption for use in release

Now setup your maven create a master password and paste the encrypted the  more info Password Encryption

touch ~/.m2/settings-security.xml &&
mvn --encrypt-master-password

Paste the encrypted master-password you get from above in `~/.m2/settings-security.xml`

<settingsSecurity>
  <master>ENCRYPTEDMASTERPASSWORD</master>
</settingsSecurity>

Encrypt your password for `apache` login, note that this is not the passphrase of your gpg-key.

cp /usr/share/maven/conf/settings.xml ~/.m2/ && mvn --encrypt-password

Paste the below with your apache username and encrypted apache password updated under `<servers>`tag, this will used to publish to apache snapshot and staging in `~/.m2/settings.xml`.

    <!-- To publish a snapshot of some part of Maven -->
    <server>
      <id>apache.snapshots.https</id>
      <username>APACHEUSERNAME</username>
      <password>APACHEPASSWORDENCRYPTED</password>
    </server>
    <!-- To stage a release of some part of Maven -->
    <server>
      <id>apache.releases.https</id>
      <username>APACHEUSERNAME</username>
      <password>APACHEPASSWORDENCRYPTED</password>
    </server>

Maven Publish

Follow Maven Publish Link https://www.apache.org/dev/publishing-maven-artifacts.html to setup repo settings, etc., before you proceed

Maven Publish Process
#if you don't set this it will fail with the error "gpg: signing failed: Inappropriate ioctl for device"
export GPG_TTY=$(tty)
#build scalapkg
make scalapkg
#make sure everything is working as expected
make scalarelease-dryrun
# Releases to Apache Snapshot
make scaladeploy. This will upload to the snapshot repo
# Verify that the package has been uploaded in https://repository.apache.org/content/repositories/snapshots/
# Preparing the release will create the new tag, automatically checking in on your behalf.(We have disabled CheckIn)
make scalarelease-prepare
# this will upload to the staging repository and make it for testing
make scalarelease-perform

You have to close the repo for it to become available on the Staging Repo, When you close be sure to add the flags you used to build the package like below, You can find how to here

```

Built with flags for OPENCV, OPENBLAS, LAPACK, DIST_KVSTORE, S3
Flattened POM

```


Once you have tested the staged package, you can promote the package and it will become available on maven central, find details here

Build Linux-CPU

To build for linux-cpu platform, we use the AWS Deep Learning Base AMI that comes built with all the necessary 

libraries such as opencv/cudnn etc,.

Export/Import GPG Keys

First you need to export the gpg keys you created on Mac and import to the linux host

Install Maven, gnupg, gnupg2 & gnupg-agent

sudo apt-get install -y maven gnupg gnupg2 gnupg-agent && 
#soft link gnupg2 to gnupg, Update the below when you find a better solution
sudo mv /usr/bin/gpg /usr/bin/gpg1 && 
sudo ln -s /usr/bin/gpg2 /usr/bin/gpg

Export and ship gpg keys you created on the Mac

gpg --export-secret-keys -a xxx@apache.org > ~/Downloads/keys.asc
scp ~/Downloads/keys.asc ubuntu@host:~/keys.asc

Import and verify they look fine

gpg --import ~/keys.asc
#note the key id
gpg --list-keys

Update the gnupg-agent cachettl, it takes a long time to build on linux and linux-gpu instances and will time out during release if you don't baby sit the build

mkdir -p ~/.gnupg && touch ~/.gnupg/gpg-agent.conf
#set to 4 hours 
echo "default-cache-ttl 14400" > ~/.gnupg/gpg-agent.conf && 
echo "max-cache-ttl 14400" >> ~/.gnupg/gpg-agent.conf
#key id you go when you ran gpg --listkeys. This isn't working, commenting out
#echo "default-key KEYIDFROMABOVE" >> ~/.gnupg/gpg-agent.conf

Start gpg-agent to use the above

eval $(gpg-agent --daemon)

Build Backend

Follow steps to clone the repo and checkout the correct release branch

cd ~/mxnet-$RELEASEVERSION
./scala-package/dev/compile-mxnet-backend.sh linux-x86_64-gpu ./

Follow steps from OSX

Build on GPU

Follow all steps for Linux-CPU

Update the nccl location approriately in ~/incubator-mxnet/scala-package/dev/compile-backend.sh

```echo "ADD_LDFLAGS += -L/lib/nccl/cuda-9.2/lib```

cd ~/mxnet-$RELEASEVERSION
./scala-package/dev/compile-mxnet-backend.sh linux-x86_64-gpu ./

Using Packages in Staging Repository

Update your pom file to include the repository info like below:

Staging Repo
<repositories>
...  
<repository>
    <id>snapshot</id>
    <url>https://repository.apache.org/content/repositories/staging</url>
  </repository>
</repositories>

<dependencies>
  <dependency>
    <groupId>org.apache.mxnet</groupId>
    <artifactId>mxnet-full_2.11-osx-x86_64-cpu</artifactId>
    <version>1.2.0</version>
  </dependency>
</dependencies>

Maven Publish Debugging 

If you find that maven release-prepare fails with tag already exists
git push --force --delete ./ refs/heads/mxnet-parent_2.11-1.2.0
git tag -d mxnet-parent_2.11-1.2.0
If you face an issue because of gpg such as  ```Enter passphrase: gpg: gpg-agent is not available in this session```
sudo apt-get install gnupg2
sudo mv /usr/bin/gpg /usr/bin/gpg.1
sudo ln -s /usr/bin/gpg2 /usr/bin/gpg
gpg: signing failed: Inappropriate ioctl for device

```export GPG_TTY=$(tty)```

Compiling with USE_SIGNAL_HANDLER throws a segmentation fault on both OSX and Linux platforms
Maven Publish fails with: Return code is: 401, ReasonPhrase: Unauthorized.

Make sure you are using the encrypted password of your apache login and not gpg passphrase

Released Artifacts do not show up in Maven Search.

Verify that the released artifacts of the version you are publishing are synced to Apache release repository and Maven Central Repository, It might be that Maven Search is not picking up.

I see two repos in the staging repo for packages(example: OSX)

do a make clean and make scala clean before you build, it might be that the size of the Jar is too big and its splitting across two repos.

Apache Release Repo

https://repository.apache.org/content/repositories/releases/org/apache/mxnet/

Maven Central Repo

http://central.maven.org/maven2/org/apache/mxnet/mxnet-full_2.11-osx-x86_64-cpu/1.2.1/
http://central.maven.org/maven2/org/apache/mxnet/mxnet-full_2.11-linux-x86_64-cpu/1.2.1
http://central.maven.org/maven2/org/apache/mxnet/mxnet-full_2.11-linux-x86_64-gpu/1.2.1/


Release Checklist

  • Test Inference examples on macOS
  • Test Inference examples on Ubuntu CPU/GPU not on DL AMI
  • Test Training examples on Ubuntu GPU, not on DL AMI.
  • Test Training and Inference on Centos, not on DL AMI

Open Issues with the existing Packages

  • Mac OSX package built with OpenCV@2. – The default version of OpenCV installed by brew is 3.4.1
  • On Ubuntu, the standard is OpenCV 3.2 - but we used DL AMI that had 3.4.1



  • No labels