MXNet-R release process

As a part of MXNet release process, R packages need to be built, tested and hosted on the S3 bucket, which acts as a CRAN-like repository for installing the mxnet-R package. Currently we host MXNet-R packages for MacOS and Windows. Details of building and testing of each package is described below.

Mac OS

For Mac, we currently maintain and distribute the binary for the CPU version of mxnet-R.

Building MXNet Source Code

Clone the MXNet GitHub repository and checkout the release branch -
- git clone --recursive https://github.com/apache/incubator-mxnet
Building the source code - (From here : http://mxnet.incubator.apache.org/versions/master/install/osx_setup.html )
- cp make/osx.mk config.mk
- echo "USE_BLAS = openblas" >> ./config.mk
  echo "ADD_CFLAGS += -I/usr/local/opt/openblas/include" >> ./config.mk
  echo "ADD_LDFLAGS += -L/usr/local/opt/openblas/lib" >> ./config.mk
  echo "ADD_LDFLAGS += -L/usr/local/lib/graphviz/" >> ./config.mk
- make -j $(nproc) USE_OPENCV=1 USE_BLAS=openblas

Build and install the R-package

Execute these commands in order from the MXNet home folder. This will generate a file called "mxnet_<version>.tgz" in your local folder

mkdir -p R-package/inst/libs
cp src/io/image_recordio.h R-package/src
cp -rf lib/libmxnet.so R-package/inst/libs
mkdir -p R-package/inst/include
cp -rf include/* R-package/inst/include
cp -rf 3rdparty/dmlc-core/include/* R-package/inst/include/
cp -rf 3rdparty/tvm/nnvm/include/* R-package/inst/include
Rscript -e "library(devtools); library(methods); options(repos=c(CRAN='https://cloud.r-project.org/'));install_deps(pkg='R-package', dependencies = TRUE)"
echo "import(Rcpp)" >> R-package/NAMESPACE
R CMD INSTALL R-package
Rscript -e "devtools::install_version('roxygen2',version='5.0.1', repos='https://cloud.r-project.org/',quiet=TRUE)"
Rscript -e "require(mxnet); mxnet:::mxnet.export('R-package'); warnings()"
Rscript -e "require(roxygen2); roxygen2::roxygenise('R-package'); warnings()"
R CMD INSTALL --build R-package

You can view the contents of the generated tgz file by: tar -tf mxnet_<version>.tgz

Verify the contents of the package and upload it in the corresponding S3 bucket. Once uploaded on the S3 bucket, we can install and verify using the following commands from an R console.

cran <- getOption("repos")
cran["dmlc"] <- "https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/R/CRAN/"
options(repos = cran)
install.packages("mxnet”)
library(mxnet)
a <- mx.nd.ones(c(2,3), ctx = mx.cpu())
b <- a * 2 + 1
b
packageVersion('mxnet')
# Should print the version number. eg : ‘1.4.0’

Windows

For Windows, we support CPU version as well as GPU version. We support 3 GPU versions- cu90, cu91, cu92
Each of the above packages needs to be build, tested and uploaded in the S3 which is mentioned in the official install instructions.

Building Windows MXNet-R from Source Code(CPU)

Clone the MXNet github repository and checkout the release branch

     git clone --recursive https://github.com/apache/incubator-mxnet

The --recursiveis to clone all the submodules used by MXNet. You will be editing the "/mxnet/R-package"folder.

Download prebuilt GPU-enabled MXNet libraries for Windows from Windows release for the release commit. If it is not available you will need to build MXNet from source on Windows using release commit . You will need mxnet_x64_vc14_cpu.7zand prebuildbase_win10_x64_vc14.7z
Create a folder called R-package/inst/libs/x64. MXNet supports only 64-bit operating systems, so you need the x64 folder.
Copy the following shared libraries (.dll files) into the R-package/inst/libs/x64folder:

libgcc_s_seh-1.dll
libgfortran-3.dll 
libopenblas.dll 
libquadmath-0.dll 
unzip.exe 
unzip32.dll 
vcomp140.dll 
wget.exe

These dlls can be found in prebuildbase_win10_x64_vc14/3rdparty
If you got the binaries for release branch then copy libmxnet.dll and libmxnet.lib from
mxnet_x64_vc14_cpu/build, mxnet_x64_vc14_cpu/lib into R-package/inst/libs/x64
or else copy libmxnet.dll and libmxnet.lib from the release folder which you should have obtained after building MXNet from source.

Copy the header files from dmlc, mxnet, mxshadowand nnvmfrom mxnet_x64_vc14_cpu/include and mxnet_x64_vc14_cpu/nvnm/include into ./R-package/inst/include if you are using rebuild binaries else copy the above header files from your source code. Assuming you have the source code in folder mxnet, these headers should be present at mxnet\include, mxnet\3rdparty\mshadow\, mxnet\3rdparty\dmlc-core\include, mxnet\3rdparty\tvm\nnvm\include
It should look like

./R-package/inst
└── include
    ├── dmlc
    ├── mxnet
    ├── mshadow
    └── nnvm

Make sure that R executable is added to your PATHin the environment variables. Running the whereRcommand at the command prompt should return the location.
Also make sure that Rtools is installed and the executable is added to your PATHin the environment variables.
Temporary patch - im2rec currently results in crashes during the build. Remove the im2rec.h and im2rec.cc files in R-package/src/ from cloned repository and comment out the two im2rec lines in R-package/src/mxnet.ccas shown below.

#include "./kvstore.h"
#include "./export.h"
//#include "./im2rec.h"
......
......
  DataIterCreateFunction::InitRcppModule();
  KVStore::InitRcppModule();
  Exporter::InitRcppModule();
//  IM2REC::InitRcppModule();
}

Now open the Windows CMD with admin rights and change the directory to the mxnetfolder(cloned repository). Then use the following commands to build R package:

echo import(Rcpp) > R-package\NAMESPACEecho import(methods) >> R-package\NAMESPACE
Rscript -e "install.packages('devtools', repos = 'https://cloud.r-project.org')"cd R-package
Rscript -e "library(devtools); library(methods); options(repos=c(CRAN='https://cloud.r-project.org')); install_deps(dependencies = TRUE)"cd ..

R CMD INSTALL --no-multiarch R-package

Rscript -e "require(mxnet); mxnet:::mxnet.export('R-package')"
rm R-package/NAMESPACE
Rscript -e "require(devtools); install_version('roxygen2', version = '5.0.1', repos = 'https://cloud.r-project.org/', quiet = TRUE)"
Rscript -e "require(roxygen2); roxygen2::roxygenise('R-package')"

R CMD INSTALL --build --no-multiarch R-package

The above step create a mxnet package for R. 
Now you can verify the package by opening R terminal and typing following commands.

library(mxnet)
a <- mx.nd.ones(c(2,3), ctx = mx.cpu())
b <- a * 2 + 1
b

Take the package and upload it in the specific S3 bucket.


"https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/R/CRAN/CPU”

Verify the Install the package from repository :


cran <- getOption("repos")
cran["dmlc"] <- "https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/R/CRAN/"
options(repos = cran)
install.packages("mxnet”)

Reverify the installation using 
library(mxnet)
a <- mx.nd.ones(c(2,3), ctx = mx.cpu())
b <- a * 2 + 1
b

Building Windows MXNet-R from Source Code(GPU)

For GPU, we support CUDA-80, 90, 91 and 92
First, we would require few dependencies to be installed based on the cuda version

Install Nvidia-driversif not installed. Latest driver based on your system configuration is recommended.
Install Microsoft Visual Studio( VS2017 is required by CUDA)
Install NVidia CUDA Toolkitaccording to versions you are trying to build(cu80, cu90, cu91 and cu92)
Download and install CuDNN(to provide a Deep Neural Network library). Latest version recommended.

Note: A pre-requisite to above softwares is Nvidia-drivers which we assume is installed.

After you have installed above software, continue with the following steps to build MXNet-R:

Clone the MXNet github repo.

The --recursiveis to clone all the submodules used by MXNet. You will be editing the "/mxnet/R-package"folder.

Download prebuilt GPU-enabled MXNet libraries for Windows from Windows releasefor the release commit. If it is not available you will need to build MXNet from source on Windows using release commit . You will need mxnet_x64_vc14_gpu_cuX.7zand prebuildbase_win10_x64_vc14.7zwhere X stands for CUDA toolkit version
Create a folder called R-package/inst/libs/x64. MXNet supports only 64-bit operating systems, so you need the x64 folder.
Copy the following shared libraries (.dll files) into the R-package/inst/libs/x64folder:

libgcc_s_seh-1.dll
libgfortran-3.dll 
libopenblas.dll 
libquadmath-0.dll 
unzip.exe 
unzip32.dll 
vcomp140.dll 
wget.exe

These dlls can be found in prebuildbase_win10_x64_vc14/3rdparty
If you got the binaries for release branch then copy libmxnet.dll and libmxnet.lib from
mxnet_x64_vc14_gpu_cuX/build, mxnet_x64_vc14_gpu_cuX/lib into R-package/inst/libs/x64
or else copy libmxnet.dll and libmxnet.lib from the release folder which you should have obtained after building MXNet from source.

Copy the header files from dmlc, mxnet, mxshadowand nnvmfrom mxnet_x64_vc14_gpu_cuX/include and mxnet_x64_vc14_gpu_cuX/nvnm/include into ./R-package/inst/include if you are using rebuild binaries else copy the above header files from your source code. Assuming you have the source code in folder mxnet, these headers should be present at mxnet\include, mxnet\3rdparty\mshadow\, mxnet\3rdparty\dmlc-core\include, mxnet\3rdparty\tvm\nnvm\include
It should look like

./R-package/inst
└── include
    ├── dmlc
    ├── mxnet
    ├── mshadow
    └── nnvm

Make sure that R executable is added to your PATHin the environment variables. Running the whereRcommand at the command prompt should return the location.
Also make sure that Rtools is installed and the executable is added to your PATHin the environment variables.
Temporary patch - im2rec currently results in crashes during the build. Remove the im2rec.h and im2rec.cc files in R-package/src/ from cloned repository and comment out the two im2rec lines in R-package/src/mxnet.ccas shown below.
Now open the Windows CMD with admin rights and change the directory to the mxnetfolder(cloned repository). Then use the following commands to build R package:

echo import(Rcpp) > R-package\NAMESPACEecho import(methods) >> R-package\NAMESPACE
Rscript -e "install.packages('devtools', repos = 'https://cloud.r-project.org')"cd R-package
Rscript -e "library(devtools); library(methods); options(repos=c(CRAN='https://cloud.r-project.org')); install_deps(dependencies = TRUE)"cd ..

R CMD INSTALL --no-multiarch R-package

Rscript -e "require(mxnet); mxnet:::mxnet.export('R-package')"
rm R-package/NAMESPACE
Rscript -e "require(devtools); install_version('roxygen2', version = '5.0.1', repos = 'https://cloud.r-project.org/', quiet = TRUE)"
Rscript -e "require(roxygen2); roxygen2::roxygenise('R-package')"

R CMD INSTALL --build --no-multiarch R-package

The above step create a mxnet package for R.
Now you can verify the package by opening R terminal and typing following commands.

library(mxnet)
a <- mx.nd.ones(c(2,3), ctx = mx.gpu())
b <- a * 2 + 1
b

Take the package and upload it in the specific S3 bucket.
Where X is 100, 90, 91 or 92 depended on the CUDA version.

Verify if installing the package from repository is working fine.

cran <- getOption("repos")
cran["dmlc"] <- "https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/R/CRAN/"options(repos = cran)
install.packages("mxnet”)

Reverify the installation using 
library(mxnet)
a <- mx.nd.ones(c(2,3), ctx = mx.gpu())
b <- a * 2 + 1
b

Linux

N/A

Page tree