This page describes how to setup PowerShell on Windows to support REEF builds.

Java

Make sure Java Development Kit 7 or 8 is installed and $JAVA_HOME points to its installation. Add the following to your PowerShell profile:

 

PowerShell Profile Snippet for Java
# Java aliases
if($Env:JAVA_HOME){
    Set-Alias javac $Env:JAVA_HOME\bin\javac.exe
    Set-Alias java $Env:JAVA_HOME\bin\java.exe
    Set-Alias jar $Env:JAVA_HOME\bin\jar.exe
    Set-Alias jps $Env:JAVA_HOME\bin\jps.exe
} else {
    Write-Host "JAVA_HOME is not set in the environment, verify Java installation and set JAVA_HOME" -ForegroundColor Red
}

Maven

Make sure Maven 3 is installed and $M2_HOME points to the installation. Add following to your PowerShell profile:

PowerShell Profile Snippet for Maven
# Setup maven
if($Env:M2_HOME){
    # Fast builds with no tests, but multithreading
    function mvn-mt{
        $cmd = "`'$Env:M2_HOME\bin\mvn.cmd`' -TC1 -DskipTests $args"
        Invoke-Expression "& $cmd"
    }
    function mvn{
        $cmd = "`'$Env:M2_HOME\bin\mvn.cmd`' $args"
        Invoke-Expression "& $cmd"
    }
} else {
    Write-Host "M2_HOME is not set in the environment, verify Maven installation and set M2_HOME" -ForegroundColor Red
}

Git(Hub)

Install GitHub Desktop. Launch the GUI client once. Launch the GitHub Shell once. Then add the following to your PowerShell profile:

PowerShell Profile Snippet for GitHub
# setup git
if(Test-Path ~\AppData\Local\GitHub){    
    . (Resolve-Path "$env:LOCALAPPDATA\GitHub\shell.ps1")
    . (Resolve-Path "$env:github_posh_git\profile.example.ps1")
}

Managing Pull Requests (Committers only)

As a committer, you frequently need to checkout pull requests in order to check them. Add the following set of functions to your profile to ease that process:

PowerShell Function to checkout pull requests
function REEF-Checkout-PullRequest{
    param
    (
        [Parameter(Position=0, Mandatory=$true, HelpMessage='The pull request number')]
        [int]$PR,
        [Parameter(Mandatory=$false, HelpMessage='The branch to check the PR out into')]
        [string]$B
    )

    $branch = if([string]::IsNullOrEmpty($B)) {"PR-$PR"} else {$B}
    Invoke-Expression "git fetch https://github.com/apache/reef pull/$PR/head:$branch"
    Invoke-Expression "git checkout $branch"
}
PowerShell Function to rebase pull request onto apache master
function REEF-Rebase-PullRequest{
    Invoke-Expression 'git fetch apache; git rebase -i apache/master'
}
PowerShell Function to merge pull request into master
function REEF-Merge-PullRequest{
    param
    (
        [Parameter(Position=0, Mandatory=$true, HelpMessage='The pull request number')]
        [int]$PR
    )
 
    $branch = {"PR-$PR"}
    Invoke-Expression "git checkout master"
    Invoke-Expression "git merge $branch"
}
PowerShell Function to push merged pull request into apache master and your local git repository master
function REEF-Push-PullRequest{
    Invoke-Expression 'git push apache master'
    Invoke-Expression 'git push'
}

With this sequence of functions in your profile, the process of reviewing and merging pull request will look as follows:

Checkout, review and merge pull request
REEF-Checkout-PullRequest N
# ... test pull request ...
REEF-Rebase-PullRequest
# git commit --amend to fix commit message
REEF-Merge-PullRequest N
# last chance to change your mind and not merge!
REEF-Push-PullRequest

Visual Studio

Install Visual Studio 2013, 2015, 2017. Then add the following to your PowerShell profile:

PowerShell Functions to Setup Visual Studio command line environment
# Setup Visual Studio
function Setup-VisualStudio{
    param
    (
        [Parameter(Mandatory=$true, HelpMessage='Version')]
        $Version
    )
    $Script = "vcvarsall.bat&set"
    if ($Version -eq "2013"){
        $VisualStudioPath = 'c:\Program Files (x86)\Microsoft Visual Studio 12.0\VC'
    }elseif ($Version -eq "2015"){
        $VisualStudioPath = 'c:\Program Files (x86)\Microsoft Visual Studio 14.0\VC'
    }elseif ($Version -eq "2017"){
        $VisualStudioPath = 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Auxiliary\Build'
        $Script = "vcvarsall.bat x64 & set"
    }else{
        throw "Please provide a Version of Visual Studio: 2013, 2015, 2017"
    }
    if(Test-Path $VisualStudioPath){
        Push-Location $VisualStudioPath
        cmd /c "$Script" |
        foreach {
            if ($_ -match "=") {
                $v = $_.split("=")
                Set-Item -Force -Path "ENV:\$($v[0])"  -Value "$($v[1])"
            }
        }
        Pop-Location
    } else {
        Write-Host "Visual Studio $Version not found at: $VisualStudioPath" -ForegroundColor Red
    }
}

With this in your profile you can start Visual Studio with:

Start Visual Studio
### 2013
Setup-VisualStudio 2013
### 2015
Setup-VisualStudio 2015

Building (and cleaning) REEF

Assuming $REEFSourcePath points to the right folder, the following two functions can be used to build and reliably clean REEF:

Build and Clean Functions
function Build-REEF{
    if($Env:REEFSourcePath){
        Invoke-Expression 'msbuild $Env:REEFSourcePath\lang\cs\Org.Apache.REEF.sln /p:Configuration="Release" /p:Platform="x64" /m'
    } else {
        Write-Host "You must set REEFSourcePath to the root of your enclistment to use Build-REEF command" -ForegroundColor Red
    }
}
function Clean-REEF{
    if($Env:REEFSourcePath){
        Invoke-Expression 'msbuild $Env:REEFSourcePath\lang\cs\Org.Apache.REEF.sln /p:Configuration="Release" /p:Platform="x64" /t:Clean /m'
        Get-ChildItem -Path $Env:REEFSourcePath\lang\cs\ -Recurse -Filter obj                | Remove-Item -Recurse
        Get-ChildItem -Path $Env:REEFSourcePath\lang\cs\ -Recurse -Filter bin                | Remove-Item -Recurse
        Get-ChildItem -Path $Env:REEFSourcePath\lang\cs\ -Recurse -Filter target             | Remove-Item -Recurse
        Get-ChildItem -Path $Env:REEFSourcePath\lang\cs\ -Recurse -Filter REEF_LOCAL_RUNTIME | Remove-Item -Recurse
        Get-ChildItem -Path $Env:REEFSourcePath\lang\cs\ -Recurse -Filter TestResults        | Remove-Item -Recurse
        Get-ChildItem -Path $Env:REEFSourcePath\lang\cs\ -Recurse -Filter packages           | Remove-Item -Recurse
    } else {
        Write-Host "You must set REEFSourcePath to the root of your enlistment to use Clean-REEF command" -ForegroundColor Red
    }
}

Verifying Release artifacts

We compute hashes for the releases using the GNU Core Utils. The functions below can be used to verify those hashes.

Hash Functions for PowerShell
function Verify-Hash{
    param
    (
        [Parameter(Mandatory=$true, HelpMessage='The .MD5 file.')]
        [string]$File,
        [Parameter(Mandatory=$true, HelpMessage='The Algorithm to use')]
        [string]$Algorithm
    ) 
    foreach ($line in (Get-Content $File)) {
        $fields = $line -split '\s+'
        $hash = $fields[0].Trim().ToUpper()
        $filename = $fields[1].Trim()
        if($filename.StartsWith("*")){
            $filename = $filename.Substring(1).Trim()
        }
        $computedHash = (Get-FileHash -Algorithm $Algorithm $filename).Hash.ToUpper()
        if($hash.Equals($computedHash)){
            Write-Host $filename, ": Passed"
        }else{
            Write-Host $filename, ": Not Passed"
            Write-Host "Read from file: ", $hash
            Write-Host "Computed:       ", $computedHash
        }
    }
}
function Verify-MD5{
    param
    (
        [Parameter(Position=0, Mandatory=$true, HelpMessage='The .MD5 file.')]
        [string]$File
    ) 
    Verify-Hash -Algorithm MD5 -File $File
}
function Verify-SHA512{
    param
    (
        [Parameter(Position=0, Mandatory=$true, HelpMessage='The .SHA512 file.')]
        [string]$File
    ) 
    Verify-Hash -Algorithm SHA512 -File $File
}
  • No labels