This wiki captures some of the points/features which Karate BDD Test Framework offers. The basic idea of creating this is to showcase how a BDD automation framework can enhance/complement our existing Automation Tests.


  • What is Karate?

Karate is a Test Automation Framework based on Cucumber which is a BDD framework.

  • What is BDD Testing?

BDD Testing is a testing approach based on Behavioural Driven Development. It uses Given ,When ,Then statements to create test scenario's. Example:

TEST CASE-1
Feature: Get List of actions based on the NameSpace
Scenario: As a user I want to get the list of actions available for the given namespace
* def path = '/api/v1/namespaces/'+nameSpace+'/actions?limit=30&skip=0'
Given url BaseUrl+path
And header Authorization = Auth
And header Content-Type = 'application/json'
When method get
Then status 200
And def json = response

The above example tests that the List Actions API's is returning a 200 ok . A lot more assertions can be put as per the requirements. Karate has some very good in built functions for 
asserting the nested JSON's


TEST CASE-2(This example shows a simple smoke test on all the wsk user functions)
Feature: This feature file will test all the wsk functions
Background: 
* configure ssl = true
* def nameSpace = 'guest'
* def params = '?blocking=true&result=false'
* def scriptcode = call read('classpath:com/karate/openwhisk/functions/hello-world.js')
* def base64encoding = read('classpath:com/karate/openwhisk/utils/base64.js')
Scenario: TC01-As a user I want to all the wsk functions available to the user and check if they give the proper response
# Get User Auth
* def getNSCreds = call read('classpath:com/karate/openwhisk/wskadmin/get-user.feature') {nameSpace:'#(nameSpace)'}
* def result = getNSCreds.result
* def Auth = base64encoding(result)
* print "Got the Creds for the guest user"
* print Auth

# Create an Action .Create an action for the above defined guest name
#* def createAction = call read('classpath:com/karate/openwhisk/wskactions/create-action.feature') {script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)', actionName:'Dammyyy'}
* def createAction = call read('classpath:com/karate/openwhisk/wskactions/create-action.feature') {script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
* def actionName = createAction.actName
* print actionName
* print "Successfully Created an action"

# Get Action Details
* def actionDetails = call read('classpath:com/karate/openwhisk/wskactions/get-action.feature') {nameSpace:'#(nameSpace)' ,Auth:'#(Auth)',actionName:'#(actionName)'}
* print "Successfully got the action details"

#Invoke Action
* def invokeAction = call read('classpath:com/karate/openwhisk/wskactions/invoke-action.feature') {params:'#(params)',requestBody:'',nameSpace:'#(nameSpace)' ,Auth:'#(Auth)',actionName:'#(actionName)'}
* def actID = invokeAction.activationId
* print = "Successfully invoked the action"
* def webhooks = callonce read('classpath:com/karate/openwhisk/utils/sleep.feature') {sheepCount:'20'}

#Get Activation details
* def getActivationDetails = call read('classpath:com/karate/openwhisk/wskactions/get-activation-details.feature') { activationId: '#(actID)' ,Auth:'#(Auth)'}
* print "Successfully pulled the activation details"

# Update Action
* def updateAction = call read('classpath:com/karate/openwhisk/wskactions/update-action.feature') {actionName:'#(actionName)',script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
* def actionName = createAction.actName
* print actionName
* print "Successfully updated the action"

# List Action
* def listActions = call read('classpath:com/karate/openwhisk/wskactions/list-action.feature') {nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
* print "Successfully pulled up the list of actions"

# Delete Action
* def deleteAction = call read('classpath:com/karate/openwhisk/wskactions/delete-action.feature') {actionName:'#(actionName)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
* print "Successfully deleted the action"


  • Some key features of BDD testing approach:

1) Shifting from thinking in “tests” to thinking in “behavior”
2) Collaboration between Business stakeholders, Business Analysts, QA Team and developers
3) Ubiquitous language, it is easy to describe
4) Driven by Business Value
5) Extends Test Driven Development (TDD) by utilizing natural language that non technical stakeholders can understand
6) It enables to create test cases even before development starts and also helps bridging the Gap between what was required and what was actually delivered.


  • Why we should consider doubling our test-coverage (and thus effort)?

As OpenWhisk is an opensource Serverless Platform, it offers user a wide range of use cases. If we implement a BDD Testing Framework it would give us an ability to create
various test flows/scenarios which mimics the user behaviour. This might help us uncover issues which we might not catch in Unit/System level tests.
Also as writing tests in Karate does not require in-depth coding knowledge we can expect a lot more contributions in terms of testing. While our existing test cases can serve as Unit/System levels tests,
the Karate Framework could focus more on testing Business Logics/Work Flows.
Also,now that Karate offers Gatling support we can utilize the test feature files for Performance Test Scenarios.


  • How do these tests relate to the other system tests we have written in Scala today?

We have already covered most of the test cases in the Karate Framework and are in Progress to create the other test cases as well. Also we are creating units/modules derived from the Scala based Tests
which is a one time effort as those units can be utilised in any of the test scenarios in future.
Also keeping in mind a testers perspective we have structured/grouped the tests in the way it can be tested. Like Smoke Tests(These would just contain the basic tests which would ensure only that that
system is working fine, Regression Tests (Covering In-depth Testing)


  • Below is the relation between a test case written in Scala and Karate for your reference .The two tests give the same result.

  1. Test Case from (https://github.com/apache/incubator-openwhisk/blob/master/tests/src/test/scala/system/basic/WskRestBasicTests.scala)

    Scala Test Case
    behavior of "Wsk REST"
it should "reject creating duplicate entity" in withAssetCleaner(wskprops) { (wp, assetHelper) =>
val name = "testDuplicateCreate"
assetHelper.withCleaner(wsk.trigger, name) { (trigger, _) =>
trigger.create(name)
}
assetHelper.withCleaner(wsk.action, name, confirmDelete = false) { (action, _) =>
action.create(name, defaultAction, expectedExitCode = Conflict.intValue)
}
}

       2. Test Case from (https://github.com/apache/incubator-openwhisk/blob/9d87202a9f91d6274b1922dd125fc321d1d92a3a/tests/functional/karate_tests/src/test/java/com/karate/openwhisk/actionbasictests/action-basic-tests.feature)


Karate Test Case
Scenario: As a user i want to verify the creation of duplicate entity
#create action.
* def createAction = call read('classpath:com/karate/openwhisk/wskactions/create-action.feature') {script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
* def actionName = createAction.actName
* match createAction.responseStatusCode == 200
* print "Successfully Created an action"

#recreate the duplicate action with the same name
* def createAction = call read('classpath:com/karate/openwhisk/wskactions/create-action.feature') {script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)' ,actionName:'#(actionName)'}
* def actionName = createAction.actName
* match createAction.responseStatusCode == 409
* match createAction.response.error == "resource already exists"




Relevant Links:

Karate PR LINK→ https://github.com/apache/incubator-openwhisk/pull/3956

DISCUSSION THREAD→ https://lists.apache.org/thread.html/0237a1a9e0a7bb1de5d16762afecf814057f1e5d04da1934991a4120@%3Cdev.openwhisk.apache.org%3E

KARATE REPO→ https://github.com/intuit/karate

CUCUMBER BDD→ https://cucumber.io/

  • No labels