There is a new testing framework out there called Karate that is build on top of the popular Cucumber framework.
Karate makes it easy to script interactions with out web-services under test and verify the results. In addition it offers us a lot of useful features like parallelization, script/feature re-use, data-tables, JsonPath and XPath support, gherkin syntax, switchable staging-configurations and many others.
In the following tutorial we’ll be writing different scenarios and features for a real-world RESTful web-service to demonstrate some of its features.
Dependencies
REST Resource under Test
Our sample REST resource allows us to ..
-
List all existing users
-
Create a new user
-
Update a user
-
Delete a user
-
Get a user (by id)
-
Authenticate a user (returning an auth-token)
-
Access a secured resource (having a valid auth-token)
More detailed information about the REST resource specification can be found in "#Appendix_A:_REST_Resource_Specification[Appendix A: REST Resource Specification]", all details about its implementation in "#Appendix_B:_REST_Resource_Implementation[Appendix B: REST Resource Implementation]".
Setting up Karate with JUnit
We’re using JUnit as test framework here, so the only thing we need to setup to get our Karate tests running, is an entry class for JUnit, a configuration file for Karate in JavaScript format and our scenario file(s).
Test Structure
This is what our directory/file structure for the following testing examples looks like:
src/test
├── java
│ └── feature
│ └── user
│ └── UserManagementTest.java
└── resources
├── feature
│ ├── security
│ │ └── user-login.feature
│ └── user
│ └── user-management.feature
└── karate-config.js
JUnit Runner
This simple class allows us to run our tests with the JUnit framework.
We may fine-tune which scenarios to include and where to find it using additional configuration but if we keep this test classes’ package and the scenario file’s path in sync, we don’t need to add anything further than the following two lines to our feature.user.UserManagementTest class.
More detailed information about naming conventions can be found in the project’s wiki here, detailed information about adding Cucumber options are available here.
package feature.user;
import org.junit.runner.RunWith;
import com.intuit.karate.junit4.Karate;
@RunWith(Karate.class)
public class UserManagementTest {}
Karate Configuration
This is our Karate configuration file named karate-config.js.
We’re specifying the base-url for our tests here (our REST service runs on localhost, port 9000).
For more detailed information about the Karate configuration file, please consult the dedicated section in the project’s wiki.
function() {
return {
baseUrl: 'http://localhost:9000'
}
}
Scenarios
We’re now ready to write our scenarios and features down using Karate’s DSL.
As I am lazy I have written everything in one file in my project but separated into multiple parts in this tutorial:
Create and Fetch Users
In our first scenario, we’re going to do some CRUD operations so we’re going to ..
-
Verify that no user does exist in our application
-
Create a new user
-
Verify the created user
-
Verify we find the created user using his identifier
Feature: User management
Background:
* url baseUrl
Scenario: List existing users
Given path '/user'
When method GET
Then status 200
And match $ == []
Scenario: Create and receive new user
Given path '/user'
And request {id:'', name:'Fred', age:22, password:'abc123'}
When method POST
Then status 200
And match response == {id:'#uuid', name:'Fred', age:22, password:'abc123'}
And def user = response
Given path '/user/'+user.id
When method GET
Then status 200
And match $ == {id:'#(user.id)', name:'#(user.name)', age:#(user.age), password:'#(user.password)'}
Using Data Tables
Another nice features is to verify using data tables so this is our rewritten scenario executed for five users:
Feature: User management
Background:
* url baseUrl
Scenario Outline: Create multiple users and verify their id, name and age
Given path '/user'
And request {id:'', name:'<name>', age: <age>}
When method POST
Then status 200
And match $ == {id:'#uuid', name: '<name>', age: <age>}
And def user = response
Given path '/user/'+user.id
When method GET
And status 200
And match $ == {id:'#(user.id)', name:'#(user.name)', age:#(user.age)}
Examples:
| name | age |
| Tim | 42 |
| Liz | 16 |
| Selma | 65 |
| Ted | 12 |
| Luise | 19 |
Re-using Features
Another interesting feature is the possibility to re-use existing features by loading them into another feature.
In the following example there’s a secured resource that can only be accessed with a valid auth-token.
To obtain the auth-token, a user needs request a new auth-token using his credentials.
To re-use the login procedure, we’re storing it in a feature file named user-login.feature.
It’s important to know that we need to pass the user’s identifier and password to the feature and that the last line of the feature file specifies a variable named authToken that we may use then in our original feature file.
Feature: User management
Background:
* url baseUrl
Scenario: Login user
Given path '/user/login'
And request {id: '#(id)', password:'#(password)'}
When method POST
Then status 200
And match $ == {authToken:'#notnull'}
And def authToken = $.authToken
This is our scenario where a new created user tries to login, we’re verifying that the resource is not accessible without the security-token and then we’re calling the feature above to obtain the auth-token.
With this auth-token, we’re able to access the secured service.
Feature: User management
Background:
* url baseUrl
Scenario: Create and login user
Given path '/user'
And request {id:'', name:'Eleonore', age:31, password:'foobar'}
When method POST
Then status 200
And def user = response
Given path '/user/secured/date'
When method GET
Then status 403
Given path '/user/secured/date'
And def userLogin = call read('classpath:feature/security/user-login.feature') {id:'#(user.id)', password:'#(user.password)'}
And header Auth-Token = userLogin.authToken
When method GET
Then status 200
And match $ == {date:'#notnull'}
Using Assertions
We may use assertions to verify special conditions e.g. to assure non-functional requirements like the response-time:
Feature: User management
Background:
* url baseUrl
Scenario: Remove all users
Given path '/user'
When method DELETE
Then status 200
And assert responseTime < 1000
Given path '/user'
When method GET
Then status 200
And match $ == []
And assert responseTime < 1000
Running the Tests
We may run the tests now using our IDE of choice or using Maven in the command line.
I’m running the tests on GitHub Pipelines, please feel free to take a look at the full test execution there.
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running feature.user.UserManagementTest
13:02:44.279 [main] DEBUG com.intuit.karate.cucumber.CucumberRunner - init test class: class feature.user.UserManagementTest
13:02:44.390 [main] DEBUG com.intuit.karate.cucumber.CucumberRunner - loading feature: /data/project/karate-bdd-testing/target/test-classes/feature/user/user-management.feature
13:02:45.769 [main] DEBUG com.intuit.karate.LoggingFilter -
1 > GET http://localhost:9000/user
13:02:45.852 [qtp1279309678-23] INFO com.hascode.tutorial.UserResource - 0 users found
13:02:46.285 [main] DEBUG com.intuit.karate.LoggingFilter -
1 < 200
1 < Content-Length: 2
1 < Content-Type: application/json
1 < Date: Thu, 06 Apr 2017 11:02:45 GMT
1 < Server: Jetty(9.4.3.v20170317)
[]
[..]
1 > POST http://localhost:9000/user
1 > Content-Type: application/json
{"id":"","name":"Fred","age":22,"password":"abc123"}
13:02:46.481 [qtp1279309678-22] INFO com.hascode.tutorial.UserResource - new user User [name=Fred, id=5746BAF9-CAFE-4E37-8F94-145FBEB148B8, age=22, password=abc123] saved
13:02:46.484 [main] DEBUG com.intuit.karate.LoggingFilter -
1 < 200
1 < Content-Length: 88
1 < Content-Type: application/json
1 < Date: Thu, 06 Apr 2017 11:02:46 GMT
1 < Server: Jetty(9.4.3.v20170317)
{"age":22,"id":"5746BAF9-CAFE-4E37-8F94-145FBEB148B8","name":"Fred","password":"abc123"}
[..]
9 Scenarios (9 passed)
109 Steps (109 passed)
0m2.345s
Tests run: 118, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.36 sec
Results :
Tests run: 118, Failures: 0, Errors: 0, Skipped: 0
Running the tests in Eclipse IDE:
Tutorial Sources
Please feel free to download the tutorial sources from my GitHub repository, fork it there or clone it using Git:
git clone https://github.com/hascode/karate-bdd-testing.git
Resources
Is this really BDD?
No, the framework’s author, Peter Thomas has written a detailed article about this topic, please feel free to read: "Yes, Karate is not true BDD".
BDD Articles of mine
The following articles of mine are covering different aspects and frameworks for Behaviour Driven Development:
REST Testing Articles of mine
The following articles of mine are covering the topic of testing RESTful web-services:
Troubleshooting
-
"java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NoClassDefFoundError: org/eclipse/jetty/util/Decorator at com.hascode.tutorial.RestServer.main(RestServer.java:14) … 6 more Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.util.Decorator at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) … 7 more" → The dependency jersey-container-jetty-http uses an older version of jetty-util, adding an exclusion to this dependency solves the problem:
<dependency> <groupId>org.glassfish.jersey.containers</groupId> <artifactId>jersey-container-jetty-http</artifactId> <version>${jersey.version}</version> <exclusions> <exclusion> <groupId>org.eclipse.jetty</groupId> <artifactId>jetty-util</artifactId> </exclusion> </exclusions> </dependency>
-
"[ERROR] Failed to execute goal com.sebastian-daschner:jaxrs-analyzer-maven-plugin:0.14:analyze-jaxrs (default) on project karate-bdd-testing: Execution default of goal com.sebastian-daschner:jaxrs-analyzer-maven-plugin:0.14:analyze-jaxrs failed: Unable to load the mojo ‘analyze-jaxrs’ (or one of its required components) from the plugin ‘com.sebastian-daschner:jaxrs-analyzer-maven-plugin:0.14′: com.google.inject.ProvisionException: Guice provision errors:
[ERROR]
[ERROR] 1) No implementation for org.eclipse.aether.RepositorySystem was bound.
[ERROR] while locating com.sebastian_daschner.jaxrs_analyzer.maven.JAXRSAnalyzerMojo
[ERROR] at ClassRealm[plugin>com.sebastian-daschner:jaxrs-analyzer-maven-plugin:0.14, parent: sun.misc.Launcher$AppClassLoader@4e25154f]
[ERROR] while locating org.apache.maven.plugin.Mojo annotated with @com.google.inject.name.Named(value=com.sebastian-daschner:jaxrs-analyzer-maven-plugin:0.14:analyze-jaxrs)
[ERROR]
[ERROR] 1 error
[ERROR] role: org.apache.maven.plugin.Mojo
[ERROR] roleHint: com.sebastian-daschner:jaxrs-analyzer-maven-plugin:0.14:analyze-jaxrs
[ERROR] -> [Help 1]" – Please use Maven in version >= 3.2.1
Appendix A: REST Resource Specification
I have generated a plain-text description of the REST resource using the JAX-RS Analyzer Maven Plugin.
If you’re interested in other possibilities to derive documentation from existing JAX-RS web-services in AsciiDoc, Swagger or plain-text format, please feel free to read my tutorial: "Documenting RESTful Webservices in Swagger, AsciiDoc and Plain-Text with Maven and the JAX-RS Analyzer".
REST resources of karate-bdd-testing:
1.0.0
GET user:
Request:
No body
Response:
Content-Type: application/json
Status Codes: 200
Response Body: com.hascode.tutorial.UserResource$1
{}
POST user:
Request:
Content-Type: application/json
Request Body: com.hascode.tutorial.User
{"age":0,"id":"string","name":"string","password":"string"}
Response:
Content-Type: application/json
Status Codes: 200
Response Body: com.hascode.tutorial.User
{"age":0,"id":"string","name":"string","password":"string"}
DELETE user:
Request:
No body
Response:
Content-Type: */*
Status Codes: 200
POST user/login:
Request:
Content-Type: */*
Request Body: com.hascode.tutorial.UserResource$Credential
{"id":"string","password":"string"}
Response:
Content-Type: */*
Status Codes: 200
Response Body: java.lang.String
Status Codes: 401
GET user/secured/date:
Request:
No body
Header Param: Auth-Token, java.lang.String
Response:
Content-Type: application/json
Status Codes: 200
Response Body: java.lang.String
Status Codes: 403
GET user/{userId}:
Request:
No body
Path Param: userId, java.lang.String
Response:
Content-Type: application/json
Status Codes: 200
Response Body:
Appendix B: REST Resource Implementation
This is the implementation of our RESTful web-service under test.
Using a tool like Wiremock would have been easier than implementing a real application but I’d like to simulate the usage of Karate with a real application.
Dependencies
We need to add multiple dependencies to our project’s pom.xml here. It is important to handle the possible version conflict of jetty-utils here by excluding it from jersey-container-jetty-http – for more information take a look at the troubleshooting section.
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
<version>${jetty.version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-servlet</artifactId>
<version>${jetty.version}</version>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.core</groupId>
<artifactId>jersey-server</artifactId>
<version>${jersey.version}</version>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.containers</groupId>
<artifactId>jersey-container-servlet-core</artifactId>
<version>${jersey.version}</version>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.containers</groupId>
<artifactId>jersey-container-jetty-http</artifactId>
<version>${jersey.version}</version>
<exclusions>
<exclusion>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-util</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.media</groupId>
<artifactId>jersey-media-moxy</artifactId>
<version>${jersey.version}</version>
</dependency>
User Entity
This is our user entity, serialized via JAX-B, setters, getters, toString etc. ommitted..
package com.hascode.tutorial;
import javax.xml.bind.annotation.XmlRootElement;
@XmlRootElement
public class User {
private String name;
private String id;
private int age;
private String password;
}
User Resource
This is our JAX-RS based user resource storing users and credentials in a simple hashmap – it’s just for demonstration ;)
For the path "secured" we’re delegating to an sub-resource responsible for secured resources. If you’re interested in such tricks, please feel free to have a look at my article: "JAX-RS Server API Snippets".
package com.hascode.tutorial;
import java.math.BigInteger;
import java.security.SecureRandom;
import java.util.Collection;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
import javax.ws.rs.Consumes;
import javax.ws.rs.DELETE;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.core.GenericEntity;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.xml.bind.annotation.XmlRootElement;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@Path("/user")
public class UserResource {
private static final Logger LOG = LoggerFactory.getLogger(UserResource.class);
private static final Map<String, User> USER_STORE = new ConcurrentHashMap<>();
private static final Map<String, String> TOKEN_TO_USERID = new ConcurrentHashMap<>();
@XmlRootElement
static class Credential {
private String id;
private String password;
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getPassword() {
return password;
}
public void setPassword(String password) {
this.password = password;
}
}
@GET
@Produces(MediaType.APPLICATION_JSON)
public Response getAll() {
Collection<User> users = USER_STORE.values();
LOG.info("{} users found", users.size());
return Response.ok(new GenericEntity<Collection<User>>(users) {
}).build();
}
@Path("/{userId}")
@GET
@Produces(MediaType.APPLICATION_JSON)
public Response getUserById(@PathParam("userId") String userId) {
LOG.info("fetching user by user-id: {}", userId);
return Response.ok(USER_STORE.get(userId)).build();
}
@POST
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
public Response save(User user) {
user.setId(UUID.randomUUID().toString().toUpperCase());
USER_STORE.put(user.getId(), user);
LOG.info("new user {} saved", user);
return Response.ok(user).build();
}
@DELETE
public Response purgeUsers() {
LOG.info("removing all {} users", USER_STORE.size());
USER_STORE.clear();
return Response.ok().build();
}
@POST
@Path("/login")
public Response login(Credential credential) {
LOG.info("login user with id: '{}' and password '{}'", credential.getId(), credential.getPassword());
if (validCredential(credential)) {
LOG.info("login for user-id '{}' successful", credential.id);
String authToken = authorize(credential);
return Response.ok("{\"authToken\":\"" + authToken + "\"}").build();
}
return Response.status(401).build();
}
private String authorize(Credential credential) {
String code = new BigInteger(130, new SecureRandom()).toString(20);
TOKEN_TO_USERID.put(code, credential.getId());
return code;
}
private boolean validCredential(Credential credential) {
if (!USER_STORE.containsKey(credential.getId())) {
LOG.warn("no user with id {} known", credential.getId());
return false;
}
if (!USER_STORE.get(credential.id).getPassword().equals(credential.getPassword())) {
LOG.warn("credentials are invalid");
return false;
}
LOG.info("credentials are valid");
return true;
}
@Path("/secured")
public SecuredResource getSecuredResource() {
return new SecuredResource(TOKEN_TO_USERID);
}
}
Secured Resource
This resource allows to obtain the current date in a JSON format but an auth-token is required.
package com.hascode.tutorial;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.Map;
import javax.ws.rs.GET;
import javax.ws.rs.HeaderParam;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.Response.Status;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class SecuredResource {
private static final Logger LOG = LoggerFactory.getLogger(SecuredResource.class);
private final Map<String, String> tokenToUserid;
public SecuredResource(Map<String, String> tokenToUserid) {
this.tokenToUserid = tokenToUserid;
}
@GET
@Path("/date")
@Produces(MediaType.APPLICATION_JSON)
public Response getDate(@HeaderParam("Auth-Token") String authToken) {
LOG.info("trying to access secured resource with authToken '{}'", authToken);
if (!authorized(authToken)) {
LOG.warn("not authorized");
return Response.status(Status.FORBIDDEN).build();
}
return Response.ok(String.format("{\"date\":\"%s\"}",
LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyy-MM-dd")))).build();
}
private boolean authorized(String authToken) {
if (authToken == null || authToken.isEmpty()) {
LOG.warn("no auth-token given");
return false;
}
if (!tokenToUserid.containsKey(authToken)) {
LOG.warn("invalid auth-token given");
return false;
}
return true;
}
}
Application Starter
The following code initializes and starts an embedded Jetty server, running on port 9000.
package com.hascode.tutorial;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.servlet.ServletContextHandler;
import org.eclipse.jetty.servlet.ServletHolder;
public class App {
private Server jettyServer;
public App() {
ServletContextHandler context = new ServletContextHandler(ServletContextHandler.SESSIONS);
context.setContextPath("/");
jettyServer = new Server(9000);
jettyServer.setHandler(context);
ServletHolder jerseyServlet = context.addServlet(org.glassfish.jersey.servlet.ServletContainer.class, "/*");
jerseyServlet.setInitParameter("jersey.config.server.provider.classnames",
UserResource.class.getCanonicalName());
jerseyServlet.setInitOrder(0);
}
public void start() throws Exception {
jettyServer.start();
}
public void stop() throws Exception {
try {
jettyServer.stop();
} finally {
jettyServer.destroy();
}
}
public static void main(String[] args) throws Exception {
new App().run();
}
private void run() throws Exception {
try {
jettyServer.start();
jettyServer.join();
} finally {
jettyServer.destroy();
}
}
}
Jetty Logging Configuration
We’re adding the following jetty-logging.properties to our project:
org.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.StdErrLog
org.eclipse.jetty.LEVEL=OFF
com.hascode.LEVEL=INFO
Directory Structure
This is our final application’s directory structure:
src/main
├── java
│ └── com
│ └── hascode
│ └── tutorial
│ ├── App.java
│ ├── SecuredResource.java
│ ├── User.java
│ └── UserResource.java
└── resources
└── jetty-logging.properties
Running the REST-Service
We may run our REST service now using our IDE or in the command-line like this:
mvn exec:java -Dexec.mainClass=com.hascode.tutorial.App
Afterwards, our RESTful web-service is available at http://localhost:9000/ and we may obtain its WADL at http://localhost:9000/application.wadl.
Article Updates
-
2017-07-26: Link to SemaphoreCI article added.