Transforming JSON Structures with Java and JOLT
January 29th, 2017 by Micha KopsWhen it comes to web-services (especially RESTful web-services), configuration files or other data-descriptors, the JavaScript Object Notation, short: JSON is often used as format of choice.
As the task arises to transform these JSON structures, I have seen a variety of different approaches from converting JSON into XML using JAX-B, applying XSLT, transforming into JSON again on the one hand to loading JSON structures into Hash-maps of Hash-maps (..) and manually removing single elements from these collections on the other hand.
JOLT is a library to make this task easier for us: It allows us to note down different types of transformations in a specification file in JSON syntax and to apply this specification to given JSON structures with ease.
In the following short tutorial I’d like to demonstrate how to use this library by transforming JSON structures programmatically with a few lines of code, by filtering a RESTful web-service using JOLT and a servlet filter and finally by adding a short example using the Apache Camel framework.
Contents
JOLT Schema Syntax
Jolt allows the following transformations that may be combined to generate the result we need. Please consult the linked JavaDocs for more details about the concrete syntax. In the following examples, we’ll be using only one simple remove command to remove elements from a given JSON structure.
- shift: copies data from input to output tree (Syntax Documentation)
- default: applies default values to the tree (Syntax Documentation)
- remove: removes nodes from the tree (Syntax Documentation)
- sort: sorts map key values alphabetically (Syntax Documentation)
- cardinality: adjusts the cardinality of input data (Syntax Documentation)
In addition, custom Java implementations may be used, they simply need to implement Transform or ContextualTransform.
Programmatic Transformation
In our first, simple example, we’re loading our transformer specification from a JSON file in the classpath and apply the transformation to a JSON structure read from another file.
Finally we’re printing the result to the screen.
Dependencies
We just need to add two dependencies to our project’s pom.xml (using Maven): jolt-core and jolt-utils:
<dependency> <groupId>com.bazaarvoice.jolt</groupId> <artifactId>jolt-core</artifactId> <version>0.1.0</version> </dependency> <dependency> <groupId>com.bazaarvoice.jolt</groupId> <artifactId>json-utils</artifactId> <version>0.1.0</version> </dependency>
JSON Structure
This is the JSON structure we’ll be using for all following demonstrations. In all examples, we’re reducing the structure by removing the page, date, startTime and endTime elements.
[ { "uuid":"cac40601-ffc9-4fd0-c5a1-772ac65f0587", "pageId":123456, "location":"Berlin", "maxParticipants":10, "page":{ "indexable":true, "rootLevel":false, "homePage":false, "latestVersion":true, "current":true, "deleted":false, "draft":false, "unpublished":false, "versionCommentAvailable":false, "persistent":true, "new":false }, "date":"15.02.17", "startTime":"09:00", "endTime":"16:30", "start":1487145600000, "end":1487172600000, "hasStartTime":true, "hasEndTime":true, "eventPageUrl":"/some/url", "categories":[ { "label":"Test", "id":123, "belongsToEvents":true } ], "participants":[ ], "isCurrentUserParticipating":false, "fullTitle":"Test Page" } ]
JOLT Schema Specification
This is our transformation schema. In this schema we may specify one or multiple operations using JSON notation.
In this schema we’re just applying a remove operation to every sub-node in a given JSON structure:
[ { "operation": "remove", "spec": { "*": { "page":"", "date":"", "startTime":"", "endTime":"" } } } ]
Application
Finally is this our application that load the transformation specs and the JSON input from the classpath, applies the transformation and prints the result to STDOUT:
package com.hascode.tutorial; import java.util.List; import com.bazaarvoice.jolt.Chainr; import com.bazaarvoice.jolt.JsonUtils; public class CustomJsonTransformer { public static void main(String[] args) throws Exception { List<Object> specs = JsonUtils.classpathToList("/spec.json"); Chainr chainr = Chainr.fromSpec(specs); Object inputJSON = JsonUtils.classpathToObject("/input.json"); Object transformedOutput = chainr.transform(inputJSON); System.out.println(JsonUtils.toPrettyJsonString(transformedOutput)); } }
Running the Example
We may now run our application using our IDE or directly in the command-line using Maven:
$ mvn clean compile exec:java -Dexec.mainClass=com.hascode.tutorial.CustomJsonTransformer [..] [ { "uuid" : "cac40601-ffc9-4fd0-c5a1-772ac65f0587", "pageId" : 123456, "location" : "Berlin", "maxParticipants" : 10, "start" : 1487145600000, "end" : 1487172600000, "hasStartTime" : true, "hasEndTime" : true, "eventPageUrl" : "/some/url", "categories" : [ { "label" : "Test", "id" : 123, "belongsToEvents" : true } ], "participants" : [ ], "isCurrentUserParticipating" : false, "fullTitle" : "Test Page" } ] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------
Filtering a RESTful Webservice using Servlet Filters
In the next example, we’re setting up a JAX-RS compliant RESTful web-service using an embedded Jetty server.
This endpoint sends the JSON structure in a GET request and a JOLT empowered servlet filter transforms this output by applying a transformation.
Dependencies
We’re adding a bunch of dependencies for Jersey (JAX-RS), Servlet-Container, Servlet-API and so on..
<properties> <jetty.version>9.4.1.v20170120</jetty.version> <jersey.version>2.25.1</jersey.version> </properties> <dependency> <groupId>org.eclipse.jetty</groupId> <artifactId>jetty-server</artifactId> <version>${jetty.version}</version> </dependency> <dependency> <groupId>org.eclipse.jetty</groupId> <artifactId>jetty-servlet</artifactId> <version>${jetty.version}</version> </dependency> <dependency> <groupId>org.glassfish.jersey.core</groupId> <artifactId>jersey-server</artifactId> <version>${jersey.version}</version> </dependency> <dependency> <groupId>org.glassfish.jersey.containers</groupId> <artifactId>jersey-container-servlet-core</artifactId> <version>${jersey.version}</version> </dependency> <dependency> <groupId>org.glassfish.jersey.containers</groupId> <artifactId>jersey-container-jetty-http</artifactId> <version>${jersey.version}</version> <exclusions> <exclusion> <groupId>org.eclipse.jetty</groupId> <artifactId>jetty-util</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.glassfish.jersey.media</groupId> <artifactId>jersey-media-moxy</artifactId> <version>${jersey.version}</version> </dependency>
REST Resource
This is our REST resource reading in our JSON file from the example above and sending it in a GET-request:
package com.hascode.tutorial; import java.io.IOException; import java.net.URISyntaxException; import java.nio.file.Files; import java.nio.file.Paths; import javax.ws.rs.GET; import javax.ws.rs.Path; import javax.ws.rs.Produces; import javax.ws.rs.core.MediaType; import javax.ws.rs.core.Response; @Path("/") public class SampleRestResource { @GET @Path("/data") @Produces(MediaType.APPLICATION_JSON) public Response getData() { try { String json = new String(Files.readAllBytes(Paths.get(getClass().getClassLoader().getResource("input.json").toURI()))); return Response.ok(json).build(); } catch (IOException | URISyntaxException e) { } return Response.serverError().build(); } }
Jetty JAX-RS Server
This is our application class where we’re starting an embedded Jetty, registering the REST resource as well as the servlet filter and starting a server instance on port 9000:
package com.hascode.tutorial; import java.util.EnumSet; import javax.servlet.DispatcherType; import org.eclipse.jetty.server.Server; import org.eclipse.jetty.servlet.ServletContextHandler; import org.eclipse.jetty.servlet.ServletHolder; public class RestServer { public static void main(String[] args) throws Exception { ServletContextHandler context = new ServletContextHandler(ServletContextHandler.SESSIONS); context.setContextPath("/"); Server jettyServer = new Server(9000); jettyServer.setHandler(context); ServletHolder jerseyServlet = context.addServlet(org.glassfish.jersey.servlet.ServletContainer.class, "/*"); jerseyServlet.setInitParameter("jersey.config.server.provider.classnames", SampleRestResource.class.getCanonicalName()); jerseyServlet.setInitOrder(0); context.addFilter(JsonTransformerFilter.class, "/*", EnumSet.of(DispatcherType.REQUEST, DispatcherType.FORWARD)); try { jettyServer.start(); jettyServer.join(); } finally { jettyServer.destroy(); } } }
Servlet Filter
Our servlet filter reads the specification from the classpath, passes the filter chain with a decorated servlet request wrapper and finally writes the result of the transformation so that we’re able to receive a modified JSON structure in the REST service’s response:
package com.hascode.tutorial; import java.io.IOException; import java.io.PrintWriter; import java.util.List; import javax.servlet.Filter; import javax.servlet.FilterChain; import javax.servlet.FilterConfig; import javax.servlet.ServletException; import javax.servlet.ServletRequest; import javax.servlet.ServletResponse; import javax.servlet.http.HttpServletResponse; import com.bazaarvoice.jolt.Chainr; import com.bazaarvoice.jolt.JsonUtils; public class JsonTransformerFilter implements Filter { @Override public void init(FilterConfig filterConfig) throws ServletException { System.out.println("json transformer servlet filter initialized"); } @Override public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { List<Object> specs = JsonUtils.classpathToList("/spec.json"); System.out.println("specs loaded:"); specs.forEach(System.out::println); Chainr chainr = Chainr.fromSpec(specs); CustomResponseWrapper res = new CustomResponseWrapper((HttpServletResponse) response); chain.doFilter(request, res); String inputJSON = res.getContent(); Object transformedOutput = chainr.transform(JsonUtils.jsonToObject(inputJSON)); System.out.printf("transformed json: %s\n", transformedOutput); PrintWriter writer = response.getWriter(); writer.write(transformedOutput.toString()); writer.flush(); } @Override public void destroy() { } }
Servlet Response Wrapper
Just a simple decorator to wrap the servlet response, if interested, have a look at my quick implementation here.
Running the Service
Now we’re ready to start our RESTful webservice in our IDE or command line using Maven like this:
$ mvn clean compile exec:java -Dexec.mainClass=com.hascode.tutorial.RestServer [INFO] ------------------------------------------------------------------------ [INFO] Building json-transform-jolt-tutorial 1.0.0 [INFO] ------------------------------------------------------------------------ [..] json transformer servlet filter initialized specs loaded: {operation=remove, spec={*={page=, date=, startTime=, endTime=}}} transformed json: [{uuid=cac40601-ffc9-4fd0-c5a1-772ac65f0587, pageId=123456, location=Berlin, maxParticipants=10, start=1487145600000, end=1487172600000, hasStartTime=true, hasEndTime=true, eventPageUrl=/some/url, categories=[{label=Test, id=123, belongsToEvents=true}], participants=[], isCurrentUserParticipating=false, fullTitle=Test Page}] specs loaded: {operation=remove, spec={*={page=, date=, startTime=, endTime=}}}
Now we should be able to access our REST service pointing our browser to http://localhost:9000/data
JOLT and Apache Camel
Apache Camel is a nice framework to apply different enterprise integration patterns and to allow our applications to use and connect endpoints in every protocol we can imagine – and of course there is also a component for JOLT transformations support.
I have written articles about Apache Camel in the past, please feel free to read further if interested:
Dependencies
We simply need to add two dependencies to our project: camel-core and camel-jolt:
<properties> <camel.version>2.18.1</camel.version> </properties> <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-core</artifactId> <version>${camel.version}</version> </dependency> <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-jolt</artifactId> <version>${camel.version}</version> </dependency>
Sample Application
This is our final application..
- starting a camel route
- reading from the input directory named scanned
- searching for files named input.json
- passing them to the JOLT transformer
- applying the transformation from our specification file named spec.json
- writing the transformed file to the output directory named transformed.
package com.hascode.tutorial; import org.apache.camel.builder.RouteBuilder; import org.apache.camel.main.Main; public class CamelExample { public static void main(String[] args) throws Exception { Main main = new Main(); main.addRouteBuilder(new RouteBuilder() { @Override public void configure() throws Exception { from("file://scanned?fileName=input.json") .to("jolt:/spec.json?inputType=JsonString&outputType=JsonString").to("file://transformed"); } }); main.run(); } }
Running the Example
Again we may now run our application in our IDE/from command line:
$ mvn clean compile exec:java -Dexec.mainClass=com.hascode.tutorial.CamelExample
When we now copy the sample input.json into the directory named scanned we should afterwards see a transformed JSON file in the output directory named transformed.
Resources
- Jolt Project on GitHub
- Jolt Introduction (Google Slides)
- Online Jolt Transformation Demo
- Apache Camel: Jolt Component Documentation
- Jetty Project on Eclipse.org
Tutorial Sources
Please feel free to download the tutorial sources from my Bitbucket repository, fork it there or clone it using Git:
git clone https://bitbucket.org/hascode/json-transform-jolt-tutorial.git
Troubleshooting
- “java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NoClassDefFoundError: org/eclipse/jetty/util/Decorator at com.hascode.tutorial.RestServer.main(RestServer.java:14) … 6 more Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.util.Decorator at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) … 7 more” -> The dependency jersey-container-jetty-http uses an older version of jetty-util, adding an exclusion to this dependency solves the problem:
<dependency> <groupId>org.glassfish.jersey.containers</groupId> <artifactId>jersey-container-jetty-http</artifactId> <version>${jersey.version}</version> <exclusions> <exclusion> <groupId>org.eclipse.jetty</groupId> <artifactId>jetty-util</artifactId> </exclusion> </exclusions> </dependency>
October 1st, 2018 at 3:59 pm
I want transform JSON format to Other Json Format how can it do it?
October 3rd, 2018 at 12:00 pm
Have you read the article?
April 5th, 2019 at 3:00 pm
Is there a way to transform the other way around without updating the spec?
May 4th, 2019 at 7:32 am
is there a way to Validate the JSON (like for mandatory elements/attributes etc) and then transform using JOLT ? or the only option is to use someting like JsonPath to validate and then transform using JOLT ?
October 3rd, 2019 at 4:10 am
Is there a way to do the following:
Input Json:
{
“x” : {
“a” : “123″,
“b” : “456″,
“c” : {
“d” : “789″
}
}
}
Output Json Expected:
{
“data” : “x={
a=123, b=456,
c={d=789}}”
}
December 6th, 2021 at 9:36 am
Thank you mate for Jolt example! Save me a lot of time!