BT

Building a RESTful Web Service with Spring Boot to Access Data in an Aerospike Cluster

Posted by Peter Milne on Nov 28, 2013 |

Spring Boot is a powerful jump start into Spring. It allows you to build Spring based applications with little effort on your part.

Aerospike is a distributed and replicated in-memory database optimized to use both DRAM and native flash/SSDs.

Aerospike also has high reliability and is ACID compliant. Developers can quickly scale their database cluster from two nodes to twenty nodes without bringing down the database service.

What you will build

This article will take you through creating a simple RESTful web service with Spring Boot.

You will build a service that accepts an HTTP GET request. It responds with the following JSON:

{"expiration":121023390,"bins":{"DISTANCE":2446,"DEST_CITY_NAME":"New York","DEST":"JFK","YEAR":2012,"ORI_AIRPORT_ID":"14679","DEP_TIME":"802","DAY_OF_MONTH":12,"DEST_STATE_ABR":"NY","ORIGIN":"SAN","FL_NUM":160,"CARRIER":"AA","ORI_STATE_ABR":"CA","FL_DATE":"2012/01/12","AIR_TIME":291,"ORI_CITY_NAME":"San Diego","ELAPSED_TIME":321,"ARR_TIME":"1623","AIRLINE_ID":19805},"generation":1}

The data you will use is commercial flight details (included in the sample code (SP: Add the link to zip file), is a data file flights_from.csv. It contains approximately one million flight records.

There are also many features added to your application out-of-the-box for managing the service in a production (or other) environment. This functionally comes from Spring, (see the Spring guide: Building a RESTful web service.)

What you will need

Set up the project

You can use any build system you like when building applications with Spring, but the Maven code is included here. If you are unfamiliar with Maven refer to the Spring guide: Building Java Projects with Maven.

You will also need to build and install the Aerospike Java client into your local Maven repository. Download the source distribution, unzip/untar it and run the following Maven commands:

  • mvn install:install-file -Dfile=client/depends/gnu-crypto.jar -DgroupId=org.gnu -DartifactId=gnu-crypto -Dversion=2.0.1 -Dpackaging=jar
  • mvn clean
  • mvn package

Create the directory structure

In the project of your choosing, create the following subdirectory structure:

->src
   ->main
     ->java
       ->com
         ->aerospike
            ->client
               ->rest

Create a Maven pom

Create a maven pom.xml file in the root directory of your project with the following code:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
      <modelVersion>4.0.0</modelVersion>
      <groupId>com.aerospike</groupId>
      <artifactId>aerospike-restful-example</artifactId>
      <version>1.0.0</version>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>0.5.0.M4</version>
    </parent>
      <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
            <!-- Aerospike client. -->
        <dependency>
              <groupId>com.aerospike</groupId>
              <artifactId>aerospike-client</artifactId>
              <version>3.0.9</version>
        </dependency>
        <!-- Apache command line parser. -->
        <dependency>
              <groupId>commons-cli</groupId>
              <artifactId>commons-cli</artifactId>
              <version>1.2</version>
        </dependency>
  </dependencies>

  <properties>
      <start-class>com.aerospike.client.rest.AerospikeRESTfulService</start-class>
  </properties>

  <build>
      <plugins>
          <plugin> 
              <artifactId>maven-compiler-plugin</artifactId> 
              <version>2.3.2</version> 
          </plugin>
          <plugin>
              <groupId>org.springframework.boot</groupId>
              <artifactId>spring-boot-maven-plugin</artifactId>
          </plugin>
      </plugins>
  </build>
  <repositories>
      <repository>
          <id>spring-snapshots</id>
          <name>Spring Snapshots</name>
          <url>http://repo.spring.io/libs-snapshot</url>
          <snapshots>
              <enabled>true</enabled>
          </snapshots>
      </repository>
  </repositories>
  <pluginRepositories> 
      <pluginRepository>
          <id>spring-snapshots</id>
          <name>Spring Snapshots</name>
          <url>http://repo.spring.io/libs-snapshot</url>
          <snapshots>
              <enabled>true</enabled>
          </snapshots>
       </pluginRepository>
  </pluginRepositories>

</project>

It looks scary, but it really isn’t.

Create a JSON translator class

The Aerospike API will return a Record object and it will contain the generation, expiry and bin values of the record. But you want to have these values returned in JSON format. The easiest way to achieve this is to use a translator class.

Create a translator class with the following code. It is a generic class that will translate an Aerospike Record object to a JSONObject.

src/main/java/com/aerospike/client/rest/JSONRecord.java
package com.aerospike.client.rest;
import java.util.Map;
import org.json.simple.JSONArray;
import org.json.simple.JSONObject;
import com.aerospike.client.Record;
/**
* JSONRecord is used to convert an Aerospike Record
* returned from the cluster to JSON format
*
*/
@SuppressWarnings("serial")
public class JSONRecord extends JSONObject {
      @SuppressWarnings("unchecked")
      public JSONRecord(Record record){
            put("generation", record.generation);
            put("expiration", record.expiration);
            put("bins", new JSONObject(record.bins));
            if (record.duplicates != null){
                  JSONArray duplicates = new JSONArray();
                  for (Map<String, Object> duplicate : record.duplicates){
                        duplicates.add(new JSONObject(duplicate));
                  }
                  put("duplicates", duplicates);
             }
       }
}

This class is not complicated and is very generic. You may wish to specialize your JSON translation for specific records.

Create a resource controller

In Spring, REST endpoints are Spring MVC controllers. The following code handles a GET request for /as/{namespace}/{set}/getAll/1234 and returns the Flight record whose key is 1234, where {namespace} is the path variable for the Aerospike namespace and {set} is the path variable for the Aerospike set.

src/main/java/com/aerospike/client/rest/RESTController.java
package com.aerospike.client.rest;
import java.io.BufferedReader;
import java.io.InputStreamReader;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.multipart.MultipartFile;
import com.aerospike.client.AerospikeClient;
import com.aerospike.client.Bin;
import com.aerospike.client.Key;
import com.aerospike.client.Record;
import com.aerospike.client.policy.Policy;
import com.aerospike.client.policy.WritePolicy;

@Controller
public class RESTController {
      @Autowired
      AerospikeClient client;
    @RequestMapping(value="/as/{namespace}/{set}/getAll/{key}", method=RequestMethod.GET)
    public @ResponseBody JSONRecord getAll(@PathVariable("namespace") String namespace, 
            @PathVariable("set") String set,
            @PathVariable("key") String keyvalue) throws Exception {
    Policy policy = new Policy();
    Key key = new Key(namespace, set, keyvalue);
     Record result = client.get(policy, key);
     return new JSONRecord(result);
  }
}

The difference between a human-facing controller and a REST endpoint controller is the response body will contain data, in your case a JSON object that represents the record read from Aerospike.

The @ResponseBody annotation tells Spring MVC to write the returned object into the response body.

Create an executable main class

Implement the main method to create a Spring MVC controller. The easiest way to do this is to use the SpringApplication helper class.

src/main/java/com/aerospike/client/rest/AerospikeRESTfulService.java
package com.aerospike.client.rest;
import java.util.Properties;
import javax.servlet.MultipartConfigElement;
import org.apache.commons.cli.CommandLine;
import org.apache.commons.cli.CommandLineParser;
import org.apache.commons.cli.Options;
import org.apache.commons.cli.ParseException;
import org.apache.commons.cli.PosixParser;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;

import com.aerospike.client.AerospikeClient;
import com.aerospike.client.AerospikeException;
@Configuration
@EnableAutoConfiguration
@ComponentScan
public class AerospikeRESTfulService {
      @Bean
      public AerospikeClient asClient() throws AerospikeException {
            Properties as = System.getProperties();
            return new AerospikeClient(as.getProperty("seedHost"), 
                  Integer.parseInt(as.getProperty("port")));
      }
      @Bean
      public MultipartConfigElement multipartConfigElement() {
            return new MultipartConfigElement("");
      }
      public static void main(String[] args) throws ParseException {
             Options options = new Options();
             options.addOption("h", "host", true, 
                   "Server hostname (default: localhost)");
             options.addOption("p", "port", true, "Server port (default: 3000)");
             // parse the command line args
             CommandLineParser parser = new PosixParser();
             CommandLine cl = parser.parse(options, args, false);
             // set properties
             Properties as = System.getProperties();
             String host = cl.getOptionValue("h", "localhost");
             as.put("seedHost", host);
             String portString = cl.getOptionValue("p", "3000");
             as.put("port", portString);
             // start app
             SpringApplication.run(AerospikeRESTfulService.class, args);
      }
}

The @EnableAutoConfiguration annotation has been added: it provides a load of defaults (like the embedded servlet container) depending on the contents of your classpath, and other things.

It is also annotated with @ComponentScan, which tells Spring to scan the hello package for those controllers (along with any other annotated component classes).

Finally this class is annotated with @Configuration. This allows you to configure an instance of the AerospikeClient as a Spring bean.

There is also a MultipartConfigElement bean defined. This allows you to process POST operations with this service.

Most of the body of the main method simply reads command line arguments and sets system properties to specify the seed host and port of the Aerospike cluster.

Too easy!

Uploading data

You may want to upload data to this service. To do this you need to add an additional method to the RESTController class to process the uploaded file. In this example, it will be a CSV file containing flight records.

src/main/java/com/aerospike/client/rest/RESTController.java
@Controller
public class RESTController {
   . . . (code omitted) . . .
   /*
    * CSV flights file upload
    */
   @RequestMapping(value="/uploadFlights", method=RequestMethod.GET)
   public @ResponseBody String provideUploadInfo() {
       return "You can upload a file by posting to this same URL.";
   }
   @RequestMapping(value="/uploadFlights", method=RequestMethod.POST)
   public @ResponseBody String handleFileUpload(@RequestParam("name") String name, 
          @RequestParam("file") MultipartFile file){
     if (!file.isEmpty()) {
           try {
                 WritePolicy wp = new WritePolicy();
                 String line = "";
                 BufferedReader br = new BufferedReader(new InputStreamReader(file.getInputStream()));
                 while ((line = br.readLine()) != null) {
                       // use comma as separator
                       String[] flight = line.split(",");

                       /*
                        * write the record to Aerospike
                        * NOTE: Bin names must not exceed 14 characters
                        */
                            client.put(wp,
                                  new Key("test", "flights",flight[0].trim() ),
                                  new Bin("YEAR", Integer.parseInt(flight[1].trim())),
                                  new Bin("DAY_OF_MONTH", Integer.parseInt(flight[2].trim())),
                                  new Bin("FL_DATE", flight[3].trim()),
                                  new Bin("AIRLINE_ID", Integer.parseInt(flight[4].trim())),
                                  new Bin("CARRIER", flight[5].trim()),
                                  new Bin("FL_NUM", Integer.parseInt(flight[6].trim())),
                                  new Bin("ORI_AIRPORT_ID", Integer.parseInt(flight[7].trim())),
                                  new Bin("ORIGIN", flight[8].trim()),
                                  new Bin("ORI_CITY_NAME", flight[9].trim()),
                                  new Bin("ORI_STATE_ABR", flight[10].trim()),
                                  new Bin("DEST", flight[11].trim()),
                                  new Bin("DEST_CITY_NAME", flight[12].trim()),
                                  new Bin("DEST_STATE_ABR", flight[13].trim()),
                                  new Bin("DEP_TIME", Integer.parseInt(flight[14].trim())),
                                  new Bin("ARR_TIME", Integer.parseInt(flight[15].trim())),
                                  new Bin("ELAPSED_TIME", Integer.parseInt(flight[16].trim())),
                                  new Bin("AIR_TIME", Integer.parseInt(flight[17].trim())),
                                  new Bin("DISTANCE", Integer.parseInt(flight[18].trim()))
                            );
                            System.out.println("Flight [ID= " + flight[0] 
                                                    + " , year=" + flight[1] 
                                                    + " , DAY_OF_MONTH=" + flight[2] 
                                                    + " , FL_DATE=" + flight[3] 
                                                    + " , AIRLINE_ID=" + flight[4] 
                                                    + " , CARRIER=" + flight[5] 
                                                    + " , FL_NUM=" + flight[6] 
                                                    + " , ORIGIN_AIRPORT_ID=" + flight[7] 
                                                    + "]");
                        }
                        br.close();
                        return "You successfully uploaded " + name;
                  } catch (Exception e) {
                        return "You failed to upload " + name + " => " + e.getMessage();
                  }
            } else {
                  return "You failed to upload " + name + " because the file was empty.";
              }
       }
}

A new method handleFileUpload() responds to a POST and reads the upload stream one line at a time. Each line is parsed and a Key object and several Bin objects are built to form the Aerospike record. Finally the Aerospike put() method is called to store the record in the Aerospike cluster.

Another new method provideUploadInfo() responds to a GET and returns a message indicating uploads are possible.

Uploading client application

Uploading can be done any way you desire. But you can use the following standalone java class to upload data to this service.

src/test/java/com.aerospike.client.rest/FlightsUploader.java
package com.aerospike.client.rest;
import org.junit.Before;
import org.junit.Test;
import org.springframework.core.io.FileSystemResource;
import org.springframework.util.LinkedMultiValueMap;
import org.springframework.util.MultiValueMap;
import org.springframework.web.client.RestTemplate;

public class FilghtsUploader {
      private static final String TEST_FILE = "flights_from.csv";
      @Before
      public void setUp() throws Exception {
      }

      @Test
      public void upload() {
        RestTemplate template = new RestTemplate();
        MultiValueMap<String, Object> parts = new LinkedMultiValueMap<String, Object>();
        parts.add("name", TEST_FILE);
        parts.add("file", new FileSystemResource(TEST_FILE));
        String response = template.postForObject("http://localhost:8080/uploadFlights",
                                                  parts, String.class);
        System.out.println(response);
      }
}

Flights Data

This is real data from 2012. It contains approximately 1 million records, so remember that it will take a few minutes to upload.

Building and running the Service

The Maven pom.xml will package the service into a single jar. Use the command:

mvn clean package

This will generate a stand-alone web service application packaged into a runnable jar file in the target subdirectory. This jar file includes an instance of Tomcat, so you can simply run the jar without installing it in an Application Server.

java -jar aerospike-restful-example-1.0.0.jar

Summary

Congratulations! You have just developed a simple RESTful service using Spring and connecting it to an Aerospike cluster.

Complete Example Code

Example code

Design Considerations

Access control is currently handled by the application versus the database. Since the authentication process slows down database speeds, virtually all NoSQL databases do not support this function. Most of our customers have prioritized increased speed over an integrated authentication feature.

Another commonly requested feature is a join of two different sets of data. This is challenge faced by all distributed databases because the data for the join is distributed. In this case, the developer has to implement a join in the application.

About the Author

Peter Milne is a seasoned IT professional with extensive experience across the full software development and product life cycles. He has technical skills and management experience with small and large development teams. Peter was most recently a senior solutions architect at Aerospike. Prior to that, he was a senior analyst and programmer at MLC, and was CTO at iTerative Consulting building a Forte/UDS to Java conversion tool with 99.999% accuracy. Peter holds an MSc in distributed computing from the University of Technology, Sydney, as well as several helicopter safety licenses and certificates.

 

Hello stranger!

You need to Register an InfoQ account or to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2013 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT