Implementing APIs to support autocomplete with Spring and Elasticsearch

Autocomplete is a common feature where a user can enter a few characters in a text box and the application can then use them to provide a list of alternative values (suggestions). The user can then select the correct one, typically using a drop down box, without having to type the full text. Example usage would include an e-commerce shopping cart where a customer is required to enter her address for delivery.

This post will demonstrate how to implement the backend support for autocomplete. In particular, APIs for returning suggestions based on input search string using Spring Boot (MVC) and Elasticsearch High Level Java Rest Client.

Project Setup

Below is the versions of Spring Boot and Elasticsearch used:

  • Spring Boot – version 2.0.4.RELEASE
  • Elasticsearch Java High Level REST Client – version 6.3.2

Maven

...
	<properties>
                ...
		<elasticsearch.version>6.3.2</elasticsearch.version>
	</properties>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-web</artifactId>
	</dependency>
	<dependency>
		<groupId>org.elasticsearch.client</groupId>
		<artifactId>elasticsearch-rest-high-level-client</artifactId>
		<version>${elasticsearch.version}</version>
	</dependency>
	<dependency>
		<groupId>org.elasticsearch.client</groupId>
		<artifactId>elasticsearch-rest-client</artifactId>
		<version>${elasticsearch.version}</version>
	</dependency>
	<dependency>
		<groupId>org.elasticsearch</groupId>
		<artifactId>elasticsearch</artifactId>
	</dependency>

Autocomplete APIs

We are going to implement an autocomplete feature for an input field of an address type. Spring MVC is used for the APIs and they will accept a text in the input request and return in the response a list of full address suggestions.

Completion Suggester

This is the standard way to implement type-as-you-go autocomplete with Elasticsearch. Below is the codes to construct the query using Elasticsearch Java REST client:

 

@Override
public SearchResultDto autocomplete(String prefixString, int size) {
     SearchRequest searchRequest = new SearchRequest(INDEX);
     CompletionSuggestionBuilder suggestBuilder = new CompletionSuggestionBuilder(FIELD_COMPLETION); // Note 1

     suggestBuilder.size(size)
                   .prefix(prefixString, Fuzziness.ONE) // Note 2
                   .skipDuplicates(true)
                   .analyzer("standard");
 
     SearchSourceBuilder sourceBuilder = new SearchSourceBuilder(); // _search
     sourceBuilder.suggest(new SuggestBuilder().addSuggestion(SUGGESTION_NAME, suggestBuilder));
     searchRequest.source(sourceBuilder);

     SearchResponse response;
     try {
          response = client.search(searchRequest);
          return getSuggestions(response); // Note 3
     } catch (IOException ex) {
          logger.error("Error in autocomplete search", ex);
          throw new HttpServerErrorException(HttpStatus.INTERNAL_SERVER_ERROR, "Error in ES search");
     }
}
  • Note 1: FIELD_COMPLETION is the name of the field to search. In order for the field to be used here, it has to be indexed with the completion type. For example, for the field named formattedAddress, the mapping setting for the index should look like below. FIELD_COMPLETION should then be “formattedAddress.completion”
    "mappings": {
...    
     "formattedAddress": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              },
              "completion": {
                "type": "completion",
                "analyzer": "standard",
                "preserve_separators": true,
                "preserve_position_increments": true,
                "max_input_length": 100
              }
            }
          },
...
  • Note 2: Fuzziness is set here to provide some leeway with typos. By default it’s 0
  • Note 3: To get the suggestions from the search response:
private SearchResultDto getSuggestions(SearchResponse response) {
	SearchResultDto dto = new SearchResultDto();
	Suggest suggest = response.getSuggest();
	Suggestion<Entry<Option>> suggestion = suggest.getSuggestion(SUGGESTION_NAME);
	for(Entry<Option> entry: suggestion.getEntries()) {
	      for (Option option: entry.getOptions()) {
	        dto.add(option.getText().toString());
	      }
	}
	return dto;
}

where SearchResultDto is just a wrapper class for list of suggestions

public class SearchResultDto {
     private List<String> suggestedAddresses;
...

Finally the Spring MVC controller for implementing the API

@RestController
@RequestMapping("/address")
@CrossOrigin
public class AddressController {

     @Autowired
     private AddressSearchService service;

     @GetMapping(params = {"type=autocomplete"})
     public SearchResultDto autocomplete(@RequestParam String search, @RequestParam(defaultValue = "20") int size) {
          return service.autocomplete(search, size);
     }
...

That’s it. As an example, a call to the API with search string “8 Rudd” would return a list of address below with the database I have. Note the fuzziness of the returned addresses, e.g. RUDA… vs RUDD

8 RUDALL STREET LATHAM ACT 2615
8 RUDD STREET CITY ACT 2601
8 RUDDER PLACE KAMBAH ACT 2902
8 RUNDLE PLACE KAMBAH ACT 2902
8 REDDALL CLOSE ISAACS ACT 2607

 

 

 

 

Advertisements

Building REST APIs with Spring Boot and SpringFox/Swagger

This blog post demonstrates step by step how to develop RESTful APIs using Spring Boot and then how to document and expose the API via Swagger (OpenAPI)  specification using SpringFox.

Project Setup

The project is a Spring Boot web application, e.g. created with Spring Initializr, with the following dependencies (in Maven):

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-data-mongodb</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>

I am using Spring Data backed by MongoDB data store here. We can also use a relational database such as MySQL instead.

To use SpringFox, we also need to add the following:

		<dependency>
			<groupId>io.springfox</groupId>
			<artifactId>springfox-swagger2</artifactId>
			<version>2.9.2</version>
		</dependency>
		<dependency>
			<groupId>io.springfox</groupId>
			<artifactId>springfox-swagger-ui</artifactId>
			<version>2.9.2</version>
		</dependency>

The first dependency is required to add SpringFox support for Swagger 2 to the project. The second dependency adds swagger-ui, which is a GUI framework for displaying the API documentation as part of the web application.

Product API

We are going to build a very simple CRUD API for product resources.

@Document(collection = "product")
public class Product {

     @Id
     private String id;
     private String sku;
     private String description;
     private BigDecimal price;
...

The entity is to be persisted to a MongoDB collection product as denoted by the @Document annotation. This is implemented with Spring Data using the MongoRepository

// ProductRepository.java
import org.springframework.data.mongodb.repository.MongoRepository;
...

public interface ProductRepository extends MongoRepository<Product, String> {

}

The above repository is accessed by the controller via the service layer as defined by the following interface

public interface ProductService {
	Product read(String id);	
	Product update(String id, Product product);
	Product create(Product product);
	void delete(String id);
}

Finally, the rest controller implemented with Spring MVC

// ProductController.java - plain old Spring MVC
@RestController
@RequestMapping("/product")
public class ProductController {

	@Autowired
	private ProductService service;

	@PostMapping(value = "")
	public Product create(@RequestBody Product product) {
		return service.create(product);
	}

	@GetMapping(value = "/{id}")
	public Product read(@PathVariable("id") String id) {
		return service.get(id);
	}

	@PatchMapping(value = "/{id}")
	public Product update(@PathVariable("id") String id, @RequestBody Product product) {
		return service.update(id, product);
	}

	@DeleteMapping(value = "/{id}")
	public void delete(@PathVariable("id") String id) {
		service.delete(id);
	}
}

Swagger with SpringFox

So far, we have a standard Spring Boot web application which can serve the product API. To use Swagger with SpringFox, we will need to update the codes as follows:

Configuration

@Configuration
@EnableSwagger2
public class SpringFoxConfig {

     // Inject API info property values
     @Value("${api.title}")
     private String title;
     ...

     @Bean
     public Docket apiDocket() {
          //@formatter:off
          return new Docket(DocumentationType.SWAGGER_2)
                     .select()
                     .apis(RequestHandlerSelectors.basePackage("com.madman.bootswagger.controller"))
                     .paths(PathSelectors.any())
                     .build()
                     .apiInfo(getApiInfo());
          //@formatter:on
      }

     private ApiInfo getApiInfo() {
     //@formatter:off
          return new ApiInfo(
                     title,
                     description,
                     version,
                     termOfServiceUrl,
                     new Contact(contactName, contactUrl, contactEmail),
                     "LICENSE",
                     "LICENSE URL",
                     Collections.emptyList()
           );
      //@formatter:on
     }
}

The configuration setup is rather straight forward. All is needed is to declare the @EnableSwagger2 annotation to indicate Swagger support should be enabled and a Docket bean as builder into SpringFox. Note we restrict the apis scanning to the controller package.

ProductController with Swagger

The controller has been updated with SpringFox as below with the changes highlighted:

// ProductController.java with Swagger
@RestController
@RequestMapping("/product")
@Api(value = "products")
public class ProductController {

     @Autowired
     private ProductService service;

     @PostMapping(value = "/")
     public Product create(Product product) {
          return service.create(product);
     }

     @GetMapping(value = "/{id}")
     public Product read(@ApiParam(value = "product id", required = true) @PathVariable("id") String id) {
          return service.get(id);
     }

     @PatchMapping(value = "/{id}")
     public Product update(@ApiParam(value = "product id", required = true) @PathVariable("id") String id, 
                           @ApiParam(value = "product", required = true) @RequestBody Product product) {
          return service.update(id, product);
     }

     @DeleteMapping(value = "/{id}")
     public void delete(@ApiParam(value = "product id", required = true) @PathVariable("id") String id) {
          service.delete(id);
     }
}

The @Api annotation at the class level tells SpringFox to scan the class methods for API documentation.

Product model document

Finally, it is possible to annotate entity class to generate more appropriate documentation in Swagger using the @ApiModal and @ApiModelProperty annotations

@Document(collection = "product")
@ApiModel(description = "Product model")
public class Product {

@Id
@ApiModelProperty(value = "id in mongodb store", accessMode = AccessMode.READ_ONLY)
private String id;

@ApiModelProperty(value = "stock keeping unit")
private String sku;

@ApiModelProperty(value = "product description")
private String description;

@ApiModelProperty(value = "product base price")
private BigDecimal price;

Product API Specification

Now fire up Spring Boot and open the Swagger UI url (http://localhost:8080/swagger-ui.html) should display something like the screenshot below

blog_swagger

The above GUI display the API documentation as set by the annotations in the domain model and the controller. More importantly, the link http://localhost:8080/v2/api-docs provides the Swagger API specifications which the API clients can use in their development and testing.

Building Serverless APIs with Spring Boot, AWS Lambda and API Gateway

This post demonstrates how to expose a RESTful API implemented with Spring MVC in a Spring Boot application as a Lambda function to be deployed via AWS API Gateway. We will be using the aws-serverless-java-container package which supports native API gateway’s proxy integration models for requests and responses.

Project Setup

Create a new Spring Boot project e.g. using the Spring Initializer or modify an existing project to include the aws-serverless-java-container package dependency:

<dependency> 
      <groupId>com.amazonaws.serverless</groupId> 
      <artifactId>aws-serverless-java-container-spring</artifactId> 
      <version>1.1</version> 
</dependency>

We can remove the Spring Boot Maven Plugin from the pom file. Instead, add the Maven Shade Plugin and remove the embedded Tomcat from the deployed package:

<plugins> 
   <plugin> 
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-shade-plugin</artifactId>
      <configuration> 
        <createDependencyReducedPom>false</createDependencyReducedPom> 
      </configuration> 
      <executions> 
        <execution> 
          <phase>package</phase>
          <goals> 
            <goal>shade</goal> 
          </goals> 
          <configuration> 
             <artifactSet> 
                <excludes> 
                   <exclude>org.apache.tomcat.embed:*</exclude>
                </excludes> 
             </artifactSet>
          </configuration> 
        </execution>
      </executions> 
   </plugin>
</plugins>

Serverless API

1. HelloController

Implement RESTful APIs using Spring MVC as usual. For example:

package com.madman.lambda;
...
@RestController
public class HelloController {
 
     @RequestMapping(path = "/greeting", method = RequestMethod.GET) 
     public GreetingDto sayHello(@RequestParam String name) { 
          String message = "Hello " + name; 
          GreetingDto dto = new GreetingDto();
          dto.setMessage(message); return dto; 
     }

     ...
}

2. StreamLambdaHandler

To deploy Java codes to run as AWS Lambda function, it needs to implement the handler interface RequestStreamHandler. The aws-serverless-java-container library makes it rather straight forward:

...
public class StreamLambdaHandler implements RequestStreamHandler {
    private static Logger logger = LoggerFactory.getLogger(StreamLambdaHandler.class);     

    public static final SpringBootLambdaContainerHandler<AwsProxyRequest, AwsProxyResponse> handler;
 
    static { 
       try { 
           handler = SpringBootLambdaContainerHandler.getAwsProxyHandler(HelloLambdaApplication.class);
       } catch (ContainerInitializationException e) { 
           // if we fail here. We re-throw the exception to force another cold start 
           String errMsg = "Could not initialize Spring Boot application"; 
           logger.error(errMsg); 
           throw new RuntimeException("Could not initialize Spring Boot application", e); 
       } 
    }

    @Override 
    public void handleRequest(InputStream inputStream, OutputStream outputStream, Context context) throws IOException {
        handler.proxyStream(inputStream, outputStream, context);
        // just in case it wasn't closed 
        outputStream.close(); 
    }
}

The class StreamLambdaHandler implements the AWS Lambda predefined handler  interface RequestStreamHandler for handling events.

Note the handling of the Lambda events is delegated to the class SpringBootLambdaContainerHandler.

3. HelloLambdaApplication

Note the SpringBootLambdaContainerHandler.getAwsProxyHandler method is provided with a Spring web application initializer interface, which is implemented by the main Spring Boot Application class by extending the implementing class SpringBootServletInitializer :

@SpringBootApplication
@ComponentScan(basePackages = "com.madman.lambda.controller")
public class HelloLambdaApplication extends SpringBootServletInitializer {
 
     public static void main(String[] args) { 
           SpringApplication.run(HelloLambdaApplication.class, args);
     }
}

4. HelloControllerTest

The aws-serverless-java-container library also supports integration testing the proxy API. Below is integration test for HelloController:

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(classes = { HelloLambdaApplication.class })
@WebAppConfiguration
public class HelloControllerTest {
    private MockLambdaContext lambdaContext;
    private SpringBootLambdaContainerHandler<AwsProxyRequest, AwsProxyResponse> handler;
    
    @Autowired 
    private ObjectMapper mapper;
 
    public HelloControllerTest() { 
       lambdaContext = new MockLambdaContext(); 
       this.handler = StreamLambdaHandler.handler; 
    }

    @Test public void testGreetingApi() throws JsonParseException, JsonMappingException, IOException {
       AwsProxyRequest request = new AwsProxyRequestBuilder("/greeting", "GET").queryString("name", "John").build(); 
       AwsProxyResponse response = handler.proxy(request, lambdaContext);
      
       assertThat(response.getStatusCode(), equalTo(200)); 
       GreetingDto responseBody = mapper.readValue(response.getBody(), GreetingDto.class);
       asserThat(responseBody.getMessage(), equalTo("Hello John")); 
    }
}

Deploying to AWS

The full source codes can be found in GitHub here. Run Maven to build the jar file and deploy it as Lambda function. I use the AWS Toolkit for Eclipse to deploy the jar package to AWS. Refer to AWS documentation for more options and information on deploying Lambda applications.

To setup AWS API Gateway as trigger for the Lambda function:

  1. Create a New API
  2. Create Resource
    1. Configure as proxy resource
    2. Resource Name: greeting
  3. Create Method
    1. Get
    2. Integration type: Lambda Function
    3. Use Lambda Proxy Integration: true
    4. Lambda Function: <Name of Lambda function>

You should then be able to test the API with the AWS console (as screenshot below).

blog_apigateway

A couple of things to note:

  1. Cold start – the Java container takes a good few seconds. The latency is ok once it’s warmed up
  2. Fat jar – the Spring Boot jar in this example is around 13MB which is still ok for Lambda (limit 50MB)

Hello Angular 6 Elements

Yet another blog post on Angular 6 elements.

Quick Start

A starter “hello world” project can be found in GitHub. To build the element, checkout the project and run the following command:

>npm run build:elements

This should generate the file hello-world.js in the elements folder. To use the newly created Angular element, embed the javascript file in a html file:

<!DOCTYPE html>
<body>
 <hello-world></hello-world>
 <scr\ipt type="text/javascript" src="hello-world.js"></script>
</body>
</html>

NgModule

The hello-world element is implemented as a Angular component the same way as in an Angular app. To use it as a custom element outside of Angular, it is bootstrapped a bit differently in the app module file (app.module.ts)

...
import { createCustomElement } from '@angular/elements';
import { HelloWorldComponent } from './hello-world/hello-world.component';
@NgModule({  
     declarations: [HelloWorldComponent],
     imports: [BrowserModule],
     entryComponents: [HelloWorldComponent],
     providers: []})
export class AppModule {
 constructor(private injector: Injector) {
 }
 ngDoBootstrap() { 
     const el = createCustomElement(HelloWorldComponent, {
          injector: this.injector }); 
     customElements.define('hello-world', el); 
 }
}

Note the missing bootstrap attribute in the @NgModule definition for an Angular app. Instead the component is registered with the browser as a custom element in the ngDoBootstrap() method using the createCustomElement() API exported by @angular/elements package.

The building and export of custom elements in Angular 6 is still a work in progress. A custom script and build task (build:elements) from this blog post is used here to concatenate the output javascript files into the single hello-world.js file.

RouteReuseStrategy for Route Caching in Angular

This post explains how to implement RouteReuseStrategy to support custom control of route snapshot caching in Angular. Typical use case is a list page which a user can search for a list of items and then select and navigates to a particular item’s detail pages. When the user clicks the back button in the browser, the Angular app should return to the list page displaying the same items as before, using the previous search criteria.

Other advantages of caching route for rendering is faster page load and reduce network traffic.

To achieve the above in Angular, we need to implement the RouteReuseStrategy to tell Angular not to destroy a component but to save it for re-rendering. There are few blog posts online already with example implementations. This blog will focus more on describing the mechanics of the interface and its methods

RouteReuseStretagy

Below is a skeleton implementation of a custom RouteReuseStrategy:

import { Injectable } from '@angular/core';
import { RouteReuseStrategy, ActivatedRouteSnapshot, DetachedRouteHandle } from '@angular/router';

@Injectable()
export class AppRouteReuseStrategyService implements RouteReuseStrategy {

     handles: {[key: string]: DetachedRouteHandle} = {};

     constructor() { }

     shouldDetach(route: ActivatedRouteSnapshot): boolean {
           // To Be Implemented
     }

     store(route: ActivatedRouteSnapshot, handle: DetachedRouteHandle): void {
           // To Be Implemented
     }

     shouldAttach(route: ActivatedRouteSnapshot): boolean {
           // To Be Implemented
     }

      retrieve(route: ActivatedRouteSnapshot): DetachedRouteHandle {
          // To Be Implemented
      }

      shouldReuseRoute(future: ActivatedRouteSnapshot, curr: ActivatedRouteSnapshot): boolean {
          // To Be Implemented
      }
}

shouldReuseRoute()

This is the first method to consider. If returns true, none of the other methods will be called. For example, when we are already reusing the current route snapshot. Note the future argument refers to the route that you come from previously. For example if the app navigates from item list page to item details page, curr would refer to the route for the item detail page and future to the route for the item list page.

shouldDetach() and store()

If the method shouldRouteReuse returns false, the method shouldDetach will be called to determine whether the current route snapshot should be detached and stored. If it returns true, the store method will be called. A handle to the detached route snapshot (of type DetachedRouteHandle) is provided as argument to the method so it can store it for later use.

Note if a null handle is provided to the method, it should erase the stored value for the input route. See the API documentation here.

Note once a route snapshot is detached, it is the developer’s responsibility to manage its lifecycle and perform any clean up as needed for proper memory management.

shouldAttach() and retrieve()

Similar to above,

If the method shouldReuseRoute returns false, the method shouldAttach will be called to determine if a cached route should be used. If it returns true, the method retrieve will be called to retrieve the saved handle to the detached route previously stored.

Note the shouldAttach method is also a good place to clean up any stored snapshots. For example when a user has logged out or the snapshot has got staled, in which case we should not be rendering the store snapshot. The method should then return false and the stored handle to route snapshot should be removed from storage.

That’s it. Hope above gives some clarity on what the class RouteReuseStrategy does.

Spring for Apache Kafka Quick Start

In this blog, I setup a basic Spring Boot project for developing Kafka based messaging system using Spring for Apache Kafka. The project also includes basic Spring config required for publishing and listening to messages from Kafka broker.

Project Setup

The following tools and versions are used here:

  1. Maven 3.x
  2. Spring Kafka 1.3.2 (current release version)
  3. Kafka client 0.11.0.2
  4. Spring Boot 1.5.9

The current Spring Boot release version (1.5.9) has Spring Kafka version  1.1.7 as the managed version. I have to override this to use 1.3.2. My Maven pom file fragment as below:

 <parent>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-parent</artifactId>
      <version>1.5.9.RELEASE</version>
      <relativePath /> <!-- lookup parent from repository -->
 </parent>

 <dependencies>
      <dependency>
           <groupId>org.springframework.kafka</groupId>
           <artifactId>spring-kafka</artifactId>
           <version>1.3.2.RELEASE</version>
      </dependency>
      
      <dependency>
           <groupId>org.springframework.kafka</groupId>
           <artifactId>spring-kafka-test</artifactId>
           <version>1.3.2.RELEASE</version>
           <scope>test</scope>
      </dependency>

      <dependency>
           <groupId>org.apache.kafka</groupId>
           <artifactId>kafka-clients</artifactId>
           <version>0.11.0.2</version>
      </dependency>

Producer Config

Spring Boot provides auto configuration for connecting to Kafka but I find it useful to setup the beans myself. Spring Kafka adopts the same approach to Kafka as in other message brokers such as ActiveMQ. For publishing message a template, KafkaTemplate, as to be configured as with JmsTemplate for ActiveMQ.

The following is my Java Config for a KafkaTemplate to publish message to the Kafka broker

@Configuration
public class KafkaProducerConfig {

     @Value("${spring.kafka.bootstrap-servers}") // (1)
     private String brokerAsString;
 
     @Bean
     public ProducerFactory<Integer, String> producerFactory() {
          return new DefaultKafkaProducerFactory<>(producerConfigs());
     }

     @Bean
     public Map<String, Object> producerConfigs() {
          Map<String, Object> props = new HashMap<>();
          props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, brokerAsString);
          props.put(ProducerConfig.RETRIES_CONFIG, 0);
          props.put(ProducerConfig.BATCH_SIZE_CONFIG, 16384);
          props.put(ProducerConfig.LINGER_MS_CONFIG, 1);
          props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 33554432);
          props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
          props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
          return props;
      }

     @Bean
     public KafkaTemplate<Integer, String> kafkaTemplate() {
          return new KafkaTemplate<Integer, String>(producerFactory());
     }
}

Note:

  1. The broker address is set using the property spring.kafka.bootstrap-servers defined in the application.properties (or yml) file. For example,
// application.properties
spring.kafka.bootstrap-servers=http://localhost:9092

Consumer Config

Consuming messages from Kafka using Spring Kafka is similar to consuming messages from Active MQ using Spring JMS support. We need to define container factory and message listener. Below is my Java Config for message listener factory.

@Configuration
public class KafkaConsumerConfig {
 
     @Value("${spring.kafka.bootstrap-servers}")
     private String brokerAsString;

     @Value("${spring.kafka.consumer.group-id}")
     private String groupId;
 
     @Value("${spring.kafka.consumer.auto-offset-reset}")
     private String autoOffsetReset;
 
     @Bean
     ConcurrentKafkaListenerContainerFactory<Integer, String> kafkaListenerContainerFactory() {
          ConcurrentKafkaListenerContainerFactory<Integer, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
          factory.setConsumerFactory(consumerFactory());
          return factory;
     }

     @Bean
     public ConsumerFactory<Integer, String> consumerFactory() {
         return new DefaultKafkaConsumerFactory<>(consumerConfigs());
     }

     @Bean
     public Map<String, Object> consumerConfigs() {
         Map<String, Object> props = new HashMap<>();
         props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, brokerAsString);
         props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
         props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, autoOffsetReset);
         props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
         props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "100");
         props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "15000");
         props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class);
         props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
         return props;
     }
}

Now we can listen to a Kafka topic by using the annotation @KafkaListener. For example

@Service
public class GreetingsTopicListener {

 private Logger logger = LoggerFactory.getLogger(getClass());
 
 @KafkaListener(topics = "greetings")
 public void listen(ConsumerRecord<?,?> cr) throws Exception {
      logger.info(cr.toString());
 }
}

@KafkaListener will use the default listener container factory defined in class ConsumerConfig above to create the message listener. It is also possible to override this by settig the containerFactory attribute in the annotation. See javadoc for more details.

Creating Topics

It is also possible to automatically add topics to the broker by defining @Beans using the new 0.11.0.x client library class AdminClient as in the Spring Kafka reference documentation

@Configuration
public class KafkaTopicConfig {
 
 @Value("${spring.kafka.bootstrap-servers}")
 private String brokerAsString;
 
 @Bean
 public KafkaAdmin admin() {
   Map<String, Object> configs = new HashMap<>();
   configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, brokerAsString);
   return new KafkaAdmin(configs);
 }

 @Bean
 public NewTopic topic1() {
   return new NewTopic("foo", 10, (short) 2);
 }

 @Bean
 public NewTopic topic2() {
   return new NewTopic("bar", 10, (short) 2);
 }
}

That’s about it. The codes included in this blog should be sufficient for setting up a Spring Boot project for a messaging system using Spring Kafka.

Background geolocation in Ionic app

This blog show an example on how to implement background geolocation in an Ionic app using the background geolocation plugin.  to capture user locations for building location aware applications

Install plugin

First add the cordova plugin to the Ionic app:

cordova plugin add cordova-plugin-mauron85-background-geolocation

Implement background geolocation service

Next, implement an Angular service:

var app = angular.module('starter');
app.factory('BackgroundGeolocationService', ['$q', '$http', function ($q, $http) { 
      var callbackFn = function(location) {
          postLocation(location);
          backgroundGeoLocation.finish();  
      },
      failureFn = function(error) {
          console.log('BackgroundGeoLocation error ' + JSON.stringify(error));  
      },

      //Enable background geolocation  
      start = function () {      
          //save settings (background tracking is enabled) in local storage    
          window.localStorage.setItem('bgGPS', 1);
          backgroundGeoLocation.configure(callbackFn, failureFn, {
               desiredAccuracy: 10,      
               stationaryRadius: 10,
               distanceFilter: 10,
               locationProvider: 'ANDROID_ACTIVITY_PROVIDER',
               interval: 10000,      
               fastestInterval: 5000,      
               stopOnStillActivity: false,
               debug: false,      
               stopOnTerminate: false
         });
         backgroundGeoLocation.start();
      };
     // Send location to a backend server, e.g. for location tracking
     postLocation = function post(location) {
         return $http(      
           {        
              method: 'POST',
              headers: {
                   "Content-Type": "application/json"
                    },
              url: 'http://<location server DNS>/location',
              data: {          
                    lat : location.latitude,
                    lng : location.longitude        
                    }      
           }).then(function (response) {
                   return response.data;      
           });  
       }
  
       return {    
            start: start,
            // Stop data tracking    
            stop: function () {
                window.localStorage.setItem('bgGPS', 0);
                backgroundGeoLocation.stop();    
            }  
      }
}]);

The codes above are largely based on this blog article.  I have modified the plugin configuration (see notes below). Details about each configuration parameter can be found in the plugin documentation. I am setting it up and tested for Android here. A few notes or tips below:

  1. Two location providers are supported in Android – Android_distance_filter_provider and Android_activity_provider. I end up using the 2nd one to adjust the intervals in which the app gets location update by setting the parameters interval and fastinterval (to 10 and 5 seconds respectively).
  2. Debug is your friend. Set this to true will trigger a sound and notification when the app receives a location update.
  3. I use a combination of parameters desiredAccuracy, stationaryRadius and distanceFilter to define how often the app should receive location update events. The values set here seems to strike a good balance between accuracy/frequency and battery usage but its largely depends on the app’s use cases and phones.
  4. The plugin includes a url paramter where you can set to the backend server to post the locations. It is also possible, and more flexible, to just implement your own method and call it in the callback function callbackFn 

To kick start the plugin, call init() method in the app.js :

angular.module('app', ['ionic', ...])
.run(function($ionicPlatform, $http, BackgroundGeolocationService) {
   $ionicPlatform.ready(function() {    
       if(window.StatusBar) {
          StatusBar.styleDefault();
       }
       BackgroundGeolocationService.init();
   });
});

Now the Ionic app can send geolocations events when it is in the background, it is rather straight forward to implement a backend server to track and display users’ current locations. Below is a screenshot of a Java app I build, for example:

blog_track.png