Monitoring Spring Boot Applications with Prometheus

This blog post will demonstrate how to use Prometheus to monitor a spring boot web application. Prometheus is an open source tool for monitoring systems by collecting metrics from target systems as time series data. It supports multiple approaches for instrumenting the application codes. I am going to show how to do this using the Prometheus JVM client library.

Instrumenting with Prometheus JVM client

POM setup

I set up a Spring Boot project in Maven and include the following dependency for the Prometheus JVM client (version 0.0.16):

 <!-- Hotspot JVM metrics -->
 <dependency>
      <groupId>io.prometheus</groupId>
      <artifactId>simpleclient_hotspot</artifactId>
      <version>${prometheus.version}</version>
 </dependency>
 <!-- Exposition servlet -->
 <dependency>
      <groupId>io.prometheus</groupId>
      <artifactId>simpleclient_servlet</artifactId>
      <version>${prometheus.version}</version>
 </dependency>
 <!-- The client -->
 <dependency>
      <groupId>io.prometheus</groupId>
      <artifactId>simpleclient</artifactId>
      <version>${prometheus.version}</version>
 </dependency>

Configure and implement Metric endpoint

The main method for Prometheus to collect metrics is via scraping an endpoint implemented by the target application on regular intervals. To do that, include a Java configuration class as follows:

@Configuration
@ConditionalOnClass(CollectorRegistry.class)
public class PrometheusConfiguration {

     @Bean
     @ConditionalOnMissingBean
     CollectorRegistry metricRegistry() {
         return CollectorRegistry.defaultRegistry;
     }

     @Bean
     ServletRegistrationBean registerPrometheusExporterServlet(CollectorRegistry metricRegistry) {
           return new ServletRegistrationBean(new MetricsServlet(metricRegistry), "/prometheus");
     }

...
}

The above code adds the endpoint (/prometheus) to the Spring Boot application. Now we are ready to add some metrics to it. The Prometheus JVM client includes a number of standard exporters to collect common JVM metrics such as memory and cpu usages. Let’s add them to our new prometheus endpoint

First, we create a exporter register class

/**
 * Metric exporter register bean to register a list of exporters to the default
 * registry
 */
public class ExporterRegister {

     private List<Collector> collectors; 

     public ExporterRegister(List<Collector> collectors) {
          for (Collector collector : collectors) {
              collector.register();
          }
          this.collectors = collectors;
     }

     public List<Collector> getCollectors() {
          return collectors;
     }

}

The above class is just a utility class to register a collection of metric collectors with the registry. Now add the standard exporters from Prometheus JVM client:

import io.prometheus.client.hotspot.MemoryPoolsExports;
import io.prometheus.client.hotspot.StandardExports;
...  
     @Bean
     ExporterRegister exporterRegister() {
           List<Collector> collectors = new ArrayList<>();
           collectors.add(new StandardExports());
           collectors.add(new MemoryPoolsExports());
           ExporterRegister register = new ExporterRegister(collectors);
           return register;
      }

We just added 2 exporters: (1) StandardExports provides CPU usage metrics and (2) MemoryPoolExports add memory usage by the JVM and host. To see what metrics are now available, go to the URL in the browser:

http://localhost:8080/prometheus

The browser should display something like below (truncated as it is too long to list)

# HELP jvm_up_time_seconds System uptime in seconds.
# TYPE jvm_up_time_seconds gauge
jvm_up_time_seconds 15.0
# HELP jvm_cpu_load_percentage JVM CPU Usage %.
# TYPE jvm_cpu_load_percentage gauge
jvm_cpu_load_percentage 37.18078068931383
# HELP os_cpu_load_percentage System CPU Usage %.
# TYPE os_cpu_load_percentage gauge
.
.
.

Install and Setup Prometheus

Now we have implemented the metric endpoint for the Spring Boot application, we are ready to install and configure Prometheus. Following the instruction here to install Prometheus and start up the server. You should now start up and access the server in your browser, e.g. http://localhost:9090/targets

blog_prom_1

By default, Prometheus is configured to monitor itself, handy. Now let’s update the configuration to scrape our Spring Boot app. Open the file prometheus.yml in the Prometheus folder and add the following lines under the scrape_configs section:

 - job_name: 'blog'

scrape_interval: 5s

 metrics_path: '/prometheus'
 static_configs:
 - targets: ['localhost:8080']

Restart Prometheus and refresh your browser to show the following:

blog_prom_2

Prometheus provides a rather basic graphing function. I will show how to integrate Prometheus with other graphing software in a later post. For now, let’s try to display memory usages of the Spring Boot application. Go to Graph tab and select from the dropdown the metric jvm_memory_bytes_used and click Execute buttonThis should end up like the following in the browser. Note the metric has two different set of time series data, head and nonheap usage.

blog_prom_3.png

Summary and What Next

In this blog, I describe how to add monitoring by Prometheus via its JVM client in a Spring Boot application, generate some JVM metrics using the provided exporter classes  and how to configure Prometheus to scrape the data. I end by demonstrating how to use the metrics with the graphing features in Prometheus.

In future blog, I will show how to implement custom metrics in the Spring Boot application using Prometheus JVM client as well as using its expression language to query time series data to return metrics relevant for monitoring purpose. I will also demonstrate how to create dashboard in Grafana using data from Prometheus.

 

Using catch-exception in JUnit Tests

I use ExpectedException a lot in my JUnit test to verify exceptions thrown. In some circumstances, you would also want to run some assertions in the tests after the exception is thrown. This is impossible with ExpectedException. In this post, I will demonstrate how to use catch-exception instead to achieve this.

Set up

Let say we have the following class we want to test:

public class MyService {

 private MyDao dao;
 
 public MyService(MyDao dao) {
      this.dao = dao;
 }
 
 public void someMethod() {
      dao.beforeException();
      // Some codes here. Throw an exception for demo purpose
      if (true) {
           throw new NullPointerException("NPE!");
      }
      dao.afterException();
 }
 
}

Note the method someMethod() will throw an NullPointerException. This simulates real codes that throws an exception when the object in certain state or when one of the methods it calls within returns with an exception.

ExpectedException

ExpectedException works by wrapping the entire test method by its own try-catch block. As a result, any assertion statements after the method that throws the exception are not called.

public class MyTest {

 @Rule
 public ExpectedException none = ExpectedException.none();
 
 @Test
 public void testUsingExpectedException() {
      MyDao mockDao = mock(MyDao.class);
      MyService service = new MyService(mockDao);
      none.expect(NullPointerException.class);
      service.someMethod();
 
      // Note the following line is not called. Test still passes
      verify(mockDao).afterException();
 }

Note the test above verifies that the NPE is thrown as expected. The dao’s afterException method is never called and the test will still pass.

Catch-Exception

Catch-exception is a library that would address the issue above with ExpectedException. To use it, include the following Maven dependency

 <dependency>
      <groupId>eu.codearte.catch-exception</groupId>
      <artifactId>catch-exception</artifactId>
      <version>1.4.4</version>
      <scope>test</scope> <!-- test scope to use it only in tests -->
 </dependency>

and update the test as follows

 @Test
 public void testUsingCatchException() {
      MyDao mockDao = mock(MyDao.class);
      MyService service = new MyService(mockDao);
 
      catchException(service).someMethod();
 
      // Hamcrest
      assertThat(caughtException(), instanceOf(NullPointerException.class));
 
      // Mockito verify got called
      verify(mockDao).beforeException();
      verify(mockDao, never()).afterException();
 }

Now the assertion (mockito verify) statements are called. This could be useful if you want to verify that the object is in the correct state or as in the example certain methods of the object’s member is (not) called when the method throws an exception.

Securing Multiple Resources in OAuth2 Resource Server using Spring Security

In my previous blog post I demonstrate how to setup an OAuth2 authorization server using Spring Security. In this post, I will demonstrate how to setup the security configurations in a resource server to secure multiple resources.

The example here uses Spring Boot 1.2.7 and is a standalone OAuth2 resource server which secures multiple resources with their own ids and access rules. To do that, instead of using @EnableResourceServer, we have to define a ResourceServerConfiguration bean for each resource to be secured as shown below

@Configuration
@EnableOAuth2Resource
public class OAuth2ServerConfig {
     @Bean
     protected ResourceServerConfiguration stockesources() {
          ResourceServerConfiguration resource = new ResourceServerConfiguration(){
                // Switch off the Spring Boot @Autowired configurers
                public void setConfigurers(List<ResourceServerConfigurer> configurers) {
                    super.setConfigurers(configurers);
                }
                @Override
                public int getOrder() {
                     return 30;
                }
     };
           resource.setConfigurers(Arrays.<ResourceServerConfigurer> asList(new ResourceServerConfigurerAdapter() {
                 @Override
                 public void configure(ResourceServerSecurityConfigurer resources) throws Exception {
                      resources.resourceId("stock");
                  }
                  @Override
                  public void configure(HttpSecurity http) throws Exception {
                        http.antMatcher("/stock/**").authorizeRequests().anyRequest().access("#oauth2.hasScope('read')");
                  }
             }));
            return resource;
}

To secure the resource “stock”, we first implements a bean of ResourceServerConfiguration by manually setting the configurers by calling the super.setConfigurers() method and then set the order by overriding the getOrder() method. Then, security configuration for the resource can then be set up by using the return ResourceServerConfiguration object (variable resource) by implementing an anonymous ResourceSourceServerConfigurerAdapter.

To secure another resource, just define another bean of ResourceServerConfiguration similar to the above with a different resource id and its own OAuth2 access rules. Also, the order value has to be unique.

Note that I have to override the getOrder() method in the ResourceServerConfiguration here instead of line like

     ResourceServerConfiguration resource = ...
     resource.setOrder(30)
     resource.setConfigurers(...

or spring security will throw exception like below

Caused by: java.lang.IllegalStateException: @Order on WebSecurityConfigurers must be unique. Order of 2147483626 was already used, so it cannot be used on . OAuth2ServerConfig$1@1a40489f too.

 

Writing parameterized Tests with Spring and JUnit 4

This blog will demonstrate how to write parameterized tests with JUnit4 with Spring.

Let say we have the following interface

public interface Logic {
     boolean xor(boolean a, boolean b);
}

and the corresponding implementation annotated as a Spring service component

@Service
public class LogicImpl implements Logic {
      @Override
      public boolean xor(boolean a, boolean b) {
            return a ^ b;
      }
}

To test the above class with different input combinations of arguments a and b, we can write the following parameterized test in JUnit

@RunWith(Parameterized.class) // Note 1
@SpringApplicationConfiguration(classes = BlogApplication.class)
public class LogicImplTest {

     @Autowired
     private LogicImpl logic;

     // Manually config for spring to use Parameterised
     private TestContextManager testContextManager;

     @Parameter(value = 0) // Note 3i
     public boolean a;

     @Parameter(value = 1) // Note 3ii
     public boolean b;

     @Parameter(value = 2) // Note 3iii
     public boolean expected;

     @Parameters // Note 4
     public static Collection<Object[]> data() {
          Collection<Object[]> params = new ArrayList<>();
          params.add(new Object[] { true, true, false});
          params.add(new Object[] { true, false, true});
          params.add(new Object[] { false, true, true});
          params.add(new Object[] { false, false, false});

          return params;
      }

     @Before // Note 2
     public void setUp() throws Exception {
          this.testContextManager = new TestContextManager(getClass());
          this.testContextManager.prepareTestInstance(this);
     }

     @Test // Note 5
     public void testXor() {
          assertThat(logic.xor(a, b), equalTo(expected));
     }

}

A few things to note here:

  1. The test class is to be run with the Parameterized runner, instead of the normal SpringJUnit4ClassRunner class.
  2. We need to manually configure the test context manager as in the @Before method. This is typically done automatically by Spring
  3. Parameters are defined as public members of the class (as in 3i to 3iii) with the @Parameter annotation. Since we have more than 1 parameter, it is also neccessary to set the value attribute. This defines the index of the parameters to use.
  4. Parameter values are set by implementing a static  method and annotate it with @Parameters. In our example, the data() method returns a list of object arrays. Each value of the list params corresponds to a set of parameter values.
  5.  Tests now can use the parameter values.

Running the test class in Eclipse will give you something like this

blogPTest

Note testXor() is run 4 times, each using the parameter set of the values defined in the list returned by the data() method.

 

Setup Spring RestTemplate to accept Self Signed Cert

This is strictly for testing only but may be useful if you need to perform integration tests. For example, the system you develop needs to access another internal or 3rd party test server via https where the server’s certificate is not signed.

PKIX path building failed

By default, if you try to access a server via https with a self signed certificate, for example with the following codes

RestTemplate template = new TestRestTemplate();
template.getForObject(https://<some server>/, String.class);

you will get the following exception:

org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<some server>":
sun.security.validator.ValidatorException: PKIX path building failed: 
sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target;

HttpClient

To fix the above, update the RestTemplate with a custom HttpClient that accepts self-signed certificate:

      SSLConnectionSocketFactory socketFactory = new SSLConnectionSocketFactory(new SSLContextBuilder().loadTrustMaterial(null, new TrustSelfSignedStrategy()).build());

      HttpClient httpClient = HttpClients.custom().setSSLSocketFactory(socketFactory).build();

      RestTemplate template = new TestRestTemplate();
      ((HttpComponentsClientHttpRequestFactory) template.getRequestFactory()).setHttpClient(httpClient);

and add the following dependency if needed

      <dependency>
           <groupId>org.apache.httpcomponents</groupId>
           <artifactId>httpclient</artifactId>
           <scope>test</scope>
      </dependency>

Again this should only be used for testing purpose only.

Configuring Multiple JPA Entity Managers In Spring Boot

This blog will demonstrate how to setup multiple entity managers in Spring to connect to different data sources. The solution here also supports Spring Data.

Update Maven Pom file

Include Spring Boot dependency for Spring Data:

 <dependency>
       <groupId>org.springframework.boot</groupId>
       <artifactId>spring-boot-starter-data-jpa</artifactId>
 </dependency>

Disable DataSourceAutoConfiguration

Since we are setting up the data sources, disable the auto configuration in Spring Boot

@Configuration
@ComponentScan
@EnableAutoConfiguration (exclude = {  DataSourceAutoConfiguration.class })
public class Application {
   ...

Configure Primary Entity Manager

Below is the Java configuration for the primary entity manager

@Profile("!test")          // 1
@Configuration
@EnableTransactionManagement
@EnableJpaRepositories(basePackages = "au.com.myblog.dao", entityManagerFactoryRef = "entityManager", transactionManagerRef = "transactionManager")        // 2
public class PrimaryMysqlDBConfiguration {
     @Bean(name = "dataSource")      // 3
     @Primary
     @ConfigurationProperties(prefix = "primary.datasource.mysql")
     public DataSource mysqlDataSource() {
          return DataSourceBuilder.create().build();
     }
    @PersistenceContext(unitName = "primary")   // 4
     @Primary
     @Bean(name = "entityManager")
     public LocalContainerEntityManagerFactoryBean mySqlEntityManagerFactory(EntityManagerFactoryBuilder builder) {
          return builder.dataSource(mysqlDataSource()).persistenceUnit("primary").properties(jpaProperties())
                    .packages("au.com.myblog.domain").build();
      }
     private Map<String, Object> jpaProperties() {
          Map<String, Object> props = new HashMap<>();
          props.put("hibernate.ejb.naming_strategy", new SpringNamingStrategy());
          return props;
      }
}

Note:

  1. @Profile annotation to use this configuration only for non test profile. This allows me to set a different datasource, e.g. H2, when running tests.
  2. @EnableJpaRepositories is used for Spring Data. Note we are using the default transaction manager setup in Spring
  3. Bean for primary datasource. The @ConfigurationProperties annotation specifies the prefix of the properties to use by this datasource. For the example here, the application.properties file should include properties like below:
    # DB Connection
    primary.datasource.mysql.url=jdbc:log4jdbc:mysql://localhost/bpmn_service
    primary.datasource.mysql.username=root
    primary.datasource.mysql.password=root
    primary.datasource.mysql.driver-class-name=net.sf.log4jdbc.DriverSpy
  4. Persistence unit name should be setup in the EntityManager bean as shown here.

Configure Secondary Entity Manager

Configuration of the secondary entity manager is very similar to that of the primary.The only thing is we have to define a new transaction manager. Make sure a different prefix is set in the configuration properties of the data source. Also, a different persistent unit name should be used.

@Profile("!test")
@Configuration
@EnableTransactionManagement
@EnableJpaRepositories(basePackages = "au.com.myblog.dao", entityManagerFactoryRef = "secondaryMySqlEntityManager", transactionManagerRef = "secondaryTransactionManager")
public class SecondaryMysqlDBConfiguration {
      @Bean
      @ConfigurationProperties(prefix = "secondary.datasource.mysql")
       public DataSource mysqlDataSource() {
            return DataSourceBuilder.create().build();
       }
      @PersistenceContext(unitName = "secondary")
      @Bean(name = "secondaryMySqlEntityManager")
      public LocalContainerEntityManagerFactoryBean mySqlEntityManagerFactory(EntityManagerFactoryBuilder builder) {
           return  builder.dataSource(mysqlDataSource()).properties(jpaProperties()).persistenceUnit("secondary").packages("au.com.myblog.domain").build();
       }
      @Bean(name = "secondaryTransactionManager")
       public PlatformTransactionManager transactionManager(EntityManagerFactoryBuilder builder) {
             JpaTransactionManager tm = new JpaTransactionManager();
             tm.setEntityManagerFactory(mySqlEntityManagerFactory(builder).getObject());
             tm.setDataSource(mysqlDataSource());
             return tm;
       }
       private Map<String, Object> jpaProperties() {
             Map<String, Object> props = new HashMap<>();
            // naming strategy to put underscores instead of camel case
            // as per auto JPA Configuration
            props.put("hibernate.ejb.naming_strategy", new SpringNamingStrategy());
            props.put("hibernate.hbm2ddl.auto", "update");
            return props;
       }

Note the setup above assumes the primary and secondary data sources are not used together in a single transaction. Hence we can use 2 independent transaction managers.

Specify which Entity Manager to use

Finally, make sure that you specify the name of the secondary transaction in the spring @Transactional annotation:

@Transactional(value = "secondaryTransactionManager")

Also, add the @PersistenceContext annotation with the unit name as defined in the configuration when injecting entity managers;

 @PersistenceContext(unitName = "secondary")
 private EntityManager entityManager;

That’s it!