Monitoring Spring Boot Applications with Prometheus – Part 2

In my previous post, I describe how to use Prometheus and its JVM client library in a Spring Boot application to gather common JVM metrics. In this blog, I will demonstrate how to write your own application specific metrics using the client library. Prometheus client libraries support 4 core metric types: Counter, Gauge, Histogram and Summary. Its documentation has a brief but clear explanation of each metric types.

I will create 2 metrics to gather statistics about incoming requests to a Spring web application. In particular, a counter to count how many requests the web application handles and a summary for measuring the processing time of the incoming requests.

Application Set Up

To demonstrate the metrics we are going to implement, I have the following controller implementing two endpoints /endpointA and /endpointB. They do nothing except wasting a random amount of time between 0 and 100 ms.

@RestController
public class HomeController {
     private Logger logger = LoggerFactory.getLogger(getClass());
 
     @RequestMapping("/endpointA")
     public void handlerA() throws InterruptedException {
          logger.info("/endpointA");
          Thread.sleep(RandomUtils.nextLong(0, 100));
     }
 
     @RequestMapping("/endpointB")
     public void handlerB() throws InterruptedException {
          logger.info("/endpointB");
          Thread.sleep(RandomUtils.nextLong(0, 100));
     }
}

Counter Metric

A counter is a metric of numeric value that never goes down. Here we add a request handler to keep track of the cumulative number of requests the web application receives:

import io.prometheus.client.Counter;
..
public class RequestCounterInterceptor extends HandlerInterceptorAdapter {

     // @formatter:off
     // Note (1)
     private static final Counter requestTotal = Counter.build()
          .name("http_requests_total")
          .labelNames("method", "handler", "status")
          .help("Http Request Total").register();
     // @formatter:on

     @Override
     public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception e)
 throws Exception {
          // Update counters
          String handlerLabel = handler.toString();
          // get short form of handler method name
          if (handler instanceof HandlerMethod) {
               Method method = ((HandlerMethod) handler).getMethod();
               handlerLabel = method.getDeclaringClass().getSimpleName() + "." + method.getName();
          }
          // Note (2)
          requestTotal.labels(request.getMethod(), handlerLabel, Integer.toString(response.getStatus())).inc();
     }
}

Note:

  1. We implement a counter using io.prometheus.client.Counter class to keep track of number of incoming requests handled by this handler. The counter is named http_requests_total and consists of a number of labels (method, handler and status). A label is an attribute of a metric which can be used in query to filter and aggregate metrics.
  2. The counter is incremented using the Counter’s inc() method. The values of the metric labels method, handler and status are populated with the request http method (get/post), the spring mvc controller and method, and response http status respectively.

Summary Metric

A summary is a complex metric which track a number of observations as well as their counts. See here for a comprehensive explanation of summary and histogram metrics. Similar to previous section, we implement the metric within a request handler class:

import io.prometheus.client.Summary;
...
public class RequestTimingInterceptor extends HandlerInterceptorAdapter {

      private static final String REQ_PARAM_TIMING = "timing";

      // @formatter:off
      // Note (1)
      private static final Summary responseTimeInMs = Summary
           .build()
           .name("http_response_time_milliseconds")
           .labelNames("method", "handler", "status")
           .help("Request completed time in milliseconds")
           .register();
      // @formatter:on

      @Override
      public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
           // Note (2)
           request.setAttribute(REQ_PARAM_TIMING, System.currentTimeMillis());
           return true;
      }

      @Override
      public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex)
 throws Exception {
           Long timingAttr = (Long) request.getAttribute(REQ_PARAM_TIMING);
           long completedTime = System.currentTimeMillis() - timingAttr;
           String handlerLabel = handler.toString();
           // get short form of handler method name
           if (handler instanceof HandlerMethod) {
                Method method = ((HandlerMethod) handler).getMethod();
                handlerLabel = method.getDeclaringClass().getSimpleName() + "." + method.getName();
           }
         // Note (3)
         responseTimeInMs.labels(request.getMethod(), handlerLabel, Integer.toString(response.getStatus())).observe(
 completedTime);
      }
}

Note:

  1. We implement a metric with class io.prometheus.client.Summary. The same set of labels as used in the request counter is used here.
  2. The response time is measured here as time lapsed between the calls to preHandle and afterCompletion methods. This is just for illustration only so I can demonstrate the use of Prometheus client library.
  3. The summary metric is updated by calling the observe() method with the response time value.

Collecting Metrics

Now that we implement the request handler interceptors, we can register them to the test endpoints and start up the Spring Boot web application. Hit the two endpoints using the browser and then go to the Prometheus url, e.g. http://localhost:8080/prometheus, should return something similar to the following:

...
# HELP http_response_time_milliseconds Request completed time in milliseconds
# TYPE http_response_time_milliseconds summary
http_response_time_milliseconds_count{method="GET",handler="HomeController.handlerA",status="200",} 3.0
http_response_time_milliseconds_sum{method="GET",handler="HomeController.handlerA",status="200",} 169.0
http_response_time_milliseconds_count{method="GET",handler="HomeController.handlerB",status="200",} 1.0
http_response_time_milliseconds_sum{method="GET",handler="HomeController.handlerB",status="200",} 59.0
# HELP http_requests_total Http Request Total
# TYPE http_requests_total counter
http_requests_total{method="GET",handler="HomeController.handlerA",status="200",} 3.0
http_requests_total{method="GET",handler="HomeController.handlerB",status="200",} 1.0

Note the summary metric http_response_time_milliseconds actually collects two time series data: _count and _sum. In practice, we don’t require a separate counter metric. It is more to demonstrate how to implement a simple counter with the Prometheus client library.

Also, data for each metric are collected separately for each distinct set of label values. In this example, we have metrics for each individual endpoints given the label values for the label handler is different. Since we also include the response status as a label, we will potentially get other metrics with non 200 status value, for example

http_response_time_milliseconds_count{method="GET",handler="HomeController.handlerA",status="500",} 3.0

Queries in Prometheus

One of the useful features of Prometheus is its powerful query language for manipulating the time series data collected.

For example, the cumulative value of total number of http requests of the counter we implement is not of much use. Typically we would want to get the number of requests a server received per second to get an idea of the system load at various time of the day. This can be achieved by query like the following:

rate(http_requests_total{job="blog", handler="HomeController.handlerA"}[5m])

The above query uses the built-in rate() function to return per-second rate of the http requests measured over the last 5 minutes ([5m]) for the endpoint /endpointA, as specified by the handler filters handler=”HomeController.handlerA”.  The query can be modified to only gather requests that are not successful (not 2xx) by including the label status in the query:

rate(http_requests_total{job="blog", handler="HomeController.handlerA", status!~"^2..$"}[5m])

Prometheus include 2 labels names job and instance for the target as defined in the prometheus.yml configuration file. It is recommended to always include the job name in the query.

Similarly, we can create query to get the average response time over last 5 minutes with the following query

rate(http_response_time_milliseconds_sum{job="blog"}[5m])/rate(http_response_time_milliseconds_count{job="blog"}[5m])

For more details on operations and functions Prometheus support, see their documentation here

This blog post introduces with examples how to implement application specific metrics using Prometheus JVM client library as well as how to use query functions provided by Prometheus to filter and query the data collected. The ability for developers to implement their own metrics together with support with powerful query language by Prometheus is particularly useful when one needs to implement, monitor and analysis specific application functions beyond typical JVM and application metrics.

Monitoring Spring Boot Applications with Prometheus – Part 1

This blog post will demonstrate how to use Prometheus to monitor a spring boot web application. Prometheus is an open source tool for monitoring systems by collecting metrics from target systems as time series data. It supports multiple approaches for instrumenting the application codes. I am going to show how to do this using the Prometheus JVM client library.

Instrumenting with Prometheus JVM client

POM setup

I set up a Spring Boot project in Maven and include the following dependency for the Prometheus JVM client (version 0.0.16):

 <!-- Hotspot JVM metrics -->
 <dependency>
      <groupId>io.prometheus</groupId>
      <artifactId>simpleclient_hotspot</artifactId>
      <version>${prometheus.version}</version>
 </dependency>
 <!-- Exposition servlet -->
 <dependency>
      <groupId>io.prometheus</groupId>
      <artifactId>simpleclient_servlet</artifactId>
      <version>${prometheus.version}</version>
 </dependency>
 <!-- The client -->
 <dependency>
      <groupId>io.prometheus</groupId>
      <artifactId>simpleclient</artifactId>
      <version>${prometheus.version}</version>
 </dependency>

Configure and implement Metric endpoint

The main method for Prometheus to collect metrics is via scraping an endpoint implemented by the target application on regular intervals. To do that, include a Java configuration class as follows:

@Configuration
@ConditionalOnClass(CollectorRegistry.class)
public class PrometheusConfiguration {

     @Bean
     @ConditionalOnMissingBean
     CollectorRegistry metricRegistry() {
         return CollectorRegistry.defaultRegistry;
     }

     @Bean
     ServletRegistrationBean registerPrometheusExporterServlet(CollectorRegistry metricRegistry) {
           return new ServletRegistrationBean(new MetricsServlet(metricRegistry), "/prometheus");
     }

...
}

The above code adds the endpoint (/prometheus) to the Spring Boot application. Now we are ready to add some metrics to it. The Prometheus JVM client includes a number of standard exporters to collect common JVM metrics such as memory and cpu usages. Let’s add them to our new prometheus endpoint

First, we create a exporter register class

/**
 * Metric exporter register bean to register a list of exporters to the default
 * registry
 */
public class ExporterRegister {

     private List<Collector> collectors; 

     public ExporterRegister(List<Collector> collectors) {
          for (Collector collector : collectors) {
              collector.register();
          }
          this.collectors = collectors;
     }

     public List<Collector> getCollectors() {
          return collectors;
     }

}

The above class is just a utility class to register a collection of metric collectors with the registry. Now add the standard exporters from Prometheus JVM client:

import io.prometheus.client.hotspot.MemoryPoolsExports;
import io.prometheus.client.hotspot.StandardExports;
...  
     @Bean
     ExporterRegister exporterRegister() {
           List<Collector> collectors = new ArrayList<>();
           collectors.add(new StandardExports());
           collectors.add(new MemoryPoolsExports());
           ExporterRegister register = new ExporterRegister(collectors);
           return register;
      }

We just added 2 exporters: (1) StandardExports provides CPU usage metrics and (2) MemoryPoolExports add memory usage by the JVM and host. To see what metrics are now available, go to the URL in the browser:

http://localhost:8080/prometheus

The browser should display something like below (truncated as it is too long to list)

# HELP jvm_up_time_seconds System uptime in seconds.
# TYPE jvm_up_time_seconds gauge
jvm_up_time_seconds 15.0
# HELP jvm_cpu_load_percentage JVM CPU Usage %.
# TYPE jvm_cpu_load_percentage gauge
jvm_cpu_load_percentage 37.18078068931383
# HELP os_cpu_load_percentage System CPU Usage %.
# TYPE os_cpu_load_percentage gauge
.
.
.

Install and Setup Prometheus

Now we have implemented the metric endpoint for the Spring Boot application, we are ready to install and configure Prometheus. Following the instruction here to install Prometheus and start up the server. You should now start up and access the server in your browser, e.g. http://localhost:9090/targets

blog_prom_1

By default, Prometheus is configured to monitor itself, handy. Now let’s update the configuration to scrape our Spring Boot app. Open the file prometheus.yml in the Prometheus folder and add the following lines under the scrape_configs section:

 - job_name: 'blog'

scrape_interval: 5s

 metrics_path: '/prometheus'
 static_configs:
 - targets: ['localhost:8080']

Restart Prometheus and refresh your browser to show the following:

blog_prom_2

Prometheus provides a rather basic graphing function. I will show how to integrate Prometheus with other graphing software in a later post. For now, let’s try to display memory usages of the Spring Boot application. Go to Graph tab and select from the dropdown the metric jvm_memory_bytes_used and click Execute buttonThis should end up like the following in the browser. Note the metric has two different set of time series data, head and nonheap usage.

blog_prom_3.png

Summary and What Next

In this blog, I describe how to add monitoring by Prometheus via its JVM client in a Spring Boot application, generate some JVM metrics using the provided exporter classes  and how to configure Prometheus to scrape the data. I end by demonstrating how to use the metrics with the graphing features in Prometheus.

In future blog, I will show how to implement custom metrics in the Spring Boot application using Prometheus JVM client as well as using its expression language to query time series data to return metrics relevant for monitoring purpose. I will also demonstrate how to create dashboard in Grafana using data from Prometheus.

 

Setup Spring Security with Active Directory LDAP in Spring Boot Web Application

This post illustrates how to set up Spring Security in Spring Boot configuration with Active Directory LDAP for a Spring MVC web application. I will also show what needs to be configured for the embedded tomcat to accept HTTPS.

Spring Security with LDAP

To configure Spring Security in Spring Boot, add the following Configuration class to your project. Note the use of annotation @EnableWebMvcSecurity. The configuration class extends the WebSecurityConfigurerAdapter class in Spring Security. More information can be found in the Spring Security Reference here.

@Configuration
@EnableWebMvcSecurity
public class WebSecurityConfig extends WebSecurityConfigurerAdapter {

     @Value("${ldap.domain}")
     private String DOMAIN;

     @Value("${ldap.url}")
     private String URL;

     @Value("${http.port}")
     private int httpPort;

     @Value("${https.port}")
     private int httpsPort;

     @Override
     protected void configure(HttpSecurity http) throws Exception {
          /*
           * Set up your spring security config here. For example...
          */
          http.authorizeRequests().anyRequest().authenticated().and().formLogin().loginUrl("/login").permitAll();
          /*
           * Use HTTPs for ALL requests
          */
          http.requiresChannel().anyRequest().requiresSecure();
          http.portMapper().http(httpPort).mapsTo(httpsPort);
     }

     @Override
     protected void configure(AuthenticationManagerBuilder authManagerBuilder) throws Exception {
          authManagerBuilder.authenticationProvider(activeDirectoryLdapAuthenticationProvider()).userDetailsService(userDetailsService());
     }

     @Bean
     public AuthenticationManager authenticationManager() {
          return new ProviderManager(Arrays.asList(activeDirectoryLdapAuthenticationProvider()));
     }
     @Bean
     public AuthenticationProvider activeDirectoryLdapAuthenticationProvider() {
          ActiveDirectoryLdapAuthenticationProvider provider = new ActiveDirectoryLdapAuthenticationProvider(DOMAIN, URL);
          provider.setConvertSubErrorCodesToExceptions(true);
          provider.setUseAuthenticationRequestCredentials(true);
          return provider;
     }
}

Add HTTPS connector for embedded Tomcat in Spring Boot

Now that Spring Security is set up, you need to update the web server to accept requests from HTTPS. To do that using the embedded Tomcat server in Spring Boot, add the following EmbeddedServletContainerCustomizer bean to the application configuration as shown below. Note I am using anonymous inner classes here instead of lambda expression as I see in other examples for Java 7 compatibility. You will need a keystore file for this to work.

@Bean
EmbeddedServletContainerCustomizer containerCustomizer (

     @Value("${https.port}") final int port, 
     @Value("${keystore.file}") Resource keystoreFile,
     @Value("${keystore.alias}") final String alias, 
     @Value("${keystore.password}") final String keystorePass,
     @Value("${keystore.type}") final String keystoreType) throws Exception {
          final String absoluteKeystoreFile = keystoreFile.getFile().getAbsolutePath();
          return new EmbeddedServletContainerCustomizer() {
               public void customize(ConfigurableEmbeddedServletContainer container) {
                    TomcatEmbeddedServletContainerFactory tomcat = (TomcatEmbeddedServletContainerFactory) container;
                    tomcat.addConnectorCustomizers(new TomcatConnectorCustomizer() {
                         public void customize(Connector connector) {
                              connector.setPort(port);
                              connector.setSecure(true);
                              connector.setScheme("https");
                              Http11NioProtocol proto = (Http11NioProtocol) connector.getProtocolHandler();
                              proto.setSSLEnabled(true);
                              proto.setKeystoreFile(absoluteKeystoreFile);
                              proto.setKeyAlias(alias);
                              proto.setKeystorePass(keystorePass);
                              proto.setKeystoreType(keystoreType);
                        }
               });
           }
     };
 }

 

Getting Start with Spring Boot Configurations

Spring Boot allows development of Spring applications with minimum configuration. This is particular useful for developing microservices. This blog post will demonstrate a few things that may help in understanding how Spring Boot does its job of auto configuring a Spring application.

Auto-Configuration

So what has been configured?

The main feature of Spring Boot is its ability to automatically configuration the Spring application based on its included jar files. So the first thing you may want to look at is what have been configured for you. This can be done by running the application with the debug flag either by adding “–debug” to the command line or JVM argument “-Ddebug“. You will then see the “Auto Configuration Report” displayed in your console like below:

=========================
AUTO-CONFIGURATION REPORT
=========================

Positive matches:
—————–

AopAutoConfiguration
– @ConditionalOnClass classes found: org.springframework.context.annotation.EnableAspectJAutoProxy,org.aspectj.lang.annotation.Aspect,org.aspectj.lang.reflect.Advice (OnClassCondition)
– SpEL expression on org.springframework.boot.autoconfigure.aop.AopAutoConfiguration: ${spring.aop.auto:true} (OnExpressionCondition)

AopAutoConfiguration.JdkDynamicAutoProxyConfiguration
– SpEL expression on org.springframework.boot.autoconfigure.aop.AopAutoConfiguration$JdkDynamicAutoProxyConfiguration: !${spring.aop.proxyTargetClass:false} (OnExpressionCondition)

// rest omitted…

Negative matches:
—————–

ActiveMQAutoConfiguration
– required @ConditionalOnClass classes not found: javax.jms.ConnectionFactory,org.apache.activemq.ActiveMQConnectionFactory (OnClassCondition)

// rest omitted…

The report should give you an indication of what have been configured for you. It depends mainly on what jar files you have included in your project dependencies.

Excluding an auto configuration

Spring Boot is designed so that you can gradually replaced the auto-configuration as needed. To exclude an auto configuration, use the exclude attribute of the @EnableAutoConfiguration annotation as below:

@Configuration
@ComponentScan
@EnableAutoConfiguration(exclude = ProcessEngineAutoConfiguration.class)
@ImportResource("classpath:/activiti.xml")
public class Application {

public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}

I am using the Activiti Spring Boot module here. Note instead of relying on its auto configuration setup, I have included the activiti.xml configuration file using the @ImportResource annotation to manually configure the process engine used by Activiti.

Configuration properties

Application properties, e.g. JDBC connection string, are to be set in the application.properties file in the classpath. Profile specific properties should be included in a separate application-<profile>.properties file located at the same directory of the application.properties file.

A list of commonly used property keys can be found in the Reference Guide (here). For example, to change the embedded web server port to 8181, add the following line the the application.properties file

### application.properties

server.port=8181

Logging

You may configure Spring Boot to use logging framework of your choice. But first, it may be useful to configure the logger properties and understand what has been configured for you. To do this, add the property of the format logging.level.<package name>=<level> to your application.properties file. For example, to display debug messages for Hibernate, add the following lilne:

logging.level.org.hibernate=DEBUG

That’s it. The above has helped me to get started with Spring Boot, to understand how auto configuration works. The Reference Guide provides a comprehensive documentation of the framework and various how-tos. There are also many tutorials and blog articles around for reference.