Tag: dropwizard

Memory Footprints of hello-world microservices

I was curious reading about openj9 as a JVM being high performance and using a low memory footprint. I was working in a project, where the environment of software systems consisted of around 30 Java Applications, a few running on tomcat, a few on weblogic and the most running on dropwizard microservice framework. The desirable goal for every developer was to start the complete platform on his local notebook and therefor a virtalbox image was build using vagrant. As there were so many applications, each microservice itself consumed around 250MB of RAM and with the number of services growing we already hit 24GB image size of the virtualbox image. I found the blogpost https://codeburst.io/microservices-in-java-never-a7f3a2540dbb which describes the same issue and that if you those java microsevices in a cloud infrastructure, you would even have to pay even more money only because of the memory footprint.

Luckily there was somebody doing a comparsion of openj9 vs hotspot VM in his two blogpots https://royvanrijn.com/blog/2018/05/openj9-jvm-shootout/ and https://royvanrijn.com/blog/2018/05/openj9-hotsport-specjvm2008/ already. He was is showing some benchmarks and results.

My own simple comparison

I tested it on my own (just for curiosity) and came to the same result. In terms of memory consumption, there is a potential improvement with openj9 as JVM alternative.

I picked three helloworld examples using maven archetypes from dropwizard, helidon and Spring Boot.

First I packacked them into docker images using adoptopenjdk/openjdk8-openj9:alpine-slim on the one hand and adoptopenjdk/openjdk8:alpine-slim on the other hand.

Then I compared memory consumption using application metrics and dockers stats and could have a rough idea of the differences. Here is the result of the dropwizard app:

openj9 without any parameters to the jvm:

  • jvm.memory.heap.used = 10563328 B, jvm.memory.total.used = 37838048
  • docker container = 48.55MiB

hotspot without any parameters to the jvm:

  • jvm.memory.heap.used = 28511520 B, jvm.memory.total.used = 61381328
  • docker container = 118.3MiB

Conclusion

Whether you have the goal of running your complete software on your developer laptop or you want to save money running your services in a public cloud, openj9 provides a possibility to reduce the memory footprint of your java application by around 50%.

As it was mentioned in the other blog posts, there are also a few downsides with this, but if you test everything and the requirements (in terms of computation performance) are met, you should give it a try.

Feign Outbound metrics 

With dropwizard microservices, you can easily add inbound metrics on your jax-rs http resource classes via annotations:

@Path("/example")
@Produces(MediaType.TEXT_PLAIN)
public class ExampleResource {
@GET
@Timed
@Metered
@ExceptionMetered
public String show() {
return "yay";
}
}
view raw ExampleResource.java hosted with ❤ by GitHub

The metrics can easily been reported to graphite database and visualized via Kibana.

WYIIWYG – What you instrument, is what you get!

On the other hand, a microservice often contains client libraries to access other services via http. Feign is client library, which provides a wrapper and simplifies the api to communicate to the target services.

In contrast to the inbound metrics from the example above, it is also desirable to monitor the outbound metrics of each of the targeted operations.

Looking at the third-party libraries of http://metrics.dropwizard.io/3.2.3/manual/third-party.html there is already something to retrieve metrics on http level. So in case you are using okhttp as http client implementation you can use https://github.com/raskasa/metrics-okhttp and you will receive information about request durations and connection pools.  Same holds good for Apache httpclient instrumentation.

okhttp example

MetricRegistry metricRegistry = new MetricRegistry();
final ConsoleReporter reporter = ConsoleReporter.forRegistry(metricRegistry).convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS).build();
GitHub github = Feign.builder().invocationHandlerFactory(
// instrumenting feign
new FeignOutboundMetricsDecorator(new InvocationHandlerFactory.Default(), metricRegistry))
// instrumenting ok http
.client(new OkHttpClient(InstrumentedOkHttpClients.create(metricRegistry)))
.decoder(new GsonDecoder()).target(GitHub.class, "https://api.github.com");
execute...
reporter.report();
view raw OkHttpExample.java hosted with ❤ by GitHub

Metric output:

-- Gauges ----------------------------------------------------------------------
okhttp3.OkHttpClient.connection-pool-idle-count
value = 1
okhttp3.OkHttpClient.connection-pool-total-count
value = 1
-- Counters --------------------------------------------------------------------
okhttp3.OkHttpClient.network-requests-running
count = 0
-- Meters ----------------------------------------------------------------------
okhttp3.OkHttpClient.network-requests-completed
count = 1
mean rate = 0,84 events/second
1-minute rate = 0,00 events/second
5-minute rate = 0,00 events/second
15-minute rate = 0,00 events/second
okhttp3.OkHttpClient.network-requests-submitted
count = 1
mean rate = 0,83 events/second
1-minute rate = 0,00 events/second
5-minute rate = 0,00 events/second
15-minute rate = 0,00 events/second
-- Timers ----------------------------------------------------------------------
okhttp3.OkHttpClient.network-requests-duration
count = 1
mean rate = 0,84 calls/second
1-minute rate = 0,00 calls/second
5-minute rate = 0,00 calls/second
15-minute rate = 0,00 calls/second
min = 215,41 milliseconds
max = 215,41 milliseconds
mean = 215,41 milliseconds
stddev = 0,00 milliseconds
median = 215,41 milliseconds
75% <= 215,41 milliseconds
95% <= 215,41 milliseconds
98% <= 215,41 milliseconds
99% <= 215,41 milliseconds
99.9% <= 215,41 milliseconds
view raw gistfile1.txt hosted with ❤ by GitHub

httpclient example

MetricRegistry metricRegistry = new MetricRegistry();
final ConsoleReporter reporter = ConsoleReporter.forRegistry(metricRegistry).convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS).build();
GitHub github = Feign.builder().invocationHandlerFactory(
// instrument feign
new FeignOutboundMetricsDecorator(new InvocationHandlerFactory.Default(), metricRegistry)).client(
// setting an instrumented httpclient
new ApacheHttpClient(InstrumentedHttpClients
.createDefault(metricRegistry, HttpClientMetricNameStrategies.HOST_AND_METHOD)))
.decoder(new GsonDecoder()).target(GitHub.class, "https://api.github.com");
execute...
reporter.report();
view raw HttpClientExample.java hosted with ❤ by GitHub

Metric output:

-- Gauges ----------------------------------------------------------------------
org.apache.http.conn.HttpClientConnectionManager.available-connections
value = 1
org.apache.http.conn.HttpClientConnectionManager.leased-connections
value = 0
org.apache.http.conn.HttpClientConnectionManager.max-connections
value = 20
org.apache.http.conn.HttpClientConnectionManager.pending-connections
value = 0
-- Meters ----------------------------------------------------------------------
-- Timers ----------------------------------------------------------------------
org.apache.http.client.HttpClient.api.github.com.get-requests
count = 1
mean rate = 4,19 calls/second
1-minute rate = 0,00 calls/second
5-minute rate = 0,00 calls/second
15-minute rate = 0,00 calls/second
min = 174,59 milliseconds
max = 174,59 milliseconds
mean = 174,59 milliseconds
stddev = 0,00 milliseconds
median = 174,59 milliseconds
75% <= 174,59 milliseconds
95% <= 174,59 milliseconds
98% <= 174,59 milliseconds
99% <= 174,59 milliseconds
99.9% <= 174,59 milliseconds
view raw gistfile1.txt hosted with ❤ by GitHub

As you can see, the provided metrics only provid information on http level, not really showing differences between different service endpoints. The only differentation is available on the httpclient metrics, which shows metrics based on host and http methods.

Closing the gap

What was missing in my eyes was a way to instrument metrics on the interface level, which is provided from the Feign builder. In my example below I am calling the github API on two different resource endpoints, contributors and repositorySearch. With the instrumentation on http, one is not able to see and monitor those one by one.

Therefore I created a library, which makes it possible to instrument metrics on method or interface level by using annotations like you do it in jersey resource classes.

Using this instrumentation you are able to retrieve metrics based on the interface and methods the client is calling. So for example when you start reporting via JMX, you are able to see the metrics in jconsole.

Usage of the library

To instrument the feign interfaces you basically have to do three things:

  1. add the maven dependency to the pom.xml of your project. 
    <dependency>
  <groupId>com.github.mwiede</groupId>
  <artifactId>metrics-feign</artifactId>
  <version>1.0</version>
</dependency>
  2. add FeignOutboundMetricsDecorator as invocationHandlerFactory in Feign.builder
  3. add the metric annotations @Timed, @Metered and @ExceptionMetered to the interface you are using with feign.

@Timed
@Metered
@ExceptionMetered
interface GitHub {
@RequestLine("GET /repos/{owner}/{repo}/contributors")
List<Contributor> contributors(@Param("owner") String owner, @Param("repo") String repo);
}
static class Contributor {
String login;
int contributions;
}
public static void main(String... args) {
MetricRegistry metricRegistry = new MetricRegistry();
final ConsoleReporter reporter = ConsoleReporter.forRegistry(metricRegistry).convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS).build();
GitHub github = Feign.builder().invocationHandlerFactory(
new FeignOutboundMetricsDecorator(new InvocationHandlerFactory.Default(), metricRegistry))
.decoder(new GsonDecoder()).target(GitHub.class, "https://api.github.com");
// Fetch and print a list of the contributors to this library.
List<Contributor> contributors = github.contributors("mwiede", "metrics-feign");
for (Contributor contributor : contributors) {
System.out.println(contributor.login + " (" + contributor.contributions + ")");
}
reporter.report();
}
view raw Example.java hosted with ❤ by GitHub

The library is available from maven central and the source is hosted at github, so please checkout https://github.com/mwiede/metrics-feign

Jax-RS + Codahale/dropwizard metrics + CDI + Prometheus

I used to use dropwizard built-in metrics annotations and graphite, but now I wanted to integrate these into my javaee project exposing prometheus metrics format. The main difference between graphite and prometheus is the push or pull mentality. So instead of pushing the metrics data to the sink, they are provided via http servlet and the prometheus server is scraping them from there.

There are two registry classes which we have to bring together. One is the com.codahale.metrics.MetricRegistry which holds all codahale metrics and the other one is the io.prometheus.client.CollectorRegistry which holds all metrics being published in the prometheus. So in our case, we will receive all metrics from our Jax-RS resource classes annotated with com.codahale.metrics.annotation.Timed or com.codahale.metrics.annotation.ExceptionMetered.

The prometheus library contains some default Jvm metrics (DefaultExports), but I could not use them because of some sun jdk classes which do not exist in OpenJDK for example. But it is good to add these in addition to the metrics coming from our annotated Jax-RS resource classes. So this is the class

import io.prometheus.client.CollectorRegistry;
import io.prometheus.client.dropwizard.DropwizardExports;
import io.prometheus.client.hotspot.ClassLoadingExports;
import io.prometheus.client.hotspot.GarbageCollectorExports;
import io.prometheus.client.hotspot.MemoryPoolsExports;
import io.prometheus.client.hotspot.ThreadExports;
import io.prometheus.client.hotspot.VersionInfoExports;
import javax.enterprise.context.ApplicationScoped;
import javax.enterprise.context.Destroyed;
import javax.enterprise.context.Initialized;
import javax.enterprise.event.Observes;
import javax.inject.Inject;
import com.codahale.metrics.MetricRegistry;
/**
* A bean, which is starting the metrics reporter during startup and its is shutting down the
* reporter when application is destroyed.
*
*
*/
@ApplicationScoped
public class MetricsBean {
@Inject
private MetricRegistry registry;
public void init(@Observes @Initialized(ApplicationScoped.class) final Object init) {
// DefaultExports.initialize();
new MemoryPoolsExports().register();
new GarbageCollectorExports().register();
new ThreadExports().register();
new ClassLoadingExports().register();
new VersionInfoExports().register();
CollectorRegistry.defaultRegistry.register(new DropwizardExports(registry));
}
public void destroy(@Observes @Destroyed(ApplicationScoped.class) final Object init) {
CollectorRegistry.defaultRegistry.clear();
}
}
view raw MetricsBean.java hosted with ❤ by GitHub

and these are the dependencies for the project:

<dependency>
<groupId>io.astefanutti.metrics.cdi</groupId>
<artifactId>metrics-cdi</artifactId>
<version>1.3.6</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient</artifactId>
<version>0.0.23</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_servlet</artifactId>
<version>0.0.23</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_dropwizard</artifactId>
<version>0.0.23</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_hotspot</artifactId>
<version>0.0.23</version>
</dependency>
view raw pom.xml hosted with ❤ by GitHub

As you can see, by the help of CDI we are able to bind everything together during the startup phase of the application inside of the JEE container.

Then we add the  servlet to the web.xml like this and we are done:

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd" version="3.0">
<welcome-file-list>
<welcome-file>/index.html</welcome-file>
</welcome-file-list>
<!-- -->
<servlet>
<servlet-name>prometheusMetrics</servlet-name>
<servlet-class>io.prometheus.client.exporter.MetricsServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>prometheusMetrics</servlet-name>
<url-pattern>/prometheusMetrics/*</url-pattern>
</servlet-mapping>
</web-app>
view raw Web.xml hosted with ❤ by GitHub

In your prometheus server create a scrape config

scrape_configs:
- job_name: 'jaxrs'
metrics_path: /prometheusMetrics/
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
static_configs:
- targets: ['www.example.com:80']
labels:
group: 'jaxrs'
view raw prometheus.yml hosted with ❤ by GitHub

and you should be able to see you metrics in prometheus expression browser or Grafana.