Month: October 2017

Feign Outbound metrics 

With dropwizard microservices, you can easily add inbound metrics on your jax-rs http resource classes via annotations:

@Path("/example")
@Produces(MediaType.TEXT_PLAIN)
public class ExampleResource {
@GET
@Timed
@Metered
@ExceptionMetered
public String show() {
return "yay";
}
}
view raw ExampleResource.java hosted with ❤ by GitHub

The metrics can easily been reported to graphite database and visualized via Kibana.

WYIIWYG – What you instrument, is what you get!

On the other hand, a microservice often contains client libraries to access other services via http. Feign is client library, which provides a wrapper and simplifies the api to communicate to the target services.

In contrast to the inbound metrics from the example above, it is also desirable to monitor the outbound metrics of each of the targeted operations.

Looking at the third-party libraries of http://metrics.dropwizard.io/3.2.3/manual/third-party.html there is already something to retrieve metrics on http level. So in case you are using okhttp as http client implementation you can use https://github.com/raskasa/metrics-okhttp and you will receive information about request durations and connection pools.  Same holds good for Apache httpclient instrumentation.

okhttp example

MetricRegistry metricRegistry = new MetricRegistry();
final ConsoleReporter reporter = ConsoleReporter.forRegistry(metricRegistry).convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS).build();
GitHub github = Feign.builder().invocationHandlerFactory(
// instrumenting feign
new FeignOutboundMetricsDecorator(new InvocationHandlerFactory.Default(), metricRegistry))
// instrumenting ok http
.client(new OkHttpClient(InstrumentedOkHttpClients.create(metricRegistry)))
.decoder(new GsonDecoder()).target(GitHub.class, "https://api.github.com");
execute...
reporter.report();
view raw OkHttpExample.java hosted with ❤ by GitHub

Metric output:

-- Gauges ----------------------------------------------------------------------
okhttp3.OkHttpClient.connection-pool-idle-count
value = 1
okhttp3.OkHttpClient.connection-pool-total-count
value = 1
-- Counters --------------------------------------------------------------------
okhttp3.OkHttpClient.network-requests-running
count = 0
-- Meters ----------------------------------------------------------------------
okhttp3.OkHttpClient.network-requests-completed
count = 1
mean rate = 0,84 events/second
1-minute rate = 0,00 events/second
5-minute rate = 0,00 events/second
15-minute rate = 0,00 events/second
okhttp3.OkHttpClient.network-requests-submitted
count = 1
mean rate = 0,83 events/second
1-minute rate = 0,00 events/second
5-minute rate = 0,00 events/second
15-minute rate = 0,00 events/second
-- Timers ----------------------------------------------------------------------
okhttp3.OkHttpClient.network-requests-duration
count = 1
mean rate = 0,84 calls/second
1-minute rate = 0,00 calls/second
5-minute rate = 0,00 calls/second
15-minute rate = 0,00 calls/second
min = 215,41 milliseconds
max = 215,41 milliseconds
mean = 215,41 milliseconds
stddev = 0,00 milliseconds
median = 215,41 milliseconds
75% <= 215,41 milliseconds
95% <= 215,41 milliseconds
98% <= 215,41 milliseconds
99% <= 215,41 milliseconds
99.9% <= 215,41 milliseconds
view raw gistfile1.txt hosted with ❤ by GitHub

httpclient example

MetricRegistry metricRegistry = new MetricRegistry();
final ConsoleReporter reporter = ConsoleReporter.forRegistry(metricRegistry).convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS).build();
GitHub github = Feign.builder().invocationHandlerFactory(
// instrument feign
new FeignOutboundMetricsDecorator(new InvocationHandlerFactory.Default(), metricRegistry)).client(
// setting an instrumented httpclient
new ApacheHttpClient(InstrumentedHttpClients
.createDefault(metricRegistry, HttpClientMetricNameStrategies.HOST_AND_METHOD)))
.decoder(new GsonDecoder()).target(GitHub.class, "https://api.github.com");
execute...
reporter.report();
view raw HttpClientExample.java hosted with ❤ by GitHub

Metric output:

-- Gauges ----------------------------------------------------------------------
org.apache.http.conn.HttpClientConnectionManager.available-connections
value = 1
org.apache.http.conn.HttpClientConnectionManager.leased-connections
value = 0
org.apache.http.conn.HttpClientConnectionManager.max-connections
value = 20
org.apache.http.conn.HttpClientConnectionManager.pending-connections
value = 0
-- Meters ----------------------------------------------------------------------
-- Timers ----------------------------------------------------------------------
org.apache.http.client.HttpClient.api.github.com.get-requests
count = 1
mean rate = 4,19 calls/second
1-minute rate = 0,00 calls/second
5-minute rate = 0,00 calls/second
15-minute rate = 0,00 calls/second
min = 174,59 milliseconds
max = 174,59 milliseconds
mean = 174,59 milliseconds
stddev = 0,00 milliseconds
median = 174,59 milliseconds
75% <= 174,59 milliseconds
95% <= 174,59 milliseconds
98% <= 174,59 milliseconds
99% <= 174,59 milliseconds
99.9% <= 174,59 milliseconds
view raw gistfile1.txt hosted with ❤ by GitHub

As you can see, the provided metrics only provid information on http level, not really showing differences between different service endpoints. The only differentation is available on the httpclient metrics, which shows metrics based on host and http methods.

Closing the gap

What was missing in my eyes was a way to instrument metrics on the interface level, which is provided from the Feign builder. In my example below I am calling the github API on two different resource endpoints, contributors and repositorySearch. With the instrumentation on http, one is not able to see and monitor those one by one.

Therefore I created a library, which makes it possible to instrument metrics on method or interface level by using annotations like you do it in jersey resource classes.

Using this instrumentation you are able to retrieve metrics based on the interface and methods the client is calling. So for example when you start reporting via JMX, you are able to see the metrics in jconsole.

Usage of the library

To instrument the feign interfaces you basically have to do three things:

  1. add the maven dependency to the pom.xml of your project. 
    <dependency>
  <groupId>com.github.mwiede</groupId>
  <artifactId>metrics-feign</artifactId>
  <version>1.0</version>
</dependency>
  2. add FeignOutboundMetricsDecorator as invocationHandlerFactory in Feign.builder
  3. add the metric annotations @Timed, @Metered and @ExceptionMetered to the interface you are using with feign.
@Timed
@Metered
@ExceptionMetered
interface GitHub {
@RequestLine("GET /repos/{owner}/{repo}/contributors")
List<Contributor> contributors(@Param("owner") String owner, @Param("repo") String repo);
}
static class Contributor {
String login;
int contributions;
}
public static void main(String... args) {
MetricRegistry metricRegistry = new MetricRegistry();
final ConsoleReporter reporter = ConsoleReporter.forRegistry(metricRegistry).convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS).build();
GitHub github = Feign.builder().invocationHandlerFactory(
new FeignOutboundMetricsDecorator(new InvocationHandlerFactory.Default(), metricRegistry))
.decoder(new GsonDecoder()).target(GitHub.class, "https://api.github.com");
// Fetch and print a list of the contributors to this library.
List<Contributor> contributors = github.contributors("mwiede", "metrics-feign");
for (Contributor contributor : contributors) {
System.out.println(contributor.login + " (" + contributor.contributions + ")");
}
reporter.report();
}
view raw Example.java hosted with ❤ by GitHub

The library is available from maven central and the source is hosted at github, so please checkout https://github.com/mwiede/metrics-feign