Getting started with QBit Microservices Stats KPI Metrics Monitoring

KPI Microservices Monitoring

There has been a lot written on the subject of Microservices Monitoring. Monitoring is a bit of an overloaded term. There is service health monitoring, which can be done with tools like Mesosphere/Marathon, Nomad, Consul, etc. There is also KPI monitoring, which is done with tools like Grafana, Graphite, InfluxDB, StatsD, etc. Then there is log monitoring and search with tools like the ELK stack (elastic-search, LogStash and Kibana) and Splunk, where you can easily trace logs down to the requests or client ID in a request header. And, then there is system monitoring (JVM, slow query logs, network traffic, etc.), with tools like SystemD, and more. You will want all of this when you are doing Microservices Development.

The more insight you have into your system, the easier it will be to support and debug. Microservices imply async distributed development. Doing async distributed development without monitoring is like running with scissors.

To summarize Microservices Monitoring is:

• KPI Monitoring (e.g., StatsD, Grafana, Graphite, InfluxDB, etc.)
• Health Monitoring (e.g., Consul, Nomad, Mesosphere/Marathon, Heroku, etc.)
• Log monitoring (e.g., ELK stack, Splunk, etc.)

QBit has support for ELK/Splunk by providing support for MDC. QBit has support for systems that can monitor health like Mesosphere/Marathon, Heroku, Consul, Nomad, etc. by having an internal health system that QBit service actors all check-in with that then gets rolled up to other systems like Mesosphere/Marathon, Heroku, Consul, Nomad, etc.

In this tutorial we are going to just cover KPI monitoring for microservices which is sometimes called Metrics Monitoring or Stats Monitoring. KPI stands for Key Performance Indicators. These are the things you really care about to see if your system is up and running, and how hard it is getting hit, and how it is performing.

At the heart of the QBit KPI system is the Metrics collector. QBit uses the Metrik interface for tracking Microservice KPIs.

Metrik Interface for tracking KPIs

public interface MetricsCollector {

    default void increment(final String name) {
        recordCount(name, 1);
    }

    default void recordCount(String name, long count) {
    }

    default void recordLevel(String name, long level) {
    }

    default void recordTiming(String name, long duration) {
    }

}

We are recording counts per time period, current level or gauge at this instance in time and timings which is how long did something take.

Demonstrating using QBit metrics

This guide assumes you have read through the main overview of QBit and have gone through the first tutorials, but you should be able to follow along if you have not, you just will be able to follow along better if you read the docs (at least skimmed) and went through the first set of tutorials.

Let's show it. First we need to build. Use Gradle as follows:

gradle.build

group 'qbit-ex'
version '1.0-SNAPSHOT'
apply plugin: 'java'
apply plugin: 'application'
mainClassName = "com.mammatustech.todo.TodoServiceMain"

compileJava {
    sourceCompatibility = 1.8
}

repositories {
    mavenCentral()
    mavenLocal()
}

dependencies {
    testCompile group: 'junit', name: 'junit', version: '4.11'

    /* Used for reactive Java programming. */ 
    compile 'io.advantageous.reakt:reakt:2.5.0.RELEASE'

    /* Used to simplify QBit config and for integration with Consul, StatsD, Heroku, etc. */
    compile group: 'io.advantageous.qbit', name: 'qbit-admin', version: '1.3.0.RELEASE'

    /* Vertx is used as the network driver for QBit. */
    compile group: 'io.advantageous.qbit', name: 'qbit-vertx', version: '1.3.0.RELEASE'

}

Read the comments in all of the code listings.

In this example we will use StatsD but QBit is not limited to StatsD for Microservice KPI monitoring. In fact QBit can do amazing things with its StatsService like clustered rate limiting based on OAuth header info, but that is beyond this tutorial.

StatsD is a protocol that you can send KPI messages to over UDP. Digital Ocean has a really nicedescription of StatsD.

The easiest way to setup StatsD is to use Docker. Docker is a great tool for development, and using Docker with Nomad, CoreOS, Mesosphere/Marathon, etc. is a great way to deploy Docker containers, but at a minimum you should be using the Docker tools for development.

Set up the StatsD server stack by using this public docker container.

QBit ships with StatsD support in the qbit-admin lib (jar). It has done this for a long time.

We will connect to StatsD with this URI.

URI to connect to with StatsD

final URI statsdURI = URI.create("udp://192.168.99.100:8125");

Depending on how you have Docker setup, your URI might look a bit different. If you are running Docker tools with a Mac, then that should be your URI. (On Linux the above IO is likely to be localhost not 192.168.99.100, go through the docker tool tutorials if you are lost at this point. It will be worth your time. I promise. I promise.invoke promise.)

If you have not already followed the instructions at statsD, grafana, influxdb docker container docs, do so now.

Running Docker

docker run -d \
  --name docker-statsd-influxdb-grafana \
  -p 3003:9000 \
  -p 3004:8083 \
  -p 8086:8086 \
  -p 22022:22 \
  -p 8125:8125/udp \
  samuelebistoletti/docker-statsd-influxdb-grafana

The above yields

Servers

Host        Port        Service

3003        9000            grafana        to see the results
8086        8086            influxdb       to store the results 
3004        8083            influxdb-admin to query the results
8125        8125            statsd         server that listens to statsD UPD messages
22022       22              sshd

If you want to see the metrics and see if this is working, go through the influxDB tutorial and look around at the measurements with the influx-admin. Influx is a time series database. Grafana allows you to see pretty graphs and charts of the microservice KPIs that we are collecting. You will want to learn grafana as well.

We use the host and port of the URI to connect to the StatsD daemon that is running on the docker container.

Setting up StatsD by using QBit managedServiceBuilder

        managedServiceBuilder.getStatsDReplicatorBuilder().setHost(host).setPort(port);
        managedServiceBuilder.setEnableStatsD(true);

We covered using and setting up the managedServiceBuilder in the first tutorials, and the complete code listing is below.

Use the managedServiceBuilder to create the StatsCollector/MetricsCollector

        StatsCollector statsCollector = managedServiceBuilder.createStatsCollector();

        /* Start the service. */
        managedServiceBuilder.addEndpointService(new TodoService(reactor, statsCollector))

The QBit StatsCollector interface extends the Metrik MetricsCollector interface (from QBit 1.5 onwards).

Using the StatsCollector.

Then we just need to use it.

Using the StatsCollector to collect KPIs about our service

For kicks, we track the KPI todoservice.i.am.alive every three seconds.

Tracking KPI i.am.am.alive

@RequestMapping("/todo-service")
public class TodoService {
...
        /** Send stat count i.am.alive every three seconds.  */
        this.reactor.addRepeatingTask(Duration.ofSeconds(3),
                () -> statsCollector.increment("todoservice.i.am.alive"));

Tracking calls to add method

    @RequestMapping(value = "/todo", method = RequestMethod.POST)
    public void add(final Callback<Boolean> callback, final Todo todo) {

        /** Send KPI add.called every time the add method gets called. */
        statsCollector.increment("todoservice.add.called");
        todoMap.put(todo.getId(), todo);
        callback.accept(true);
    }

Tracking calls to remove method

    @RequestMapping(value = "/todo", method = RequestMethod.DELETE)
    public void remove(final Callback<Boolean> callback, final @RequestParam("id") String id) {

        /** Send KPI add.removed every time the remove method gets called. */
        statsCollector.increment("todoservice.remove.called");
        Todo remove = todoMap.remove(id);
        callback.accept(remove!=null);

    }

Managing callbacks and repeating tasks

    @QueueCallback({EMPTY, IDLE, LIMIT})
    public void process() {
        reactor.process();
    }

Complete example

Todo.java

package com.mammatustech.todo;

public class Todo {

    private  String id;

    private final String name;
    private final String description;
    private final long createTime;

    public Todo(String name, String description, long createTime) {
        this.name = name;
        this.description = description;
        this.createTime = createTime;

        this.id = name + "::" + createTime;
    }


    public String getId() {
        if (id == null) {
            this.id = name + "::" + createTime;
        }
        return id;
    }

    public String getName() {
        return name;
    }

    public String getDescription() {
        return description;
    }

    public long getCreateTime() {
        return createTime;
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;

        Todo todo = (Todo) o;

        if (createTime != todo.createTime) return false;
        return !(name != null ? !name.equals(todo.name) : todo.name != null);

    }

    @Override
    public int hashCode() {
        int result = name != null ? name.hashCode() : 0;
        result = 31 * result + (int) (createTime ^ (createTime >>> 32));
        return result;
    }
}

TodoService.java to show tracking KPIs.

package com.mammatustech.todo;

import io.advantageous.qbit.annotation.*;
import io.advantageous.qbit.reactive.Callback;
import io.advantageous.qbit.service.stats.StatsCollector;
import io.advantageous.reakt.reactor.Reactor;

import java.time.Duration;
import java.util.*;

import static io.advantageous.qbit.annotation.QueueCallbackType.*;


@RequestMapping("/todo-service")
public class TodoService {


    private final Map<String, Todo> todoMap = new TreeMap<>();

    /** Used to manage callbacks and such. */
    private final Reactor reactor;

    /** Stats Collector for KPIs. */
    private final StatsCollector statsCollector;

    public TodoService(Reactor reactor, StatsCollector statsCollector) {
        this.reactor = reactor;
        this.statsCollector = statsCollector;

        /** Send stat count i.am.alive every three seconds.  */
        this.reactor.addRepeatingTask(Duration.ofSeconds(3),
                () -> statsCollector.increment("todoservice.i.am.alive"));

        this.reactor.addRepeatingTask(Duration.ofSeconds(1), statsCollector::clientProxyFlush);
    }


    @RequestMapping(value = "/todo", method = RequestMethod.POST)
    public void add(final Callback<Boolean> callback, final Todo todo) {

        /** Send KPI add.called every time the add method gets called. */
        statsCollector.increment("todoservice.add.called");
        todoMap.put(todo.getId(), todo);
        callback.accept(true);
    }



    @RequestMapping(value = "/todo", method = RequestMethod.DELETE)
    public void remove(final Callback<Boolean> callback, final @RequestParam("id") String id) {

        /** Send KPI add.removed every time the remove method gets called. */
        statsCollector.increment("todoservice.remove.called");
        Todo remove = todoMap.remove(id);
        callback.accept(remove!=null);

    }



    @RequestMapping(value = "/todo", method = RequestMethod.GET)
    public void list(final Callback<ArrayList<Todo>> callback) {

        /** Send KPI add.list every time the list method gets called. */
        statsCollector.increment("todoservice.list.called");

        callback.accept(new ArrayList<>(todoMap.values()));
    }


    @QueueCallback({EMPTY, IDLE, LIMIT})
    public void process() {
        reactor.process();
    }


}

TodoServiceMain.java showing how to configure StatsD QBit for MicroService KPI tracking

package com.mammatustech.todo;


import io.advantageous.qbit.admin.ManagedServiceBuilder;
import io.advantageous.qbit.service.stats.StatsCollector;
import io.advantageous.reakt.reactor.Reactor;

import java.net.URI;
import java.time.Duration;
import java.util.Objects;

public class TodoServiceMain {


    public static void main(final String... args) throws Exception {

        //To test locally use https://hub.docker.com/r/samuelebistoletti/docker-statsd-influxdb-grafana/
        final URI statsdURI = URI.create("udp://192.168.99.100:8125");

        //For timer
        final Reactor reactor = Reactor.reactor();


        /* Create the ManagedServiceBuilder which manages a clean shutdown, health, stats, etc. */
        final ManagedServiceBuilder managedServiceBuilder =
                ManagedServiceBuilder.managedServiceBuilder()
                        .setRootURI("/v1") //Defaults to services
                        .setPort(8888); //Defaults to 8080 or environment variable PORT

        enableStatsD(managedServiceBuilder, statsdURI);

        StatsCollector statsCollector = managedServiceBuilder.createStatsCollector();



        /* Start the service. */
        managedServiceBuilder.addEndpointService(new TodoService(reactor, statsCollector)) //Register TodoService
                .getEndpointServerBuilder()
                .build().startServer();

        /* Start the admin builder which exposes health end-points and swagger meta data. */
        managedServiceBuilder.getAdminBuilder().build().startServer();

        System.out.println("Todo Server and Admin Server started");

    }

    /**
     * Enable Stats D.
     *
     * @param host statsD host
     * @param port statsD port
     */
    public static void enableStatsD(ManagedServiceBuilder managedServiceBuilder, String host, int port) {
        if (port < 1) throw new IllegalStateException("StatsD port must be set");
        Objects.requireNonNull(host, "StatsD Host cannot be null");
        if (host.isEmpty()) throw new IllegalStateException("StatsD Host name must not be empty");
        managedServiceBuilder.getStatsDReplicatorBuilder().setHost(host).setPort(port);
        managedServiceBuilder.setEnableStatsD(true);
    }

    /**
     * Enable Stats D.
     *
     * @param uri for statsd
     */
    public static void enableStatsD(ManagedServiceBuilder managedServiceBuilder, URI uri) {
        if (!uri.getScheme().equals("udp")) throw new IllegalStateException("Scheme must be udp");
        enableStatsD(managedServiceBuilder, uri.getHost(), uri.getPort());
    }
}

Read more

• QBit Reactive Microservices
• Reakt Reactive Java
• Reakt Guava Bridge
• QBit Extensions
• Reactive Microservices

• QBit tutorials
• Microservices Intro
• Microservice KPI Monitoring
• Microservice Batteries Included
• RESTful APIs
• Rsourceful REST
• Microservices Reactor
• Working with JSON maps and lists

Still not enough, then try:

Further Reading

QBit Website What is Microservices Architecture?

QBit Java Micorservices lib tutorials

The Java microservice lib. QBit is a reactive programming lib for building microservices - JSON, HTTP, WebSocket, and REST. QBit uses reactive programming to build elastic REST, and WebSockets based cloud friendly, web services. SOA evolved for mobile and cloud. ServiceDiscovery, Health, reactive StatService, events, Java idiomatic reactive programming for Microservices.

Find more tutorial on QBit.

Reactive Programming, Java Microservices, Rick Hightower

High-speed microservices consulting firm and authors of QBit with lots of experience with Vertx - Mammatus Technology

Highly recommended consulting and training firm who specializes in microservices architecture and mobile development that are already very familiar with QBit and Vertx as well as iOS and Android - About Objects

Java Microservices Architecture

Microservice Service Discovery with Consul

Microservices Service Discovery Tutorial with Consul

Reactive Microservices

High Speed Microservices

Java Microservices Consulting

Microservices Training

Reactive Microservices Tutorial, using the Reactor

QBit is mentioned in the Restlet blog

•