Saturday, August 29, 2015

Getting started with QBit Microservices Lib Batteries-Included Part 2

If you are new to QBit. It might make more sense to skim the overview. We suggest reading the landing page of the QBit Microservices Lib's wiki for a background on QBit. This will let you see the forrest while the tutorials are inspecting the trees. There is also a lot of documents linked to off of the wiki landing page as well as in the footer section of the tutorials.

Getting started with QBit Microservices Lib Batteries Included Part 2

Mammatus Tech
This is part two in this tutorial series. Part 1
QBit is small and wicked fast, but it comes batteries included.
QBit comes with Service DiscoveryHealthMonitoringRealtime service stats,AsyncReactive Async call managementJob ControlEvent Bus, built-in. QBit is very similar to the Typed Actor model but streamlined for Microservices and using idiomatic Java constructs.
Already, even the simple Hello World example has runtime stats support, health monitoring that can be integrated with service discovery, and more.
To expose end points to some of these services, we merely have to create an admin end point as follows:

Turning on the admin end point

    public static void main(final String... args) {
        final ManagedServiceBuilder managedServiceBuilder =

        /* Start the service. */
        managedServiceBuilder.addEndpointService(new HelloWorldService())

        /* Start the admin builder which exposes health end-points and meta data. */

        System.out.println("Servers started");

We just turn on the admin.

Turning on Admin Support

        /* Start the admin builder which exposes health end-points and meta data. */


By default when you use the ManagedServiceBuilder you get a health check on the same port as your main web service port. Microserivces Health and Stats are very important in a Microservices Architecture. This is part of the batteries included approach of QBit Microservices lib. If it is important in a Microservices Architecture, it is supported by QBit Java Microservices Lib.

To see health for this service:

$ curl http://localhost:8080/__health
Your web service port is settable by call setPort on the ManagedServiceBuilder or by passing in the environment variable PORT or WEB_PORT.
The health check is not a simple endpoint that returns "ok" with a status 200. It in fact it will ask check to see if all ServiceQueue services (internal and exposed) are still running. If a ServiceQueue or ServiceServerEndpoint does not check-in with the health system the health end point will return a 500 code with the message "fail". Services can also mark themselves as unhealthy, and mark themselves as healthy again if they can recover.
The health check is on by default endpoint, but it can be disabled(managedServiceBuilder.setEnableLocalHealth(false)).
ManagedServiceBuilder is setup so that your microservice just runs as expected in EC2, Heroku or Docker. ManagedServiceBuilder makes it easy to hook up your service to Heroku health checks, Load Balancer health checks, NagiosConsul, etc.
Side note: We strongly recommend you implement Consul for Microservices Service Discovery and Health. If you are not familiar with Consul, we wrote this tutorial Consul for Microservices Architecture Service Discovery and Health For Tutorial.
This health end point is on by default with or without the admin. It can be disabled.
Once you turn on the admin port, you can also see stats on the admin port as well as health. (You could for example disable the __health endpoint on the main port and still have access to health via the admin port). We will have a full tutorial on Health support. It is beyond on the scope of this batteries included microservice tutorial.

See health over admin back-end port.

$ curl http://localhost:7777/__admin/ok
The above returns "true" if all registered health systems nodes are healthy.
A node is a Service (SerivceQueue, service actor), ServiceBundle (a group of services), queue, or ServiceServerEndpoint (a ServiceBundle that is exposed via REST and WebSocket) that is being monitored. Later we will show you how to list all service nodes or endpoints. This example only has one.
Before we show the stats, let's hit the service a few times with wrk.

Hitting the service a few times so we can show stats

$ wrk -t 2 -d 2s -c 1000  http://localhost:8888/root/hello/hello
Running 2s test @ http://localhost:8888/root/hello/hello
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.18ms    7.78ms 308.89ms   99.66%
    Req/Sec    41.64k     7.89k   55.16k    57.50%
  165635 requests in 2.02s, 14.53MB read
  Socket errors: connect 0, read 250, write 0, timeout 0
Requests/sec:  82109.35
Transfer/sec:      7.20MB
Recall that wrk is a load testing tool.
Now let's ask the admin to show some stats.

Show service stats and metrics

$ curl http://localhost:8080/__stats/instance 

Output showing microservice stats and metrics

    "MetricsKV": {
        "apptitle.hostname.jvm.thread.peak.count": 33,
        "apptitle.hostname..jvm.mem.heap.free": 232961928,
        "apptitle.hostname.jvm.thread.count": 33,
        "apptitle.hostname.jvm.mem.non.heap.max": -1,
        "apptitle.hostname.jvm.os.load.level": 3,
        "apptitle.hostname.jvm.mem.heap.max": 3817865216,
        "HelloWorldService": 1,
        "apptitle.hostname.jvm.mem.heap.used": 24463480,
        "apptitle.hostname.jvm.mem.non.heap.used": 18423344,
        "apptitle.hostname.jvm.thread.daemon.count": 12,
        "apptitle.hostname.jvm.mem.total": 257425408,
        "apptitle.hostname.jvm.thread.started.count": 35
    "MetricsMS": {
        "HelloWorldService.callTimeSample": [
    "MetricsC": {
        "HelloWorldService.startBatchCount": 17900,
        "HelloWorldService.receiveCount": 447169
    "version": 1
Local stats collection is on by default if you use the ManagedServiceBuilder. It can be disabled (managedServiceBuilder.setEnableLocalStats(false)).
This again is so you can see critical stats about your service before you set things up in Grafana, Graphite using StatsD which we will cover later.
The Stats system can be passive (StatsDGrafanaGraphite). Also the QBit Stats system can be clustered and shared and query-able so that the Stats become realtime analyticsthat can be reacted upon (reactive programming). We will cover those later in this tutorial series, but it is beyond the scope of this tutorial.
Side Note: We have used the Stats system in production to provide application key rate limiting with OAuth headers. We have also written custom plugins (Stats System is pluggable) to provide stats in prexisting stats systems that clients were already using. Every core service in QBit has an interface that can be replaced with your own implementation.
Notice: The stats are not just for the JVM but for every service actor (ServiceQueue) running in the system.
QBit Microservice Lib takes KPIs, runtime stats, and health of microservices very seriously (Microservices Monitoring thoughts behind QBit approach). If you take microservices seriously, then you need a library that supports microservices monitoring and KPI as core to its internals.
We will cover Stats more when we cover setting up StatsD/Graphite. This microservice tutorial series will also show how to setup the application names, and such (the keys, names of the stats).
This endpoint is nice if you want to implement a pull model for stats collection (versus a push model like StatsD) so that all services can publish stats, and background jobs can aggregate them and push them into a time series database using the REST/microservice friendly JSON interface. (We prefer StatsD, but depending on the number of nodes you are running that might be more difficult.)

Microservice API Gateway

QBit Microservices Lib creates on-the-fly proxies that can do high-speed WebSocket calls. QBit's WebSocket support which uses JSON and ASCII is often faster than many competitors solutions which use binary protocols and regular sockets. This is one way that QBit provides support for Microservice API Gateways. (We are planning similar support for on-the-fly REST interfaces).
In addition to the WebSocket support for Microservice API Gateways, QBit provides access via REST and Swagger. QBit tracks a rich set of meta-data about the microserivce endpoints which it can expose via Swagger support. Once the API is exposed as swagger it is easy to generate Python, Ruby, Scala, Objective-C clients. This is the very definition of a Microservice API gateway.
Swagger, leading provider of API Gateway services, provides a RESTful API to meta-data about your Microservice API gateways. Swagger has the largest ecosystem of API tooling (a lot of it open source), and is supported by thousands of developers. It has a meta-data spec. With a Swagger-enabled API, QBit gets interactive documentation, as well as client SDK generation, API Gateway documentation and additional discoverability. We will cover this more when we cover API gateways.

To see swagger file for this service:

$ curl http://localhost:7777/__admin/meta/

Output Swagger

    "swagger": "2.0",
    "info": {
        "title": "application title goes here",
        "description": "Description not set",
        "contact": {
            "name": "ContactName not set",
            "url": "Contact URL not set",
            "email": "no.contact.email@set.me.please.com"
        "version": "0.1-NOT-SET",
        "license": {
            "name": "licenseName not set",
            "url": "http://www.license.url.com/not/set/"
    "host": "localhost:8888",
    "basePath": "/root",
    "schemes": [
    "consumes": [
    "produces": [
    "paths": {
        "/hello/hello": {
            "get": {
                "operationId": "hello",
                "summary": "no summary",
                "description": "no description",
                "produces": [
                "responses": {
                    "200": {
                        "description": "no return description",
                        "schema": {
                            "type": "string"
Most QBit REST features can be exposed via Swagger. We support the full array of Swagger features so you can develop in your polyglot Microservice environment.

YAML version of Microservice Hello World interface in Swagger

swagger: '2.0'
  title: application title goes here
  description: Description not set
    name: ContactName not set
    url: Contact URL not set
    email: no.contact.email@set.me.please.com
  version: 0.1-NOT-SET
    name: licenseName not set
    url: 'http://www.license.url.com/not/set/'
host: 'localhost:8888'
basePath: /root
  - http
  - https
  - wss
  - ws
  - application/json
  - application/json
      operationId: hello
      summary: no summary
      description: no description
        - application/json
          description: no return description
            type: string
You can import the JSON file into a Swagger Editor and generate all sorts of clients.
Later we will show more about this and how to set the description, summary and return descriptions as well as the other data and documents about the microservice that we can expose via Swagger.
We also have some Swagger generated examples for our TODO microservice exampleand our RESTful Microserivce Resource Example.

More about the admin

Show all nodes

$ curl http://localhost:7777/__admin/all-nodes/

All of the services running in this JVM

There is only one at this point.
We can show all of the running nodes (service actors, REST endpoints, service bundles) from this admin endpoint.
If you want to see just the healthy nodes.

List healthy nodes by name:

$ curl http://localhost:7777/__admin/healthy-nodes/
Keep in mind that the QBit health system integrates with the QBit ServiceDiscoveryservice so if a service becomes unhealthy it can be unregistered in theServiceDiscovery system. You can see this in action with QBit's clustered event bus which uses the ServiceDiscovery and its Consul implementation to provide a clustered event bus which removes nodes and discovers nodes via Consul. Nodes can be removed if they become unhealthy. It works and has been used in production for quite some time.
You can also get the complete information about the health of nodes.

List complete node information:

$ curl http://localhost:7777/__admin/load-nodes/

Complete information about Health of all services

        "name": "HelloWorldService",
        "ttlInMS": 10000,
        "lastCheckIn": 1440877933151,
        "status": "PASS"


The QBit Microservice lib comes with batteries-included microservice development support from real-time queryable stats to do real-time analytics to health monitoring to polyglot API gateways and service discovery. QBit supports a true Microservice Architecture.
Kafka and Cassandra support, training for AWS EC2 Cassandra 3.0 Training