Step 8 Setup ELK and use it from Java

If we are going to have a distributed application, we need a way to see the logs. Searchable logging is essential to using Docker and Mesos effectively, and is also essential for microserivce architecture. It is the monitoring that makes distributed computing not suck as much.

Let's setup ELK. ELK is logstash, Kibana, and ElasticSearch.

Following the instructions for the ELK setup, you need to monitor the log files of the Mesos master server (or servers for your production environment).

Follow those instructions to setup an ELK server.

Once you follow the ELK install instructions then add this for our Java microservices.

Config for logback to send messages.

cat /etc/logstash/conf.d/02-tcp-input.conf 
input {
    tcp {
        port => 4560
        codec => json_lines
    }
}

Then restart logstash.

sudo service logstash restart

Java setup.

Now we will use the LogStashEncoder for Logback using these instructions.

In the Java world, we need to add the jar file that plugins into Logback and sends the log files to LogStash.

build.gradle

dependencies {
    compile "io.vertx:vertx-core:3.2.0"
    compile 'ch.qos.logback:logback-core:1.1.3'
    compile 'ch.qos.logback:logback-classic:1.1.3'
    compile 'net.logstash.logback:logstash-logback-encoder:4.6' // <------- NEW
    compile 'org.slf4j:slf4j-api:1.7.12'
    testCompile group: 'junit', name: 'junit', version: '4.11'
}

Note we added net.logstash.logback:logstash-logback-encoder:4.6 to the dependencies.

Next we configure the logback.xml file to the mix.

logback.xml

<configuration>

    <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>10.0.0.62:4560</destination>

        <!-- encoder is required -->
        <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
        <keepAliveDuration>5 minutes</keepAliveDuration>
    </appender>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <root level="DEBUG">
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="stash" />
    </root>
</configuration>

Notice we added the stash appender to use the LogstashEncoder. Now use logging as you normally would.

Using logging

package com.github.vertx.node.example;

import io.vertx.core.AsyncResult;
import io.vertx.core.Future;
import io.vertx.core.Handler;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;


public class HelloWorldServiceImpl implements HelloWorldServiceInterface {
    private final Logger logger = LoggerFactory.getLogger(HelloWorldServiceImpl.class);


    @Override
    public void hello(final String message, final Handler<AsyncResult<String>> resultHandler) {


        logger.info("HelloWorldServiceImpl hello was called {}", message);
        resultHandler.handle(Future.succeededFuture("Hello World! " + message));

    }

    @Override
    public void close() {

    }
}

The source code for this step is in this branch.

Common commands I use to redeploy the example to mesos

I edit the version of the docker deploy after I make changes in the build.gradle file.

Build microserivce/docker image on my dev box and soon Jenkins

./gradlew clean shadowJar buildDocker
docker push advantageous/vertx-node-eventbus-example

After this, you will see the image on docker hub.

Deploy from any member of Amazon VPC

Edit deploy file and add version that I want to deploy

sudo nano VertxNodeDocker.json

Deploy file to deploy new version of docker image to Mesos

{
  "id": "vertx-node-eventbus-example",
  "instances": 1,
  "cpus": 0.5,
  "mem": 512,
  "uris": [],
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "advantageous/vertx-node-eventbus-example:1.0.1.3",
     "network": "BRIDGE",
     "portMappings": [
        { "containerPort": 8080, "hostPort": 0,
           "servicePort": 9000, "protocol": "tcp" }
      ]
    }
  }
}

I use curl to remove the old copy of the app install for this microservice from mesos.

Remove old docker container from mesos

curl -X DELETE http://10.0.0.148:8080/v2/apps/vertx-node-eventbus-example

Then I redeploy the service to mesos.

Redeploy docker image

curl -X POST -H "Content-Type: application/json" http://10.0.0.148:8080/v2/apps -d@VertxNodeDocker.json | jq .

Then I test the microservice but first I need to know where it was deployed and what port it is using.

Using mesos service discovery

curl http://10.0.0.148:8080/v2/tasks/ | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   439    0   439    0     0  92811      0 --:--:-- --:--:-- --:--:--  107k
{
  "tasks": [
    {
      "servicePorts": [
        9000
      ],
      "appId": "/vertx-node-eventbus-example",
      "id": "vertx-node-eventbus-example.cf1d43a1-c6e5-11e5-bee9-02424e94d413",
      "host": "ip-10-0-0-188.us-west-2.compute.internal",
      "ipAddresses": [
        {
          "protocol": "IPv4",
          "ipAddress": "172.17.0.2"
        }
      ],
      "ports": [
        31516
      ],
      "startedAt": "2016-01-30T00:10:10.010Z",
      "stagedAt": "2016-01-30T00:10:02.932Z",
      "version": "2016-01-30T00:10:02.901Z",
      "slaveId": "71ad679f-18ef-4941-a912-69c6cd5adbd4-S1"
    }
  ]
}

With the above host and port, we can now test our deploy.

Testing our deploy

 curl http://10.0.0.188:31516/hello/
HELLO WORLD FROM KOTLIN

Related projects:

• QBit Java Microservices Lib
• Vertx Toolkit for building reactive applications
• Node
• Java Microservices - Mammatus Technology
• Developing Ops-Friendly Microservices Training Video