Rick

Rick
Rick

Saturday, February 27, 2016

Step 8 Setup ELK and use it from Java

If we are going to have a distributed application, we need a way to see the logs. Searchable logging is essential to using Docker and Mesos effectively, and is also essential for microserivce architecture. It is the monitoring that makes distributed computing not suck as much.
Let's setup ELK. ELK is logstash, Kibana, and ElasticSearch.
Following the instructions for the ELK setup, you need to monitor the log files of the Mesos master server (or servers for your production environment).
Follow those instructions to setup an ELK server.
Once you follow the ELK install instructions then add this for our Java microservices.

Config for logback to send messages.

cat /etc/logstash/conf.d/02-tcp-input.conf 
input {
    tcp {
        port => 4560
        codec => json_lines
    }
}
Then restart logstash.
sudo service logstash restart

Java setup.

Now we will use the LogStashEncoder for Logback using these instructions.
In the Java world, we need to add the jar file that plugins into Logback and sends the log files to LogStash.

build.gradle


dependencies {
    compile "io.vertx:vertx-core:3.2.0"
    compile 'ch.qos.logback:logback-core:1.1.3'
    compile 'ch.qos.logback:logback-classic:1.1.3'
    compile 'net.logstash.logback:logstash-logback-encoder:4.6' // <------- NEW
    compile 'org.slf4j:slf4j-api:1.7.12'
    testCompile group: 'junit', name: 'junit', version: '4.11'
}
Note we added net.logstash.logback:logstash-logback-encoder:4.6 to the dependencies.
Next we configure the logback.xml file to the mix.

logback.xml

<configuration>

    <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>10.0.0.62:4560</destination>

        <!-- encoder is required -->
        <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
        <keepAliveDuration>5 minutes</keepAliveDuration>
    </appender>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <root level="DEBUG">
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="stash" />
    </root>
</configuration>
Notice we added the stash appender to use the LogstashEncoder. Now use logging as you normally would.

Using logging

package com.github.vertx.node.example;

import io.vertx.core.AsyncResult;
import io.vertx.core.Future;
import io.vertx.core.Handler;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;


public class HelloWorldServiceImpl implements HelloWorldServiceInterface {
    private final Logger logger = LoggerFactory.getLogger(HelloWorldServiceImpl.class);


    @Override
    public void hello(final String message, final Handler<AsyncResult<String>> resultHandler) {


        logger.info("HelloWorldServiceImpl hello was called {}", message);
        resultHandler.handle(Future.succeededFuture("Hello World! " + message));

    }

    @Override
    public void close() {

    }
}
The source code for this step is in this branch.

Common commands I use to redeploy the example to mesos

I edit the version of the docker deploy after I make changes in the build.gradle file.

Build microserivce/docker image on my dev box and soon Jenkins

./gradlew clean shadowJar buildDocker
docker push advantageous/vertx-node-eventbus-example
After this, you will see the image on docker hub.

Deploy from any member of Amazon VPC

Edit deploy file and add version that I want to deploy

sudo nano VertxNodeDocker.json

Deploy file to deploy new version of docker image to Mesos

{
  "id": "vertx-node-eventbus-example",
  "instances": 1,
  "cpus": 0.5,
  "mem": 512,
  "uris": [],
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "advantageous/vertx-node-eventbus-example:1.0.1.3",
     "network": "BRIDGE",
     "portMappings": [
        { "containerPort": 8080, "hostPort": 0,
           "servicePort": 9000, "protocol": "tcp" }
      ]
    }
  }
}
I use curl to remove the old copy of the app install for this microservice from mesos.

Remove old docker container from mesos

curl -X DELETE http://10.0.0.148:8080/v2/apps/vertx-node-eventbus-example
Then I redeploy the service to mesos.

Redeploy docker image

curl -X POST -H "Content-Type: application/json" http://10.0.0.148:8080/v2/apps -d@VertxNodeDocker.json | jq .
Then I test the microservice but first I need to know where it was deployed and what port it is using.

Using mesos service discovery

curl http://10.0.0.148:8080/v2/tasks/ | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   439    0   439    0     0  92811      0 --:--:-- --:--:-- --:--:--  107k
{
  "tasks": [
    {
      "servicePorts": [
        9000
      ],
      "appId": "/vertx-node-eventbus-example",
      "id": "vertx-node-eventbus-example.cf1d43a1-c6e5-11e5-bee9-02424e94d413",
      "host": "ip-10-0-0-188.us-west-2.compute.internal",
      "ipAddresses": [
        {
          "protocol": "IPv4",
          "ipAddress": "172.17.0.2"
        }
      ],
      "ports": [
        31516
      ],
      "startedAt": "2016-01-30T00:10:10.010Z",
      "stagedAt": "2016-01-30T00:10:02.932Z",
      "version": "2016-01-30T00:10:02.901Z",
      "slaveId": "71ad679f-18ef-4941-a912-69c6cd5adbd4-S1"
    }
  ]
}
With the above host and port, we can now test our deploy.

Testing our deploy

 curl http://10.0.0.188:31516/hello/
HELLO WORLD FROM KOTLIN

Step 7 Adding docker support to gradle and deploying our image to Mesos Marathon

This step was done after adding zookeeper clustering, but it depends on version of the branch that predates the zookeeper cluster manager. We do this because the zookeeper support is not in the maven public repo yet, and we want the master branch build to work.
The code for this is in this branch.

Adding Docker Support Begin with the end in mind.

We want to be able to launch this service via Docker as follows:

Build and install docker instance

 $ gradle clean shadowJar buildDocker

Run docker instance

 $ docker run advantageous/vertx:1.0

Test the instance with curl

$ echo $DOCKER_HOST
tcp://192.168.99.100:2376

$ curl http://192.168.99.100:5000/hello-world/?msg="rick"
Hello World! rick

Setup docker on your dev box (assuming OSX, adjust accordingly)

There is an installation guide to using docker on OSX. You will need to install the Docker Toolbox, and run through some of the tutorial (even if you used boot2docker before as a lot has changed). Once you feel comfortable with building and deploying images for Docker come back here.

Add docker support to gradle

We will need to add the gradle docker plugin.
buildscript {
    repositories {
        mavenCentral()
        jcenter() //ADDED THIS
    }
    dependencies {
        classpath 'org.jetbrains.kotlin:kotlin-gradle-plugin:1.0.0-beta-4584'
        classpath 'se.transmode.gradle:gradle-docker:1.2' //ADDED THIS
    }
}

Use the docker plugin

Apply plugin

apply plugin: 'docker'

Use plugin

group 'advantageous' //changed this
version '1.0' //and this

docker {
    baseImage "vertx/vertx3-exec"
    maintainer 'Rick Hightower "richardhightowerATgmailDOTcom"'



}

Create build task to build and install docker

task buildDocker(type: Docker) {

        tagVersion = System.getenv("BUILD_NUMBER") ?: project.version
        push = Boolean.getBoolean("docker.push")
        applicationName = project.applicationName

        addFile {
            from "${project.shadowJar.outputs.files.singleFile}"
            into "/opt/hello/"
        }
        exposePort 8080
        entryPoint = ["sh",  "-c"]
        defaultCommand = ["java -jar /opt/hello/${project.name}-${project.version}-fat.jar"]

}
The above will both create the Dockerfile and install docker in your local docker instance. It assumes docker is setup correctly on your build box.

Contents of generated Dockerfile

cat ./build/docker/Dockerfile 
FROM vertx/vertx3-exec
MAINTAINER Rick Hightower "richardhightowerATgmailDOTcom"
ADD add_1.tar /
EXPOSE 8080
ENTRYPOINT ["sh", "-c"]
CMD ["java -jar /opt/hello/vertx-1.0-fat.jar"]

Deploying to Mesos / Marathon

We setup a Mesos master and two Mesos Slaves with Marathon and Zookeeper all up in our private EC2 VPC using this guide. We also setup an DockerHub public organization calledadvantageous.
This should work:

Forcing push to DockerHub

gradle -Ddocker.push=true clean shadowJar buildDocker
But instead it takes FOREVER.

Forcing push to DockerHub

gradle  clean shadowJar buildDocker
docker push advantageous/vertx-node-eventbus-example
Note I changed the name of the Docker container from vertx to vertx-node-eventbus-example.

Deploying to Mesos / Marathon Create deploy file

 cat VertxNodeDocker.json 
{
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "advantageous/vertx-node-eventbus-example:1.0"
    }
  },
  "id": "myimage",
  "instances": 1,
  "cpus": 0.5,
  "mem": 512,
  "uris": [],
  "cmd": "while sleep 10; do date -u +%T; done"
}

Use curl to deploy Docker image you stored in DockerHub to Marathon/Mesos

curl -X POST -H "Content-Type: application/json" http://10.0.0.148:8080/v2/apps -d@VertxNodeDocker.json
For this we setup three Mesos instances (1 master and two slaves) in EC2 in a VPC (isolated cloud resources).
The two slaves (called Agents in Mesos 1.0) point to the one master (for QA, prod you would want at least three master mesos instances)

Slave boxes pointing to master mesos instances.

$ cat /etc/mesos/zk
zk://10.0.0.148:2181/mesos
There are plenty of guides to show how to install Meso and Docker on Linux. Follow those guides and setup your own Mesos master.

Debugging

Logs for Mesos are found in /var/log/mesos and /var/log/zookeeper.
To see a list of deployments in mesos.

List of deployments in Mesos

$ curl http://10.0.0.148:8080/v2/deployments/ | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   317    0   317    0     0   115k      0 --:--:-- --:--:-- --:--:--  154k
[
  {
    "totalSteps": 2,
    "currentStep": 2,
    "currentActions": [
      {
        "app": "/myimage",
        "action": "ScaleApplication"
      }
    ],
    "steps": [
      [
        {
          "app": "/myimage",
          "action": "StartApplication"
        }
      ],
      [
        {
          "app": "/myimage",
          "action": "ScaleApplication"
        }
      ]
    ],
    "affectedApps": [
      "/myimage"
    ],
    "version": "2016-01-23T00:12:39.099Z",
    "id": "176ae76a-3f3a-4c6f-a9b6-b539214a4ee9"
  }
]
To delete an image, you hit the mesos REST endpoint as follows:

Delete a deployment

$ curl -X DELETE http://10.0.0.148:8080/v2/deployments/176ae76a-3f3a-4c6f-a9b6-b539214a4ee9 | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    92    0    92    0     0   6028      0 --:--:-- --:--:-- --:--:--  6133
{
  "deploymentId": "dd370d17-3b9e-4682-a107-3b3bd018728c",
  "version": "2016-01-25T19:45:16.584Z"
}

Not working

At this point, you deployed the vertx application but it is not able to run because port 8080 is already taken by Marathon. In order to get the vertx application to work, we will need to map its port 8080 with an outside port for the Marathon agent/slave box.
We can do this by following mesos marathon docker guide and crafting a deploy script.
      "portMappings": [
        { "containerPort": 8080, "hostPort": 0, 
           "servicePort": 9000, "protocol": "tcp" }
      ]
This goes into the docker area of the deploy VertxNodeDocker.json that we have been using.
{
  "container": {
    "type": "DOCKER",
    "docker": {
     "image": "advantageous/vertx-node-eventbus-example:1.0",
     "network": "BRIDGE",
     "portMappings": [
        { "containerPort": 8080, "hostPort": 0, 
           "servicePort": 9000, "protocol": "tcp" }
      ]
    }
  },
  "id": "vertx-node-eventbus-example",
  "instances": 1,
  "cpus": 0.5,
  "mem": 512,
  "uris": []
}
Undeploy the older version. Then deploy this image.
You can ssh to slave boxes and run docker ps to see where it ended up.

Hooks up fine

ubuntu@ip-10-0-0-188:~$ sudo docker ps
CONTAINER ID        IMAGE                                          COMMAND                  CREATED             STATUS              PORTS                     NAMES
04e5b69a3bf9        advantageous/vertx-node-eventbus-example:1.0   "sh -c 'java -jar /op"   7 minutes ago       Up 7 minutes        0.0.0.0:31627->8080/tcp   mesos-00481535-158b-4812-aff3-bf02d4176ef8-S0.84d6bd59-c5cc-4c56-b2d6-e30af1ea11aa

ubuntu@ip-10-0-0-188:~$ curl http://10.0.0.188:31627/hello/
HELLO WORLD FROM KOTLIN
We looked up the ports and then used that information to make a call into the service.
To actually get this information from Marathon, we do this.

Getting service discovery information from Marathon

$ curl http://10.0.0.148:8080/v2/tasks/ | jq .
{
  "tasks": [
    {
      "servicePorts": [
        9000
      ],
      "appId": "/vertx-node-eventbus-example",
      "id": "vertx-node-eventbus-example.21e8f58e-c3aa-11e5-aace-02422bec481e",
      "host": "ip-10-0-0-188.us-west-2.compute.internal",
      "ipAddresses": [
        {
          "protocol": "IPv4",
          "ipAddress": "172.17.0.2"
        }
      ],
      "ports": [
        31627
      ],
      "startedAt": "2016-01-25T21:25:19.459Z",
      "stagedAt": "2016-01-25T21:25:18.576Z",
      "version": "2016-01-25T20:47:56.140Z",
      "slaveId": "00481535-158b-4812-aff3-bf02d4176ef8-S0"
    }
  ]
}
Note that the port in ports is the correct docker port for our HelloWorld sample. We can curl using this host and port.

Working with host

$ curl http://ip-10-0-0-188.us-west-2.compute.internal:31627/hello/
HELLO WORLD FROM KOTLIN

Common commands I use to redeploy the example to mesos

I edit the version of the docker deploy after I make changes in the build.gradle file.

Build microserivce/docker image on my dev box and soon Jenkins

./gradlew clean shadowJar buildDocker
docker push advantageous/vertx-node-eventbus-example
After this, you will see the image on docker hub.

Deploy from any member of Amazon VPC

Edit deploy file and add version that I want to deploy

sudo nano VertxNodeDocker.json

Deploy file to deploy new version of docker image to Mesos

{
  "id": "vertx-node-eventbus-example",
  "instances": 1,
  "cpus": 0.5,
  "mem": 512,
  "uris": [],
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "advantageous/vertx-node-eventbus-example:1.0.1.3",
     "network": "BRIDGE",
     "portMappings": [
        { "containerPort": 8080, "hostPort": 0,
           "servicePort": 9000, "protocol": "tcp" }
      ]
    }
  }
}
I use curl to remove the old copy of the app install for this microservice from mesos.

Remove old docker container from mesos

curl -X DELETE http://10.0.0.148:8080/v2/apps/vertx-node-eventbus-example
Then I redeploy the service to mesos.

Redeploy docker image

curl -X POST -H "Content-Type: application/json" http://10.0.0.148:8080/v2/apps -d@VertxNodeDocker.json | jq .
Then I test the microservice but first I need to know where it was deployed and what port it is using.

Using mesos service discovery

curl http://10.0.0.148:8080/v2/tasks/ | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   439    0   439    0     0  92811      0 --:--:-- --:--:-- --:--:--  107k
{
  "tasks": [
    {
      "servicePorts": [
        9000
      ],
      "appId": "/vertx-node-eventbus-example",
      "id": "vertx-node-eventbus-example.cf1d43a1-c6e5-11e5-bee9-02424e94d413",
      "host": "ip-10-0-0-188.us-west-2.compute.internal",
      "ipAddresses": [
        {
          "protocol": "IPv4",
          "ipAddress": "172.17.0.2"
        }
      ],
      "ports": [
        31516
      ],
      "startedAt": "2016-01-30T00:10:10.010Z",
      "stagedAt": "2016-01-30T00:10:02.932Z",
      "version": "2016-01-30T00:10:02.901Z",
      "slaveId": "71ad679f-18ef-4941-a912-69c6cd5adbd4-S1"
    }
  ]
}
With the above host and port, we can now test our deploy.

Testing our deploy

 curl http://10.0.0.188:31516/hello/
HELLO WORLD FROM KOTLIN
Kafka and Cassandra support, training for AWS EC2 Cassandra 3.0 Training