Rick

Rick
Rick

Tuesday, May 12, 2015

Vertx 3 looks amazing, nice reactive microservices framework/lib

It looks like Vertx 3 is a very significant release. It fills many gaps and fixes direction with Vertx 2 (which was already very compelling). Vertx 3 seems even more targeted at the reactive microservice space. It adds support for pluggable messaging, it has more than one cluster manager, it has async MySQL support, as well as async support for Redis, and PostgreSQL support, and MongoDB. (We are not using any of those but at least there are patterns in play for Cassandra). 

A web style framework to make dealing with Vertx core HTTP a little smoother, etc.

Anyway.. I used Vertx 2 quite a bit, and am very interested in Vertx 3. 

QBit is mostly based on Vertx2. Many ideas for QBit came straight out of working with Vertx2. I am a big fan.

I plan on porting QBit to Vertx3 very soon.

Also most of the examples are using Vertx2 more like a lib than using Vertx as described in the docs. I want to start to change that and have more examples of working with QBit inside of Vertx.

I really like direction of having a pluggable cluster manager in vertx3. It was pluggable before but only one plugin. I can see someone plugging in ZooKeeper, or Consul or what have you. I tend to use Consul personally because service clients are often written in non-JVM environments (that I work with anyway) and Consul has clients for most non-JVM clients. 

Apex looks like a nice way to organize the needed gaps between traditional web frameworks and something like Vertx. I like that it is separate (at least in the docs), and well documented. I could have really used that level of support with my first Vertx project (I think some of the feature were there but certainly not so clear cut).

The integration with Dropwizard Metrics looks spot on (Microservices Metrics, Stats and monitoring are essential for microservices). I hope to see some future integration with StatsD/Graphite. Not sure how easy/hard that would be. 

Messaging and Integration supports seems really nice. QBit does a lot of messaging. Early versions (non released) of QBit could plug into the Vertx pub/sub or use its own for same JVM. QBIt then just used it own (with the goal of one day working with Vertx again) and a remote one based on Vertx Websocket support. I was planning on porting QBit to work over Kafka. Now I am more likely to make it work with Vertx again, and then write a Vertx messaging plugin for Kafka.

Currently, I am using Consul (it is in theory pluggable) in QBit to find peer nodes and broadcast events to other peers using WebSocket support via Vertx 2. When I port QBit to Vertx 3, I will try to use the Vertx clustering support to find peers, etc. Not sure if that exists, but it would be something that I would be interested in working on. This way Vertx 3 handles the plumbing. 


QBit has this type of support that works with Vertx 2.

@RequestMapping("/todo-service")
public class TodoService {

    @RequestMapping("/todo/count")
    public int size() {...

    @RequestMapping("/todo/")
    public List<TodoItem> list() {...



 Then it would add REST support (JSON only) for that object. 

It implements the full REST style URL mapping...

Adder Service using URI params

    @RequestMapping("/adder-service")
    public class AdderService {

        @RequestMapping("/add/{0}/{1}")
        public int add(@PathVariable int a, @PathVariable int b) {...
    }

REST is really about links in the docs and less about the URLs but most Java folks (that I have worked with) are more concerned about the URL style. (Usually not me per se.)

Using a microservice remotely with WebSocket

       /* Start QBit client for WebSocket calls. */
        final Client client = clientBuilder()
                   .setPort(7000).setRequestBatchSize(1).build();


       /* Create a proxy to the service. */
        final AdderServiceClientInterface adderService =
                client.createProxy(AdderServiceClientInterface.class, 
                "adder-service");

        client.start();



       /* Call the service */
        adderService.add(System.out::println, 1, 2);

It works with Vertx2 now. There is also a strongly typed message bus that works remotely (https://github.com/advantageous/qbit/wiki/%5BDetailed-Tutorial%5D-Using-event-channels-and-strongly-typed-event-bus-with-QBit-(The-employee-example)). The strongly typed message bus is using Consul to find remote nodes.


It needs this method from "clustering" / "service discovery" piece.

public interface ServiceDiscovery extends Startable, Stoppable {
...

    default List<ServiceDefinition> loadServices(final String serviceName) {

        return Collections.emptyList();
    }
 

...


public class ServiceDefinition {

    private final HealthStatus healthStatus;
    private final String id;
    private final String name;
    private final String host;
    private final int port;
    private final long timeToLive;


If I had a magic genie... and I could make three Vertx 3 wishes.


1) A service discovery pluggable thing on top of clustering support  (which may or may not already exist) 
3) Integration with StatsD as a metrics stats collection framework (http://rick-hightower.blogspot.com/2015/05/working-with-statsd-and-java.html)


Thursday, May 7, 2015

Working with StatsD and Java

Run statsD daemon with docker

sudo docker run -d \
  --name graphite \
  -p 80:80 \
  -p 2003:2003 \
  -p 8125:8125/udp \
  hopsoft/graphite-statsd
Make sure you upgrade to the latest docker. I had to update boot2docker and update docker to get the above to work.

Update boot2docker

boot2docker update

brew upgrade docker

I installed docker manually, so to get the latest version with brew, I had to do this.
brew install docker
brew link --overwrite docker

Run script (OSX/Windows)

while true
do
   echo -n "example.statsd.counter.changed:$(((RANDOM % 10) + 1))|c" | nc -w 1 -u 192.168.59.103 8125
done
Then you can view the dashboard data at: http://192.168.59.103/dashboard

Note to get the ip address of docker

$ boot2docker ssh
$ ifconfig 
eth1      Link encap:Ethernet  HWaddr 08:00:27:E1:F5:54  
          inet addr:192.168.59.103  Bcast:192.168.59.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fee1:f554/64 Scope:Link

Note to get the ip address of the container running statsd

First find the container id.
$ docker ps

Then use the id to look up the container.
$ docker inspect --format '{{ .NetworkSettings.IPAddress }}' <CONTAINER ID FROM LAST STEP>
$ docker ps
CONTAINER ID        IMAGE                            COMMAND             CREATED             STATUS              PORTS                                                                NAMES
9183f205a3aa        hopsoft/graphite-statsd:latest   "/sbin/my_init"     About an hour ago   Up About an hour    0.0.0.0:80->80/tcp, 0.0.0.0:2003->2003/tcp, 0.0.0.0:8125->8125/udp   graphite   

$ docker inspect --format '{{ .NetworkSettings.IPAddress }}' 9183f205a3aa
172.17.0.1

If everything goes well

Graphite

To start with let's try out the Java client.

The Java client is based on the reference Java client from the etsy project.
public class UsingStatsDIncrement {

    public static void main(String... args) throws Exception {
        StatsdClient client = new StatsdClient("192.168.59.103", 8125);

        while (true) {
            client.increment("foo.bar.baz", 10, .1);
            Thread.sleep(1000);
        }
    }
}
I let this run for a while. Then I go to http://192.168.59.103/dashboard I look under "stats.foo.bar." in the nav tree. You many not really understand what you are seeing but there is a graph that goes from 0 to 20 sort of randomly. It can even go all the way up to 30.
graphite2
Changing the sleep to 100 ms instead of 1000 should yield some different results.
Checking....
It does...
graphite3
Now when we read and re-read the docs, they will more sense.
Now I changed 0.1 to 1.0 and let it run for a while and I get this nice flat line.
graphite4
I added a gauge.
        StatsdClient client = new StatsdClient("192.168.59.103", 8125);

        int guageValue = 10;

        while (true) {
            client.increment("foo.bar.baz", 10, 1.0);
            client.gauge("gauge.foo.bar.baz", guageValue++, 1.0);

            if (guageValue > 100) {
                guageValue = 20;
            }
            Thread.sleep(100);
        }
    }
gauge
Then I added timings and started clicking around.
    public static void main(String... args) throws Exception {
        StatsdClient client = new StatsdClient("192.168.59.103", 8125);

        int gaugeValue = 10;

        while (true) {
            client.increment("foo.bar.baz", 10, 1.0);
            client.gauge("gauge.foo.bar.baz", gaugeValue++, 1.0);
            client.timing("foo.bar.baz.mytiming", gaugeValue, 1.0);

            if (gaugeValue > 100) {
                gaugeValue = 20;
            }
            Thread.sleep(100);
        }
    }
}
many charts

Concepts

A good description of concepts related to the domain model of statsD is documented here:
It is sparse but perhaps complete.
The core concepts for StatsD came from this 2008 blog post (according to Etsy documentation).
Although the early versions of StatsD seemed to use RRDtool and Ganglia.
While Statsd tends to use Graphite, and Whisper.
A good description of the wire protocol in more of a tutorial form can be found here:
StatsD was written to work with Graphite. Graphite is used to visualize the state of microservices. Graphite is made up of Graphite-Web that renders graphs and dashboards, Carbon metric processing daemons, and Whisper which is a time-series database library.
When you send a stat, you send these basic types:
c: This indicates a "count". g: This indicates a gauge. s: a mathematical set. ms: time span.

Count

The counts adds up values that StatsD receives for a metric within the flush interval and sends the total value. StatsD will collect all of the data it receives during its ten second flush interval and add them together to send a single value for that time frame.

Gauge

The gauge tells the current level of something, like memory used or #number of threads being used, etc. With the gauge you send the most recent value. StatsD sends Carbon the same value until it gets a different value.

Set

With sets, you send a bunch of values and StatsD and it will count the number of times it received unique values. Think of a set of enumerators, UP, DOWN, WARNING, CRITICAL, OK. You want to count how many times each occurs.

Time Span

With time spans, you can send StatsD timing values. StatsD the values to Carbon which calculates averages, percentiles, standard deviation, sum, etc.
A good description of using StatsD/Graphite:

Using Docker, Gradle to create Java docker distributions for java microservices draft 4

Using Docker, Gradle to create Java docker distributions for java microservices draft 4

I have used Docker and Vagrant quite a lot to setup series of servers. This is a real lifesaver when you are trying to do some integration tests and run into issues that would be hard to track down without running "actual servers". Running everything on one box is not the same as running many "servers". Even if your final deployment is VMWare or EC2 or bare metal servers, Docker and Vagrant are great for integration testing and writing setup documentation.
I also tend to use gradle a lot these days and grown quite fond of the application and distribution plugins. To me gradle application plugin and docker (or vargrant or EC2 with boto) are sort of essential way to doing Java microservice development.
Before we get into Vagrant or Docker, let's try to do something very simple. Let's use the gradle plugin to create a simple Java application that reads its config from\etc\myapp\conf.properties and \etc\myapp\logging.xml and that we can deploy easily to \opt\myapp\bin (startup scripts) and \opt\myapp\lib (jar files).

Using Gradle and the Gradle Application plugin and Docker

Gradle can create a distribution zip or tar file which is a archive file with the libs and shell scripts you need to run on Linux/Windows/Cygwin/OSX. Or it can just install of this stuff into a directory of your choice.
What I typically do is this….
  • Create a dist tar file using gradle.
  • Create a dockerfile.
The docker file copies the dist tar to the container, untars it and then runs it inside of docker. Once it is a docker file, then you can make a docker container that you can ship around. The gradle and docker file have all of the config info that is common.
You may even have special gradle build options for different environments. Or your app talks to Consul or etcd on startup and look up the special environments stuff like server locations so the docker binary dist can be identical. Consul and etcd are essential ingredients in a microservices architecture both for elastic consistent config and service discovery.
Our binary deliverable is the runnable docker container not a jar file or a zip.
The distZip, and/or distTar is just a way to package up our code and make it easy to shove into our docker container.
If you go the docker route, then the docker container is our binary (runnable) distribution not the tar or zip. We do not have to guess what JVM, because we configure the docker container with exactly the JVM we want to use. We can install any drivers or daemons or utilities that we might need from the Linux world into our container.
Think of it this way. With maven and/or gradle you can create a zip or war file that has the right version of the MySQL jar file. With Docker, you can create a Linux runnable binary that has all of the jar files and not only the right MySQL jar file but the actual right version MySQL server which can be packaged in the same runnable binary (the Linux Docker container).
Gradle application plugin generates a zip or tar file with everything we need and does not require a master Java process, or another repo cache of jars, etc. Between gradle application plugin and docker, we do whatever we need to do with our binary configuration but in a much more precise manner. Every jar, every linux utility, every thing we need, all in one binary that can be deployed in a prviate cloud, public cloud or just run on your laptop. No need to guess the OS, JVM, or libs. We ship exactly what we need.
Docker is used to make deployements faster and more precise.
If part of the tests include running some integration with virtualization than Docker should be the fastest route for creating new virtual instances (since it is just a chgroot like and not a full virtual machine).
I think Docker, gradle and gradle application plugin is your best option for creating fast integration tests. But of course if you have EC2/boto, Vagrant, etc., Docker is not the only option.

Gradle application plugin

Our first goal is to do the following. Use the gradle application plugin to create a simple Java application that reads its config from \etc\myapp\conf.properties and\etc\logging.xml and that we can deploy easily to \opt\myapp\bin (startup scripts) and \opt\myapp\lib (jar files).
Before we get started let's do some prework.
$ sudo mkdir /etc/myapp
$ sudo chown rhightower /etc/myapp
Do the same for /opt/myapp. Where rhightower is your username. :)

The Java app

package com.example;

import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.util.Properties;

public class Main {

    public static void main(String... args) throws IOException {
        final String configLocation = System.getProperty("myapp.config.file");
        final File confFile = configLocation==null ?
            new File("./conf/conf.properties") :
            new File(configLocation);

        final Properties properties = new Properties();

        properties.load(Files.newInputStream(confFile.toPath()));

        System.out.printf("The port is %s\n", properties.getProperty("port"));
    }

}
It is a simple Java app, it looks at a configuration file that has the port. The location of the configuration file is passed via a System.property. If the System.property is null, then it loads the config file from the current working directory.
When you run this program from an IDE, you will get.
The port is 8080
But we want the ability to create an /etc/myapp/conf.properties and an /opt/myappinstall dir. To do this we will use the application plugin.

Creating an install directory with the applicaiton plugin

To create /etc/myapp/conf.properties and an /opt/myapp install dir, we will use the gradle application plugin.

gradle application plugin

apply plugin: 'java'
apply plugin: 'application'

mainClassName = 'com.example.Main'
applicationName = 'myapp'
applicationDefaultJvmArgs = ["-Dmyapp.config.file=/etc/myapp/conf.properties"]

repositories {
    mavenCentral()
}

task copyDist(type: Copy) {
    dependsOn "installApp"
    from "$buildDir/install/myapp"
    into '/opt/myapp'
}

task copyConf(type: Copy) {
    from "conf/conf.properties"
    into "/etc/myapp/"
}


dependencies {
}
Running the copyDist task will also run the installApp which is provided by theapplication plugin which is configured at the top of the file. We can use the copyConffile to copy over a sample configuration file.
Here is our build dir layout.

Build dir layout of the myapp gradle project

.
├── build.gradle
├── conf
│   └── conf.properties
├── settings.gradle
└── src
    └── main
        └── java
            └── com
                └── example
                    └── Main.java

conf/conf.properties

port=8080
To build and deploy the project into /opt/myapp, we do the following:

Building and installing our app

$ gradle build copyDist
This creates this directory structure for the install operation.

Our app install

$ tree /opt/myapp/
/opt/myapp/
├── bin
│   ├── myapp
│   └── myapp.bat
└── lib
    └── gradle-app.jar

To deploy a sample config we do this:

Copy sample config

$ gradle build copyConf
Now edit the config file and change the port from 8080 to 9090.

Edit file and change property

$ nano /etc/myapp/conf.properties 
Now run it.
$ /opt/myapp/bin/myapp
The port is 9090
Change the properties file again. Run the app again.

Next up

Configuring logging under /etc/myapp/logging.properties.

Logging

Sl4j is the standard way to install loggers. Logback is the successor to Log4j. The nice thing about Sl4j is you can use built-in logging, log4j or Logback. For now, we are recommending Logback.
We are going to use Logback. Technically we are going to use sl4j, and we are going to use the logback implementation of it.
Logback allows you to set the location of the log configuration via a System property called logback.configurationFile
#### Example setting logback via System property
java -Dlogback.configurationFile=/path/to/config.xml chapters.configuration.MyApp1
We need to add these dependencies to our gradle file.
  • logback-core-1.1.3.jar
  • logback-classic-1.1.3.jar
  • slf4j-api-1.7.7.jar

Adding dependencies to gradle file

dependencies {
    compile 'ch.qos.logback:logback-core:1.1.3'
    compile 'ch.qos.logback:logback-classic:1.1.3'
    compile 'org.slf4j:slf4j-api:1.7.12'
}
The distribution/install that we generate with gradle needs to pass the location to our application. We do that with the applicationDefaultJvmArgs in the gradle build.

Adding logback.configurationFile System property to launcher script

applicationDefaultJvmArgs = [
        "-Dmyapp.config.file=/etc/myapp/conf.properties",
        "-Dlogback.configurationFile=/etc/myapp/logging.xml"]
Now we can store a logging config in our project so it gets stored in git.

./conf/logging.xml log config

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>conf %d{HH:mm:ss.SSS} [%thread] %-5level %logger{5} - %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/opt/logging/logs</file>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <Pattern>%d{yyyy-MM-dd_HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</Pattern>
        </encoder>

        <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
            <FileNamePattern>/opt/logging/logs%i.log.zip</FileNamePattern>
            <MinIndex>1</MinIndex>
            <MaxIndex>10</MaxIndex>
        </rollingPolicy>

        <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
            <MaxFileSize>2MB</MaxFileSize>
        </triggeringPolicy>
    </appender>

    <logger name="com.example.Main" level="DEBUG" additivity="false">
        <appender-ref ref="STDOUT" />
        <appender-ref ref="FILE" />
    </logger>

    <root level="INFO">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>
Then we can add some tasks in our build script to copy it to the right location.

Scripts to copy logging script into correct location for install

task copyLogConf(type: Copy) {
    from "conf/logging.xml"
    into "/etc/myapp/"
}

task copyAllConf() {
    dependsOn "copyConf", "copyLogConf"
}

To deploy our logging script run
gradle copyAllConf
Now after you install the logging config, you can turn it on or off.
Let's change our main method to use the logging configuration.

Main method that uses logkit to do logging.

package com.example;

import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.util.Properties;


import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class Main {

    static final Logger logger = LoggerFactory.getLogger(Main.class);

    public static void main(final String... args) throws IOException {
        final String configLocation = System.getProperty("myapp.config.file");
        final File confFile = configLocation==null ?
            new File("./conf/conf.properties") :
            new File(configLocation);

        final Properties properties = new Properties();

        properties.load(Files.newInputStream(confFile.toPath()));

        logger.debug(String.format("The port is %s\n", properties.getProperty("port")));
    }

}

Next up after that

Configuring dockerfile

Raw Notes

allprojects {

    group = 'mycompany.router'
    apply plugin: 'idea'
    apply plugin: 'java'
    apply plugin: 'maven'
    apply plugin: 'application'
    version = '0.1-SNAPSHOT'

}


subprojects {


    repositories {
        mavenLocal()
        mavenCentral()
    }

    sourceSets.main.resources.srcDir 'src/main/java'
    sourceCompatibility = JavaVersion.VERSION_1_8
    targetCompatibility = JavaVersion.VERSION_1_8

    dependencies {
        compile "io.fastjson:boon:$boonVersion"

        testCompile "junit:junit:4.11"
        testCompile "org.slf4j:slf4j-simple:[1.7,1.8)"
    }

    task buildDockerfile (type: Dockerfile) {
        dependsOn distTar
        from "java:openjdk-8"
        add "$distTar.archivePath", "/"
        workdir "/$distTar.archivePath.name" - ".$distTar.extension" + "/bin"
        entrypoint "./$project.name"
        if (project.dockerPort) {
            expose project.dockerPort
        }
        if (project.jmxPort) {
            expose project.jmxPort
        }
    }

    task buildDockerImage (type: Exec) {
        dependsOn buildDockerfile
        commandLine "docker", "build", "-t", "mycompany/$project.name:$version", buildDockerfile.dockerDir
    }


    task pushDockerImage (type: Exec) {
        dependsOn buildDockerfile
        commandLine "docker", "push", "mycompany/$project.name"
    }


    task runDockerImage (type: Exec) {
        dependsOn buildDockerImage
        if (project.dockerPort) {
        commandLine "docker", "run", "-i", "-p", "$project.dockerPort:$project.dockerPort", "-t", "mycompany/$project.name:$version"
        } else {
        commandLine "docker", "run", "-i", "-t", "mycompany/$project.name:$version"
        }
    }


    task runDocker (type: Exec) {
        if (project.dockerPort) {
        commandLine "docker", "run", "-i", "-p", "$project.dockerPort:$project.dockerPort", "-t", "mycompany/$project.name:$version"
        } else {
        commandLine "docker", "run", "-i", "-t", "mycompany/$project.name:$version"
        }
    }

}


project(':sample-web-server') {

    mainClassName = "mycompany.sample.web.WebServerApplication"

    applicationDefaultJvmArgs = ["-Dcom.sun.management.jmxremote", "-Dcom.sun.management.jmxremote.port=${jmxPort}",
                                 "-Dcom.sun.management.jmxremote.authenticate=false",  "-Dcom.sun.management.jmxremote.ssl=false"]

    dependencies {
        compile "io.fastjson:boon:$boonVersion"

        compile group: 'io.advantageous.qbit', name: 'qbit-boon', version: '0.5.2-SNAPSHOT'
        compile group: 'io.advantageous.qbit', name: 'qbit-vertx', version: '0.5.2-SNAPSHOT'

        testCompile "junit:junit:4.11"
        testCompile "org.slf4j:slf4j-simple:[1.7,1.8)"
    }

    buildDockerfile {
        add "$project.buildDir/resources/main/conf/sample-web-server-config.json", "/etc/sample-web-server/conf.json"
        add "$project.buildDir/resources/main/conf/sample-web-server-config.ctmpl", "/etc/sample-web-server/conf.ctmpl"
        add "$project.buildDir/resources/main/conf/sample-web-server-consul-template.cfg", "/etc/consul-template/conf/sample-web-server/sample-web-server-consul-template.cfg"
        volume "/etc/consul-template/conf/sample-web-server"
        volume "/etc/sample-web-server"
    }

}


class Dockerfile extends DefaultTask {
    def dockerfileInfo = ""
    def dockerDir = "$project.buildDir/docker"
    def dockerfileDestination = "$project.buildDir/docker/Dockerfile"
    def filesToCopy = []

    File getDockerfileDestination() {
        project.file(dockerfileDestination)
    }

    def from(image="java") {
        dockerfileInfo += "FROM $image\r\n"
    }

    def maintainer(contact) {
        maintainer += "MAINTAINER $contact\r\n"
    }

    def add(sourceLocation, targetLocation) {
        filesToCopy << sourceLocation
        def file = project.file(sourceLocation)
        dockerfileInfo += "ADD $file.name ${targetLocation}\r\n"
    }

    def run(command) {
        dockerfileInfo += "RUN $command\r\n"
    }

    def volume(path) {
        dockerfileInfo += "VOLUME $path\r\n"
    }

    def env(var, value) {
        dockerfileInfo += "ENV $var $value\r\n"
    }

    def expose(port) {
        dockerfileInfo += "EXPOSE $port\r\n"
    }

    def workdir(dir) {
        dockerfileInfo += "WORKDIR $dir\r\n"
    }

    def cmd(command) {
        dockerfileInfo += "CMD $command\r\n"
    }

    def entrypoint(command) {
        dockerfileInfo += "ENTRYPOINT $command\r\n"
    }

    @TaskAction
    def writeDockerfile() {
        for (fileName in filesToCopy) {
            def source = project.file(fileName)
            def target = project.file("$dockerDir/$source.name")
            target.parentFile.mkdirs()
            target.delete()
            target << source.bytes
        }
        def file = getDockerfileDestination()
        file.parentFile.mkdirs()
        file.write dockerfileInfo
    }
}

Kafka and Cassandra support, training for AWS EC2 Cassandra 3.0 Training