Sleepless Dev: Working with StatsD and Java

Run statsD daemon with docker

sudo docker run -d \
  --name graphite \
  -p 80:80 \
  -p 2003:2003 \
  -p 8125:8125/udp \
  hopsoft/graphite-statsd

Make sure you upgrade to the latest docker. I had to update boot2docker and update docker to get the above to work.

Update boot2docker

boot2docker update

brew upgrade docker

I installed docker manually, so to get the latest version with brew, I had to do this.

brew install docker
brew link --overwrite docker

Run script (OSX/Windows)

while true
do
   echo -n "example.statsd.counter.changed:$(((RANDOM % 10) + 1))|c" | nc -w 1 -u 192.168.59.103 8125
done

Then you can view the dashboard data at: http://192.168.59.103/dashboard

Note to get the ip address of docker

$ boot2docker ssh
$ ifconfig 
eth1      Link encap:Ethernet  HWaddr 08:00:27:E1:F5:54  
          inet addr:192.168.59.103  Bcast:192.168.59.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fee1:f554/64 Scope:Link

Note to get the ip address of the container running statsd

First find the container id.

$ docker ps

Then use the id to look up the container.

$ docker inspect --format '{{ .NetworkSettings.IPAddress }}' <CONTAINER ID FROM LAST STEP>

$ docker ps
CONTAINER ID        IMAGE                            COMMAND             CREATED             STATUS              PORTS                                                                NAMES
9183f205a3aa        hopsoft/graphite-statsd:latest   "/sbin/my_init"     About an hour ago   Up About an hour    0.0.0.0:80->80/tcp, 0.0.0.0:2003->2003/tcp, 0.0.0.0:8125->8125/udp   graphite   

$ docker inspect --format '{{ .NetworkSettings.IPAddress }}' 9183f205a3aa
172.17.0.1

If everything goes well

To start with let's try out the Java client.

The Java client is based on the reference Java client from the etsy project.

https://github.com/etsy/statsd/edit/master/examples/StatsdClient.java

public class UsingStatsDIncrement {

    public static void main(String... args) throws Exception {
        StatsdClient client = new StatsdClient("192.168.59.103", 8125);

        while (true) {
            client.increment("foo.bar.baz", 10, .1);
            Thread.sleep(1000);
        }
    }
}

I let this run for a while. Then I go to http://192.168.59.103/dashboard I look under "stats.foo.bar." in the nav tree. You many not really understand what you are seeing but there is a graph that goes from 0 to 20 sort of randomly. It can even go all the way up to 30.

Changing the sleep to 100 ms instead of 1000 should yield some different results.

Checking....

It does...

Now when we read and re-read the docs, they will more sense.

Now I changed 0.1 to 1.0 and let it run for a while and I get this nice flat line.

I added a gauge.

        StatsdClient client = new StatsdClient("192.168.59.103", 8125);

        int guageValue = 10;

        while (true) {
            client.increment("foo.bar.baz", 10, 1.0);
            client.gauge("gauge.foo.bar.baz", guageValue++, 1.0);

            if (guageValue > 100) {
                guageValue = 20;
            }
            Thread.sleep(100);
        }
    }

Then I added timings and started clicking around.

    public static void main(String... args) throws Exception {
        StatsdClient client = new StatsdClient("192.168.59.103", 8125);

        int gaugeValue = 10;

        while (true) {
            client.increment("foo.bar.baz", 10, 1.0);
            client.gauge("gauge.foo.bar.baz", gaugeValue++, 1.0);
            client.timing("foo.bar.baz.mytiming", gaugeValue, 1.0);

            if (gaugeValue > 100) {
                gaugeValue = 20;
            }
            Thread.sleep(100);
        }
    }
}

Concepts

A good description of concepts related to the domain model of statsD is documented here:

https://github.com/b/statsd_spec and here https://github.com/etsy/statsd/wiki

It is sparse but perhaps complete.

The core concepts for StatsD came from this 2008 blog post (according to Etsy documentation).

http://code.flickr.net/2008/10/27/counting-timing/

Although the early versions of StatsD seemed to use RRDtool and Ganglia.

http://oss.oetiker.ch/rrdtool/ http://code.flickr.net/2008/10/13/flickr-digs-ganglia/

While Statsd tends to use Graphite, and Whisper.

A good description of the wire protocol in more of a tutorial form can be found here:

https://www.digitalocean.com/community/tutorials/how-to-configure-statsd-to-collect-arbitrary-stats-for-graphite-on-ubuntu-14-04

StatsD was written to work with Graphite. Graphite is used to visualize the state of microservices. Graphite is made up of Graphite-Web that renders graphs and dashboards, Carbon metric processing daemons, and Whisper which is a time-series database library.

When you send a stat, you send these basic types:

c: This indicates a "count". g: This indicates a gauge. s: a mathematical set. ms: time span.

Count

The counts adds up values that StatsD receives for a metric within the flush interval and sends the total value. StatsD will collect all of the data it receives during its ten second flush interval and add them together to send a single value for that time frame.

Gauge

The gauge tells the current level of something, like memory used or #number of threads being used, etc. With the gauge you send the most recent value. StatsD sends Carbon the same value until it gets a different value.

Set

With sets, you send a bunch of values and StatsD and it will count the number of times it received unique values. Think of a set of enumerators, UP, DOWN, WARNING, CRITICAL, OK. You want to count how many times each occurs.