tag:blogger.com,1999:blog-61285252811010589332024-03-18T02:48:50.760-07:00Sleepless DevRick's Sleepless in Dublin. Random thoughts about software development. RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.comBlogger270125tag:blogger.com,1999:blog-6128525281101058933.post-28955985038848961612020-02-24T18:39:00.002-08:002020-02-24T18:39:40.686-08:00Tutorial Part 3: Kubernetes StatefulSet with ZooKeeper using Kustomize to target multiple environments<h1 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-left: 0px; margin-right: 0px; margin-top: 0px !important; padding-bottom: 0.3em;">
Kubernetes StatefulSet with ZooKeeper using Kustomize to target multiple environments</h1>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#background" id="user-content-background" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Background</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
ZooKeeper is a nice tool to start StatefulSets with because it is small and lightweight, yet exhibits a lot of the same needs as many disturbed, stateful, clustered applications.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
This is part 3. In part 1 of this tutorial, we got an example of a ZooKeeper StatefulSet running locally with <a href="https://kubernetes.io/docs/tasks/tools/install-minikube/" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">minkube</a>. In part 2, we got the same application running under Red Hat OSE CRC.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
This really builds on the last two tutorial and uses Kustomize to target multiple deployment environments. You could probably skip tutorial 2 and this one would still make sense.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Please refer to the</div>
<ul style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">first tutorial</a> and</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">second tutorial</a></li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
As stated before, I base this tutorial from the one on the Kubernetes site on ZooKeeper and StatefulSet but I am going to deploy to MiniKube, local Open Shift, and KIND as well as add support for Kustomize and a lot more.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
I have a similar version of this Kubernetes ZooKeeper deploy working on a multi-node shared, corporate locked-down environment. This is a new version based on the example.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
In past tutorials, I simulated some of the problems that I ran into and hope it helps.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
If for some reason you would like more background on this tutorial series - <a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#background" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">background</a>.</div>
<blockquote style="background-color: white; border-left: 0.25em solid rgb(223, 226, 229); box-sizing: border-box; color: #6a737d; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin: 0px 0px 16px; padding: 0px 1em;">
<div style="box-sizing: border-box;">
BTW, This is not my first rodeo with <a href="https://github.com/cloudurable/zookeeper-cloud" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">ZooKeeper</a> or <a href="https://github.com/cloudurable/kafka-cloud" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Kafka</a> or even <a href="https://github.com/cloudurable/cassandra-image" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">deploying stateful clustered services (cassandra)</a> or <a href="https://www.linkedin.com/pulse/spark-cluster-metrics-influxdb-rick-hightower/" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">managing them</a> or setting up <a href="https://github.com/cloudurable/spark-cluster" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">KPIs</a>, but this is the first time I wrote about doing it with Kubernetes. I have also written <a href="https://github.com/advantageous/elekt" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">leadership election libs</a> and have done <a href="https://github.com/advantageous/qbit" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">clustering</a> with tools like ZooKeeper, namely, etcd and <a href="https://github.com/advantageous/elekt-consul" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Consul</a>.</div>
</blockquote>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#objectives" id="user-content-objectives" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Objectives</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
After this tutorial, you will know the following.</div>
<ul style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">How to deploy a ZooKeeper ensemble to multiple environments using Kustomize</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to create base configs</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to create overlays.</li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Later follow on tutorials might show:</div>
<ul style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">How to write deploy scripts with Helm 3 to target local vs. remote deployments</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to write deploy scripts with Helm 3 to target local vs. remote deployments</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to create your metrics gatherers and use them with Prometheus</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to install Kafka on top of ZooKeeper</li>
</ul>
<hr style="background: rgb(225, 228, 232); border: 0px; box-sizing: initial; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; height: 0.25em; margin: 24px 0px; padding: 0px;" />
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#before-you-begin" id="user-content-before-you-begin" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Before you begin</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Before starting this tutorial, you should be familiar with the following Kubernetes concepts.</div>
<ul style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">Pods</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Cluster DNS</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Headless Services</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">PersistentVolumes</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">PersistentVolume Provisioning</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">StatefulSets</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">PodDisruptionBudgets</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">PodAntiAffinity</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">kubectl CLI</li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Recall that the tutorial on the Kubernetes site required a cluster with at least four nodes (with 2 CPUs and 4 GiB of memory), this one will work with local Kubernetes dev environments, namely, Open Shift CRC and MiniKube. This tutorial will show how to use <a href="https://kustomize.io/" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Kustomize</a> to target local dev and a real cluster.</div>
<hr style="background: rgb(225, 228, 232); border: 0px; box-sizing: initial; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; height: 0.25em; margin: 24px 0px; padding: 0px;" />
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#use-kustomize-to-deploy-our-zookeeper-statefulset-to-multiple-environments" id="user-content-use-kustomize-to-deploy-our-zookeeper-statefulset-to-multiple-environments" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Use Kustomize to deploy our ZooKeeper StatefulSet to multiple environments</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
To target multiple environments we will use Kubernetes Kustomize.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Kustomize is built into Kubernetes. It is the default way to target multiple deployment environments.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Kustomize is a template-free way to customize Kubernetes object files by using overlays and <a href="https://kubectl.docs.kubernetes.io/pages/reference/kustomize.html" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">directives</a> called transformers, meta sources, and generators.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
In this tutorial we will use:</div>
<ul style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">bases</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">commonLabels</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">patchesStrategicMerge</li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">bases</code> is path list which consists of: directories, URL or git referring to kustomization.yamls. You can think of <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">bases</code> similar to a base image in a Dockerfile (using FROM) or a parent pom in maven or a base class. From this <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">bases</code> you can transform and add additional details. This allows you to layer config and override config declared in the base. In our example, we will extend the base first for <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">dev</code> then later for <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">dev-3</code> and <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">prod</code>. Let's look a the directory structure.</div>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#where-to-find-the-code" id="user-content-where-to-find-the-code" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Where to find the code</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
You can find the code for this project at:</div>
<ul style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/tree/kustomize" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Kustomize</a> - branch that split up the manifest into multiple deployment environments.</li>
</ul>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#directory-structure-with-base-and-overlay" id="user-content-directory-structure-with-base-and-overlay" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Directory structure with base and overlay.</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">├── README.md
├── base
│ ├── kustomization.yml
│ └── zookeeper.yml
├── container
│ ├── Dockerfile
│ ├── README.md
│ ├── build.sh
│ ├── scripts
│ │ ├── metrics.sh
│ │ ├── ready_live.sh
│ │ └── start.sh
│ └── util
│ ├── debug.sh
│ ├── ...
└── overlays
├── dev
│ ├── kustomization.yml
│ └── zookeeper-dev.yml
├── dev-3
│ ├── kustomization.yml
│ └── zookeeper-dev3.yml
└── prod
├── kustomization.yml
└── zookeeper-prod.yml</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Notice we have an <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">overlays</code> directory and a <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">base</code> directory. The base directory will look a lot like our <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper.yml</code> manifest file from the last two tutorials. The <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">overlays</code> directory has a directory per target environment, namely, <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">prod</code>, <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">dev</code> and <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">dev-3</code>. Then you can split the data specific to production (<code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">prod</code>) or lightweight <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">dev</code> or our <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">dev-3</code> (dev but running three zookeeper instances in the ensemble).</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#bases-and-overlays-for-this-tutorial" id="user-content-bases-and-overlays-for-this-tutorial" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Bases and overlays for this tutorial</h4>
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 16px; overflow-wrap: normal; overflow: auto; padding: 16px;"><code style="background: initial; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow-wrap: normal; overflow: visible; padding: 0px; word-break: normal;">+---------------+ +---------------------+ +-------------------+
| | overrides | | ovr | |
| base +----------->+ overlay/dev +------> overlay/dev-3 |
| | | | | |
+--------+------+ +---------------------+ +-------------------+
|
|
|
| +----------------------+
| overrides | |
+-------------------------->+ overlay/prod |
| |
+----------------------+
</code></pre>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#base-directory" id="user-content-base-directory" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Base Directory</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
This is a simple example so we just have two files in our <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">base</code> directory.</div>
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 16px; overflow-wrap: normal; overflow: auto; padding: 16px;"><code style="background: initial; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow-wrap: normal; overflow: visible; padding: 0px; word-break: normal;">├── base
├── kustomization.yml
└── zookeeper.yml
</code></pre>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#basekustomizationyml---kustomize-manifest-file-for-base-directory" id="user-content-basekustomizationyml---kustomize-manifest-file-for-base-directory" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>base/kustomization.yml - kustomize manifest file for base directory</h4>
<div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-ent" style="box-sizing: border-box; color: #22863a;">resources</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper.yml</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">commonLabels</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">app</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kustomization.yml</code> file is the manifest file for <em style="box-sizing: border-box;"><span style="box-sizing: border-box; font-weight: 600;">kustomize</span></em>. The directive <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">resources: - zookeeper.yml</code> is specifying the yaml file, which looks a lot like the last one from the last tutorial. The <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">commonLabels: app: zookeeper</code> is specifying common labels and selectors so you don't have to repeat the same labels over and over.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">commonLabels</code> sets labels on all kube objects. The <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">commonLabels</code> are applied both to label selector fields and label fields.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper.yaml</code> file is just like before except there is nothing specific to <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">prod</code> or <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">dev</code> and there are no labels because we supplied them with <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">commonLabels</code> already.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#basezookeeperyml---zookeeper-manifest-with-no-labels" id="user-content-basezookeeperyml---zookeeper-manifest-with-no-labels" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>base/zookeeper.yml - zookeeper manifest with no labels</h4>
<div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-ent" style="box-sizing: border-box; color: #22863a;">apiVersion</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">v1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">kind</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">Service</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper-headless</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">ports</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">port</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">2888</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">server</span>
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">port</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">3888</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">leader-election</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">clusterIP</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">None</span>
---
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">apiVersion</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">v1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">kind</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">Service</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper-service</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">ports</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">port</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">2181</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">client</span>
---
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">apiVersion</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">policy/v1beta1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">kind</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">PodDisruptionBudget</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper-pdb</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">maxUnavailable</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1</span>
---
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">apiVersion</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">apps/v1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">kind</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">StatefulSet</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">serviceName</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper-headless</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">replicas</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">updateStrategy</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">type</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">RollingUpdate</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">podManagementPolicy</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">OrderedReady</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">template</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">containers</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">kubernetes-zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">imagePullPolicy</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">Always</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">image</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>cloudurable/kube-zookeeper:0.0.4<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">resources</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">requests</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">memory</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>500Mi<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">cpu</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>0.25<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">ports</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">containerPort</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">2181</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">client</span>
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">containerPort</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">2888</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">server</span>
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">containerPort</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">3888</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">leader-election</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">command</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">sh</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">-c</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>start.sh --servers=3 <span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">readinessProbe</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">exec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">command</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">sh</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">-c</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>ready_live.sh 2181<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">initialDelaySeconds</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">10</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">timeoutSeconds</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">5</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">livenessProbe</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">exec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">command</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">sh</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">-c</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>ready_live.sh 2181<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">initialDelaySeconds</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">10</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">timeoutSeconds</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">5</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">volumeMounts</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">datadir</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">mountPath</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">/var/lib/zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">securityContext</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">runAsUser</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1000</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">runAsGroup</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1000</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">fsGroup</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1000</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">initContainers</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">init-zoo</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">command:
</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">chown
</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">-R
</span>
- <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1000:1000</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">/var/lib/zookeeper
</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">image</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">ubuntu:18.04</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">imagePullPolicy</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">Always
</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">resources</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">{}
</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">securityContext:
</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">runAsUser</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0
</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">terminationMessagePath</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">/dev/termination-log
</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">terminationMessagePolicy</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">File
</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">volumeMounts:
</span>
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">datadir</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">mountPath</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">/var/lib/zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">volumeClaimTemplates</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">datadir</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">accessModes</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">[ "ReadWriteOnce" ]</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">resources</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">requests</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">storage</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">10Gi</span>
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">commonLabels</code> is a <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kustomize</code> transform as it transforms the ZooKeeper Kubernetes and adds labels and match labels to resources. This transform gets rid of a lot of duplicate code.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Now we want to base another config based on this base. It is like we inherit all of the settings and just override the ones that we want. Let's first do this with dev.</div>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#dev-directory" id="user-content-dev-directory" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Dev Directory</h2>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">└── overlays
└─── dev
├── kustomization.yml
└── zookeeper-dev.yml</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
You want to override the replicas and set it to 1 so we save memory when we run this on our laptop. You want to use the pod affinity <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">preferredDuringSchedulingIgnoredDuringExecution</code> so that we can run this on a local dev Kubernetes cluster that only has one node. You also want to change the command that starts up zookeeper to specify the number of servers in the ensemble, e.g., <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">command: ... start.sh --servers=1</code>. Lastly, you want to run with less memory and less CPU so it fits nicely on our laptop.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
To do this we will specify a yaml file with just the things we want to override and then specify that in the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kustomization.yml</code> manifest file we want to merge/patch using this file by using <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">patchesStrategicMerge:</code> and <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">bases:</code>. This is another transformation. This time against the output of the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">base</code> manifest.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#devkustomizationyml---kustomize-manifest-used-to-override-base" id="user-content-devkustomizationyml---kustomize-manifest-used-to-override-base" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>dev/kustomization.yml - kustomize manifest used to override base</h4>
<div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-ent" style="box-sizing: border-box; color: #22863a;">bases</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">../../base/</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">patchesStrategicMerge</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper-dev.yml</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">commonLabels</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">deployment-env</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">dev</span>
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Notice that <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">bases</code> specifies the base yaml file from before. Then <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">patchesStrategicMerge</code> specifies a yaml file with just the parts that you want to override. We go ahead and add a <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">dev</code> label.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">patchesStrategicMerge</code> applies patches to the matching Kube object (it matches by Group/Version/Kind + Name/Namespace). The patch file you specify is <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper-dev.yml</code> and it contains just the config definitions that you want to override. This keeps files for an environment small as the environment specific file just overrides the parts that are different.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Let's look at the bits that we override.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#devzookeeper-devyml---dev-file-used-to-override-base" id="user-content-devzookeeper-devyml---dev-file-used-to-override-base" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>dev/zookeeper-dev.yml - dev file used to override base</h4>
<div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-ent" style="box-sizing: border-box; color: #22863a;">apiVersion</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">apps/v1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">kind</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">StatefulSet</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">replicas</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">template</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">affinity</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">podAntiAffinity</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">preferredDuringSchedulingIgnoredDuringExecution</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">weight</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">100</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">podAffinityTerm</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">labelSelector</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">matchExpressions</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">key</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>app<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">operator</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">In</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">values</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">topologyKey</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>kubernetes.io/hostname<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">containers</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">kubernetes-zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">command</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">sh</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">-c</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>start.sh --servers=1 <span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">resources</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">requests</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">memory</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>250Mi<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">cpu</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>0.12<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Notice we added dev labels to everything, just in case we want to deploy another version at the same time in the same namespace and to show we used the correct manifest.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
You don't have to extend just a base image, you can base and overlay on another overlay.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
To deploy this run the following:</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#deploy-dev" id="user-content-deploy-dev" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Deploy dev</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span> kubectl delete -f zookeeper.yaml from the other branch</span>
kubectl apply -k overlay/dev
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The -k option is for Kustomize. You specify the directory of the config that you want to run, which is usually an overlay directory because bases don't usually deploy.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Now would be a good time to refer to the first tutorial and test the ZooKeeper instance (perhaps read and write a key).</div>
<h3 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#can-you-overlay-and-overlay" id="user-content-can-you-overlay-and-overlay" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Can you overlay and overlay?</h3>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Can you overlay and overlay?</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
For example, let's say we ran into an issue that we can not easily reproduce and we think it is because we use three zookeeper instances in the integration environment but the only one when we are testing locally so you decide to create an overlay that has three images that run locally on your laptop cluster to closer mimic integration. We call this environment <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">dev-3</code> because we are not very creative.</div>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#dev-3-for-testing-with-zookeeper-for-three-node-ensemble" id="user-content-dev-3-for-testing-with-zookeeper-for-three-node-ensemble" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Dev 3 for testing with ZooKeeper for three node ensemble</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
You decide to create an overlay that has three images that run locally on your laptop cluster to closer mimic integration to chase down a bug. The files are laid out similar to before.</div>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"> ├── dev-3
│ ├── kustomization.yml
│ └── zookeeper-dev3.yml</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The manifest file will overlay the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">dev</code> environment, and override the number of replicas as well as passing the replica count to the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">start.sh</code> script of the container.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#dev-3kustomizationyaml" id="user-content-dev-3kustomizationyaml" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>dev-3/kustomization.yaml</h4>
<div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-ent" style="box-sizing: border-box; color: #22863a;">bases</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">../dev/</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">patchesStrategicMerge</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper-dev3.yml</span>
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Noice the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">base</code> refers to <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">../dev/</code> (AKA <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">overlay/dev</code>) so you can see that this overlay is just overriding values from <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">overlay/dev</code> like affinity <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">preferredDuringSchedulingIgnoredDuringExecution</code> so we can still run on a dev cluster.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#dev-3zookeeper-dev3yml" id="user-content-dev-3zookeeper-dev3yml" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>dev-3/zookeeper-dev3.yml</h4>
<div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-ent" style="box-sizing: border-box; color: #22863a;">apiVersion</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">apps/v1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">kind</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">StatefulSet</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">replicas</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">3</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">template</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">containers</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">kubernetes-zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">command</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">sh</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">-c</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>start.sh --servers=3<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The above just overrides the number of <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">servers</code> passed to <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">start.sh</code> and the number of <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">replicas</code>.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#deploy-dev-3" id="user-content-deploy-dev-3" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Deploy dev 3</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span> kubectl delete -k overlay/dev if you want first or not</span>
kubectl apply -k overlay/dev-3
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The main reason we did not specify affinity in the very base manifest for ZooKeeper is <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kustomize</code> does not allow us to remove YAML attributes. It only allows us to add. To add <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">requiredDuringSchedulingIgnoredDuringExecution</code> if <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">preferredDuringSchedulingIgnoredDuringExecution</code> was in the base would require that we remove an attribute. If you need to add / remove attributes based on logic, then you will need a template language like the one that ships with Helm which is a subject of a future tutorial for sure. :)</div>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#prod-directory" id="user-content-prod-directory" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Prod directory</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">prod</code> directory is our fictitious production environment for ZooKeeper. It is laid out much like <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">dev</code> or <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">dev-3</code>. Like <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">dev</code> its base is the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">base</code> directory.</div>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"> └── prod
├── kustomization.yml
└── zookeeper-prod.yml</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Notice like <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">dev</code> that <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">bases</code> specifies the base directory which contains the base ZooKeeper manifest. Also like before <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">patchesStrategicMerge</code> specifies a yaml file with just the parts that you want to override for production. You of course want to go ahead and add a <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">production</code> label.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Recall that the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">patchesStrategicMerge</code> applies patches to the matching Kube object (matched by Group/Version/Kind + Name/Namespace). The patch file you specify is <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper-prod.yml</code> and it contains just the config definitions that you want to override for production.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#prodkustomizationyaml---kustomize-manifest-file-for-prod" id="user-content-prodkustomizationyaml---kustomize-manifest-file-for-prod" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>prod/kustomization.yaml - kustomize manifest file for prod</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">bases:
- ../../base/
patchesStrategicMerge:
- zookeeper-prod.yml
commonLabels:
deployment-env: production
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Now you just specify the correct affinity for prod and goose up the memory and CPU for the pods.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#prodzookeeper-prodyaml" id="user-content-prodzookeeper-prodyaml" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>prod/zookeeper-prod.yaml</h4>
<div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-ent" style="box-sizing: border-box; color: #22863a;">apiVersion</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">apps/v1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">kind</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">StatefulSet</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">replicas</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">3</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">template</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">affinity</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">podAntiAffinity</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">requiredDuringSchedulingIgnoredDuringExecution</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">labelSelector</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">matchExpressions</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">key</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>app<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">operator</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">In</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">values</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">topologyKey</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>kubernetes.io/hostname<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">containers</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">kubernetes-zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">resources</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">requests</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">memory</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>500Mi<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">cpu</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>0.5<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Notice that this is specifies <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">requiredDuringSchedulingIgnoredDuringExecution</code> It also specifies more RAM and CPU per node as well as bumping up the ZooKeeper ensemble count to three.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#deploy-prod" id="user-content-deploy-prod" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Deploy prod</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span> kubectl delete -k overlay/dev-3 if you want first or not</span>
kubectl apply -k overlay/prod
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Notice that prod won't run unless you have multiple kubernetes workers so don't be surprised. After, just go back to dev.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#go-back-to-dev" id="user-content-go-back-to-dev" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Go back to dev</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl delete -k overlay/prod
kubectl apply -k overlay/dev
</pre>
</div>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#debugging-and-testing-that-it-all-works" id="user-content-debugging-and-testing-that-it-all-works" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Debugging and testing that it all works</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Recall how to do the following in case something didn't work or you want to ensure that the ZooKeeper ensemble is really working (hint: you should test it):</div>
<ul style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#check-status-of-pods-in-statefulset-for-the-third-time" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">How to check the status of pods in statefulset</a></li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#debug-why-the-statefulset-did-not-work" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">How to debug why the statefulset did not work</a></li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#delete-statefulset-objects-using-zookeeperyaml" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">How to delete statefulset objects using zookeeper.yaml</a></li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#checking-status-of-zookeeper-2-node-with-kubectl-describe" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">How to check the status of zookeeper-2 node with kubectl describe</a></li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#connecting-to-zookeeper-instance-to-and-see-if-it-is-working-if-you-need-to-debug-later" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">How to connect to ZooKeeper instance and see if it is working using netcat</a></li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#the-ruok-shows-up-a-lot-in-the-logs-beause-it-is-used-for-liveness-and-readiness-probes" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">How to use ruok shows</a></li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#use-kubectl-exec-and-zkclish-to-write-world-to-znode-hello-on-zookeeper-0" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">How to test with kubectl exec and zkCli.sh to write 'world' to znode /hello on zookeeper-0</a></li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#use-kubectl-exec-and-zkclish-to-read-from-znode-hello-on-zookeeper-1" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">How to test with kubectl exec and zkCli.sh to read from znode /hello on zookeeper-1</a></li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
You should really ensure that it works before you end this tutorial.</div>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#conclusion" id="user-content-conclusion" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Conclusion</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
In this tutorial, you created three overlays and on base config directory. The dev and prod overlays used the base dir base. The dev-3 directory used the dev directory as its base. The overlay directories inherit attributes from their parents (bases). The overlay directories can override or add attributes from its parent. The directive <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">patchesStrategicMerge</code> is used to override attributes from a base set of config. The directive <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">commonLabels</code> is used to transform resources by adding <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">labels</code> to resources and <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">select</code> <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">matches</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Kustomize is a template-free way to customize Kubernetes object files by using overlays and <a href="https://kubectl.docs.kubernetes.io/pages/reference/kustomize.html" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">directives</a> called transformers, meta sources, and generators. Kustomize is simpler to use than a full-blown template engine like Helm and allows you to have multiple deployment environments from the same base manifest files.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-3:-Kubernetes-StatefulSet-with-ZooKeeper-using-Kustomize-to-target-multiple-environments#bases-and-overlays-for-this-tutorial-1" id="user-content-bases-and-overlays-for-this-tutorial-1" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Bases and overlays for this tutorial</h4>
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px;"><code style="background: initial; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow-wrap: normal; overflow: visible; padding: 0px; word-break: normal;">+---------------+ +---------------------+ +-------------------+
| | overrides | | ovr | |
| base +----------->+ overlay/dev +------> overlay/dev-3 |
| | | | | |
+--------+------+ +---------------------+ +-------------------+
|
|
|
| +----------------------+
| overrides | |
+-------------------------->+ overlay/prod |
| |
+----------------------+
</code></pre>
<div>
<code style="background: initial; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow-wrap: normal; overflow: visible; padding: 0px; word-break: normal;"><br /></code></div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com7tag:blogger.com,1999:blog-6128525281101058933.post-1146369551836574782020-02-24T18:35:00.001-08:002020-02-24T18:35:59.938-08:00Tutorial Part 2: Kubernetes StatefulSet with ZooKeeper as an example on OpenShift<h1 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-left: 0px; margin-right: 0px; margin-top: 0px !important; padding-bottom: 0.3em;">
Kubernetes StatefulSet with ZooKeeper as an example on OpenShift</h1>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#background" id="user-content-background" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Background</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
This is part 2. In part 1 of this tutorial, we got an example of a ZooKeeper StatefulSet running locally with <a href="https://kubernetes.io/docs/tasks/tools/install-minikube/" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">minkube</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
This really builds on the last tutorial and deploys to Red Hat Open Shift local dev. Please refer to the <a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">first tutorial</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
As stated before, I base this tutorial from the one on the Kubernetes site on ZooKeeper and StatefulSet but I am going to deploy to MiniKube, local Open Shift, and KIND. I have a similar version of this Kubernetes ZooKeeper deploy working on a multi-node shared, corporate locked-down environment. This is a new version based on the example. I will simulate some of the issues that I encountered as I think there is a lot to learn.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
If for some reason you would like more background on this tutorial - <a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#background" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">background</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
ZooKeeper is a nice tool to start StatefulSets with because it is small and lightweight, yet exhibits a lot of the same needs as many disturbed, stateful, clustered applications.</div>
<blockquote style="background-color: white; border-left: 0.25em solid rgb(223, 226, 229); box-sizing: border-box; color: #6a737d; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin: 0px 0px 16px; padding: 0px 1em;">
<div style="box-sizing: border-box;">
BTW, This is not my first rodeo with <a href="https://github.com/cloudurable/zookeeper-cloud" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">ZooKeeper</a> or <a href="https://github.com/cloudurable/kafka-cloud" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Kafka</a> or even <a href="https://github.com/cloudurable/cassandra-image" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">deploying stateful clustered services (cassandra)</a> or <a href="https://www.linkedin.com/pulse/spark-cluster-metrics-influxdb-rick-hightower/" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">managing them</a> or setting up <a href="https://github.com/cloudurable/spark-cluster" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">KPIs</a>, but this is the first time I wrote about doing it with Kubernetes. I have also written <a href="https://github.com/advantageous/elekt" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">leadership election libs</a> and have done <a href="https://github.com/advantageous/qbit" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">clustering</a> with tools like ZooKeeper, namely, etcd and <a href="https://github.com/advantageous/elekt-consul" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Consul</a>.</div>
</blockquote>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#where-to-find-the-code" id="user-content-where-to-find-the-code" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Where to find the code</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
You can find the code for this project at:</div>
<ul style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/tree/redhat-minikube-1" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Mini-Kube</a> - branch that got the zookeeper example running in minikube and RedHat OSE installed on my local laptop.</li>
</ul>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#running-zookeeper-a-distributed-system-coordinator" id="user-content-running-zookeeper-a-distributed-system-coordinator" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Running ZooKeeper, A Distributed System Coordinator</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
This tutorial shows how to use StatefulSets in local dev environments as well as real clusters with many nodes and uses Apache Zookeeper. This tutorial will demonstrate Kubernetes StatefulSets as well as PodDisruptionBudgets, and PodAntiAffinity. Specifically, you will deploy to a local Open Shift instance (4.2, but 4.3 is out).</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
If you don't want to deploy to a Red Hat Open Shift environment, then just skip to tutorial three.</div>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#objectives" id="user-content-objectives" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Objectives</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
After this tutorial, you will know the following.</div>
<ul style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">How to deploy a ZooKeeper ensemble on Open Shift</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to debug common issues</li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Later follow on tutorials might show:</div>
<ul style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">How to write deploy scripts with Kustomize to target local vs. remote deployments</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to write deploy scripts with Helm 3 to target local vs. remote deployments</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to create your metrics gatherers and use them with Prometheus</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to install Kafka on top of ZooKeeper</li>
</ul>
<hr style="background: rgb(225, 228, 232); border: 0px; box-sizing: initial; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; height: 0.25em; margin: 24px 0px; padding: 0px;" />
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#before-you-begin" id="user-content-before-you-begin" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Before you begin</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Before starting this tutorial, you should be familiar with the following Kubernetes concepts.</div>
<ul style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">Pods</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Cluster DNS</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Headless Services</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">PersistentVolumes</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">PersistentVolume Provisioning</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">StatefulSets</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">PodDisruptionBudgets</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">PodAntiAffinity</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">kubectl CLI</li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
While the tutorial on the Kubernetes site required a cluster with at least four nodes (with 2 CPUs and 4 GiB of memory), this one will work with local Kubernetes dev environments, namely, Open Shift. The very next tutorial will show how to use <a href="https://kustomize.io/" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Kustomize</a> to target local dev and a real cluster. The default set up for minikube and Red Hat CodeReady Containers either dynamically provision PersistentVolumes or comes with enough out of the box to work.</div>
<hr style="background: rgb(225, 228, 232); border: 0px; box-sizing: initial; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; height: 0.25em; margin: 24px 0px; padding: 0px;" />
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#recall-our-trouble-in-paradise" id="user-content-recall-our-trouble-in-paradise" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Recall our trouble in paradise</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The zookeeper manifests that we started with would not run on minikube or local Open Shift (minishift or Red Hat CodeReady Containers) when we started. In the last tutorial, we showed how to run the ZooKeeper ensemble on minikube and how to debug issues like affinity/anti-affinity, lack of resources, etc.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The key was to change <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">requiredDuringSchedulingIgnoredDuringExecution</code> which blocks zookeeper nodes from being deployed on the same Kubernetes worker node/host (<code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">topologyKey: "kubernetes.io/hostname"</code>) to <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">preferredDuringSchedulingIgnoredDuringExecution</code>.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#zookeeperyaml---affinity-rules-preferredduringschedulingignoredduringexecution" id="user-content-zookeeperyaml---affinity-rules-preferredduringschedulingignoredduringexecution" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>zookeeper.yaml - affinity rules preferredDuringSchedulingIgnoredDuringExecution</h4>
<div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-ent" style="box-sizing: border-box; color: #22863a;">apiVersion</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">apps/v1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">kind</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">StatefulSet</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">selector</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">matchLabels</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">app</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">serviceName</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper-headless</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">...</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">template</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">labels</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">app</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">affinity</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">podAntiAffinity</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">preferredDuringSchedulingIgnoredDuringExecution</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">weight</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">100</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">podAffinityTerm</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">labelSelector</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">matchExpressions</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">key</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>app<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">operator</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">In</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">values</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">topologyKey</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>kubernetes.io/hostname<span class="pl-pds" style="box-sizing: border-box;">"</span></span></pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
In the next, tutorial we would like to use an overlay with Kustomize to override such config for local dev vs. an industrial integration or prod cluster. Please check that one out too.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Let's recap what you did so far. You modified the yaml manifest for our ZooKeeper <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">statefulset</code> to use <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">requiredDuringSchedulingIgnoredDuringExecution</code> versus <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">preferredDuringSchedulingIgnoredDuringExecution</code>. You then noticed that you did not have enough memory for Minikube so you increased. Along the way you did some debugging with <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kubectl describe</code>, <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kubectl get</code>, and <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kubectl exec</code>. Then you walked through the logs and compared what you know about <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">statefulsets</code> and ZooKeeper with the output of the logs. Then you ran a bunch of commands to prove to you that the ZooKeeper ensemble was really working. You even added znode and read them. Now let's get this running on Open Shift CRC.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#running-openshift-install-on-your-laptop-with-red-hat-codeready-containers" id="user-content-running-openshift-install-on-your-laptop-with-red-hat-codeready-containers" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Running OpenShift Install on your Laptop with Red Hat CodeReady Containers</h4>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
First, you need to install a local Open Shift.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
They used to call the Open Shift for local deploys minishift which was like minikube but for Open Shift. Now they changed it to Red Hat CodeReady Container which just rolls off the tongue. Fire that marketing guy! Minishift is for Open Shift 3.9 but not for the latest Open Shift of 4.3.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Follow the instructions <a href="https://cloud.redhat.com/openshift/install/crc/installer-provisioned" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">here (Install on your Laptop with Red Hat CodeReady Containers)</a>, you may have to sign up and become a Red Hat member.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
You may want to delete <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">minikube</code> (<code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">minikube delete</code>) or at least stop it unless you have an amazing laptop.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Once you install the open shift minishift like tool, aka CRC for short, you will want to run it.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#start-open-shift-crc" id="user-content-start-open-shift-crc" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Start Open Shift CRC</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">crc start --memory=16384
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
...
INFO Starting CodeReady Containers VM <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">for</span> OpenShift 4.2.13...
INFO Verifying validity of the cluster certificates ...
INFO Network restart not needed
INFO Check internal and public DNS query ...
INFO Check DNS query from host ...
INFO Starting OpenShift cluster ... [waiting 3m]
...
INFO
INFO You can now run <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">'</span>crc console<span class="pl-pds" style="box-sizing: border-box;">'</span></span> and use these credentials to access the OpenShift web console</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Then you want to add all of the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">oc</code> tools to the command line. The oc tools work with Open Shift.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#set-up-the-environment-variables-with-oc-env" id="user-content-set-up-the-environment-variables-with-oc-env" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Set up the environment variables with oc-env</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">eval</span> <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">$(</span>crc oc-env<span class="pl-pds" style="box-sizing: border-box;">)</span></span>
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Next, you will want to make sure you switch contexts using kubecx. If you don't know about kubectx, please see this <a href="http://cloudurable.com/blog/kubernetes_k8s_kubectl_cheat_sheet/index.html" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Kubernetes cheatsheet that I wrote</a>.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#use-kubectx-to-list-kubernetes-contexts" id="user-content-use-kubectx-to-list-kubernetes-contexts" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Use kubectx to list Kubernetes contexts</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectx
/api-crc-testing:6443/developer
default/api-crc-testing:6443/kube:admin
k8s-kafka
minikube
sys</pre>
</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#switch-context-with-kubectx-defaultapi-crc-testing6443kubeadmin" id="user-content-switch-context-with-kubectx-defaultapi-crc-testing6443kubeadmin" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Switch context with kubectx default/api-crc-testing:6443/kube:admin</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectx default/api-crc-testing:6443/kube:admin</pre>
</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#log-into-openshift-with-oc" id="user-content-log-into-openshift-with-oc" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Log into OpenShift with oc</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">oc login -u kubeadmin -p cznQP-pass-pass-pass
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
Login successful.
You have access to 51 projects, the list has been suppressed. You can list all projects with <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">'</span>oc projects<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
Using project <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>default<span class="pl-pds" style="box-sizing: border-box;">"</span></span>.</pre>
</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#apply-zookeeper-yaml-file-with-kubectl-and-see-by-default-that-it-fails" id="user-content-apply-zookeeper-yaml-file-with-kubectl-and-see-by-default-that-it-fails" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Apply zookeeper yaml file with kubectl and see by default that it fails</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl apply -f zookeeper.yaml
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## Output</span>
service/zookeeper-headless created
service/zookeeper-service created
poddisruptionbudget.policy/zookeeper-pdb created
statefulset.apps/zookeeper created
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span># See that the zookeeper-0 has an error</span>
kubectl get pods
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
NAME READY STATUS RESTARTS AGE
zookeeper-0 0/1 Error 0 13s
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span># describe zookeeper-0 and see about the error </span>
kubectl describe pods zookeeper-0
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
Name: zookeeper-0
Namespace: default
Priority: 0
...
Controlled By: StatefulSet/zookeeper
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 69s default-scheduler pod has unbound immediate PersistentVolumeClaims
Normal Scheduled 69s default-scheduler Successfully assigned default/zookeeper-0 to crc-k4zmd-
...
Normal Pulling 5s (x4 over 62s) kubelet, crc-k4zmd-master-0 Pulling image <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>cloudurable/kube-zookeeper:0.0.1<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
Normal Pulled 2s (x4 over 59s) kubelet, crc-k4zmd-master-0 Successfully pulled image <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>cloudurable/kube-zookeeper:0.0.1<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
Normal Created 2s (x4 over 59s) kubelet, crc-k4zmd-master-0 Created container kubernetes-zookeeper
Normal Started 2s (x4 over 59s) kubelet, crc-k4zmd-master-0 Started container kubernetes-zookeeper
Warning BackOff 1s (x7 over 57s) kubelet, crc-k4zmd-master-0 Back-off restarting failed container</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Now you can see that the pod <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeeper-0</code> is failing and Kubernetes is trying to restart it but why? Let's look at the logs and find out.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#use-kubectl-logs-zookeeper-0-to-see-what-is-going-on" id="user-content-use-kubectl-logs-zookeeper-0-to-see-what-is-going-on" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Use kubectl logs zookeeper-0 to see what is going on</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl logs zookeeper-0
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>This file was autogenerated DO NOT EDIT</span>
clientPort=2181
dataDir=/var/lib/zookeeper/data
dataLogDir=/var/lib/zookeeper/data/log
tickTime=2000
initLimit=10
syncLimit=5
maxClientCnxns=60
minSessionTimeout=4000
maxSessionTimeout=40000
autopurge.snapRetainCount=3
autopurge.purgeInteval=12
server.1=zookeeper-0.zookeeper-headless.default.svc.cluster.local:2888:3888
server.2=zookeeper-1.zookeeper-headless.default.svc.cluster.local:2888:3888
server.3=zookeeper-2.zookeeper-headless.default.svc.cluster.local:2888:3888
Creating ZooKeeper log4j configuration
mkdir: cannot create directory <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">'</span>/var/lib/zookeeper<span class="pl-pds" style="box-sizing: border-box;">'</span></span>: Permission denied
chown: cannot access <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">'</span>/var/lib/zookeeper/data<span class="pl-pds" style="box-sizing: border-box;">'</span></span>: Permission denied
mkdir: cannot create directory <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">'</span>/var/lib/zookeeper<span class="pl-pds" style="box-sizing: border-box;">'</span></span>: Permission denied
chown: invalid group: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">'</span>zookeeper:USER<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
/usr/bin/start.sh: line 161: /var/lib/zookeeper/data/myid: Permission denied
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
It looks like the container can't access the volume due to a permissions issue. Recall that the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper.yaml</code> runs the containers with a security context</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#zookeeperyaml---securitycontext-runasuser-and-fsgroup" id="user-content-zookeeperyaml---securitycontext-runasuser-and-fsgroup" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>zookeeper.yaml - securityContext runAsUser and fsGroup</h4>
<div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-ent" style="box-sizing: border-box; color: #22863a;">apiVersion</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">apps/v1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">kind</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">StatefulSet</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">selector</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">matchLabels</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">app</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">serviceName</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper-headless</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">...</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">...</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">containers</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">kubernetes-zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">image</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>cloudurable/kube-zookeeper:0.0.1<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">...</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">volumeMounts</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">datadir</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">mountPath</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">/var/lib/zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">securityContext</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">runAsUser</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1000</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">fsGroup</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1000</span></pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
When the volumes are mounted there is an issue with permissions. You could run our instances as root but you don't want to do that as a practice.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#kubernetes-is-a-v1162-while-minikube-was-at-v1160" id="user-content-kubernetes-is-a-v1162-while-minikube-was-at-v1160" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Kubernetes is a v1.16.2 while Minikube was at v1.16.0</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">% kubectl version
Client Version: version.Info{Major:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>1<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, Minor:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>16<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, GitVersion:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>v1.16.2<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, GitCommit:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>c97fe5036ef3df2967d086711e6c0c405941e14b<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, GitTreeState:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>clean<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, BuildDate:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>2019-10-15T23:42:50Z<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, GoVersion:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>go1.12.10<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, Compiler:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>gc<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, Platform:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>darwin/amd64<span class="pl-pds" style="box-sizing: border-box;">"</span></span>}
Server Version: version.Info{Major:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>1<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, Minor:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>14+<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, GitVersion:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>v1.14.6+a8d983c<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, GitCommit:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>a8d983c<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, GitTreeState:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>clean<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, BuildDate:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>2019-12-23T12:16:26Z<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, GoVersion:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>go1.12.12<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, Compiler:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>gc<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, Platform:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>linux/amd64<span class="pl-pds" style="box-sizing: border-box;">"</span></span>}</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
You ask, why are we getting a permissions error with Open Shift 4.2 and not Minikube. I don't know, but we are. I do know of one workaround. We can use an <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">initContainer</code>.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#zookeeperyaml---initcontainers-init-zoo" id="user-content-zookeeperyaml---initcontainers-init-zoo" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>zookeeper.yaml - initContainers->init-zoo</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">apiVersion: apps/v1
kind: StatefulSet
metadata:
name: zookeeper
spec:
selector:
matchLabels:
app: zookeeper
...
spec:
...
containers:
- name: kubernetes-zookeeper
imagePullPolicy: Always
image: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>cloudurable/kube-zookeeper:0.0.1<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
...
volumeMounts:
- name: datadir
mountPath: /var/lib/zookeeper
securityContext:
runAsUser: 1000
fsGroup: 1000
initContainers:
- name: init-zoo
command:
- chown
- -R
- 1000:1000
- /var/lib/zookeeper
image: ubuntu:18.04
imagePullPolicy: Always
name: volume-permissions
resources: {}
securityContext:
runAsUser: 0
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- name: datadir
mountPath: /var/lib/zookeeper</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Notice the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">initContainer init-zoo</code> has a <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">command</code> that uses <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">chown</code> to change the ownership of the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">/var/lib/zookeeper
</code> folder.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Now all of this works so delete and apply.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#recreate-the-zookeeper-statefulset" id="user-content-recreate-the-zookeeper-statefulset" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Recreate the zookeeper statefulset</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl apply -f zookeeper.yaml
kubectl delete -f zookeeper.yaml
</pre>
</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#recreate-the-zookeeper-statefulset-1" id="user-content-recreate-the-zookeeper-statefulset-1" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Recreate the zookeeper statefulset</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl get pods
NAME READY STATUS RESTARTS AGE
zookeeper-0 1/1 Running 0 90s
zookeeper-1 1/1 Running 0 61s
zookeeper-2 1/1 Running 0 30s</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Let's log into the OpenShift web console.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#start-up-openshift-web-console" id="user-content-start-up-openshift-web-console" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Start up OpenShift web console</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">crc console
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span># then login to the OpenShift web console</span>
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<img alt="" src="https://github.com/cloudurable/kube-zookeeper-statefulsets/raw/master/images/open-shift-zookeeper-topology.png" style="border-style: none; box-sizing: initial; max-width: 100%;" /></div>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#debugging-and-testing-that-it-all-works" id="user-content-debugging-and-testing-that-it-all-works" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Debugging and testing that it all works</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Recall how to do the following in case something didn't work or you want to ensure that the ZooKeeper ensemble is really working:</div>
<ul style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#check-status-of-pods-in-statefulset-for-the-third-time" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">How to check the status of pods in statefulset</a></li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#debug-why-the-statefulset-did-not-work" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">How to debug why the statefulset did not work</a></li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#delete-statefulset-objects-using-zookeeperyaml" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">How to delete statefulset objects using zookeeper.yaml</a></li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#checking-status-of-zookeeper-2-node-with-kubectl-describe" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">How to check the status of zookeeper-2 node with kubectl describe</a></li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#connecting-to-zookeeper-instance-to-and-see-if-it-is-working-if-you-need-to-debug-later" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">How to connect to ZooKeeper instance and see if it is working using netcat</a></li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#the-ruok-shows-up-a-lot-in-the-logs-beause-it-is-used-for-liveness-and-readiness-probes" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">How to use ruok shows</a></li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#use-kubectl-exec-and-zkclish-to-write-world-to-znode-hello-on-zookeeper-0" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">How to test with kubectl exec and zkCli.sh to write 'world' to znode /hello on zookeeper-0</a></li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#use-kubectl-exec-and-zkclish-to-read-from-znode-hello-on-zookeeper-1" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">How to test with kubectl exec and zkCli.sh to read from znode /hello on zookeeper-1</a></li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
You should really ensure that it works before you end this tutorial.</div>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-2:-Kubernetes-StatefulSet-with-ZooKeeper-as-an-example-on-OpenShift#conclusion" id="user-content-conclusion" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Conclusion</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px;">
You created the same project in Red Hat OpenShift CRC. You then debugged a permissions issue by using an <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">initContainer</code> to change the permissions.</div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com4tag:blogger.com,1999:blog-6128525281101058933.post-90484482475823516872020-02-21T12:06:00.003-08:002020-02-21T12:06:49.418-08:00Tutorial Part 1: Managing Kubernetes StatefulSets using ZooKeeper and Minikube<h1 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-left: 0px; margin-right: 0px; margin-top: 0px !important; padding-bottom: 0.3em;">
Kubernetes StatefulSet with ZooKeeper as an example</h1>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#background" id="user-content-background" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Background</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
I was having a really hard time deploying Kafka to Kubernetes. It worked fine when we were doing development and integration. We started with <a href="https://github.com/kubernetes/minikube" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Minikube</a> for local development.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
We created a MicroSerivce that uses Kafka in Spring Boot. We ran Kafka in minikube with Helm 2. By the way, Minikube is a mini Kubernetes that easily runs on macOS, Linux, and Windows. Minikube is great for local application development and supports a lot of Kubernetes. It is great for local testing and we also used it for integration testing.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Later we set up a pipeline and we wanted to use some of the same end to end tests that we used for local dev in Jenkins so we decided to switch to <a href="https://kind.sigs.k8s.io/" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Kind</a> for a variety of reasons. A chief reason was this was a shared Jenkins environment (so we can't just install stuff and docker was there already) and the other reason was although we could get <em style="box-sizing: border-box;"><span style="box-sizing: border-box; font-weight: 600;">minikube</span></em> to run on a Jenkins AWS worker instance there were too many limitations which KIND did not seem to have. And it could have just been our knowledge of Minikube, but Kind worked so we switched.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Kind is similar to minikube and it is also a tool for running mini Kubernetes clusters using Docker container as nodes. It was created to test Kubernetes but it fits well with our use cases.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
One issue we had with Kind was running some of the Helm 2 installs for Kafka. After trying to debug for a good 1/2 day, we tried Helm 3, and lo and behold, it just worked. We switched. Just like that. Bump and run. The path of least resistance.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
By the way, I wrote a <a href="http://cloudurable.com/blog/kubernetes_k8s_osx_setup_brew/index.html" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">getting started with minikube</a> and a <a href="http://cloudurable.com/blog/kubernetes_k8s_kubectl_cheat_sheet/index.html" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Kubernetes cheatsheet</a>, if you are new to minikube and Kubernetes, start there. I also wrote down a lot of tools that I use. The cheat sheet will be updated at some point to include Kind. Also, I would love some collaboration and tips to add to the cheatsheet. I need to give it a second look.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Let's see, we wanted to deploy to a shared environment. We tested our integration test scripts and Kafka installs on GKE (using Helm 3), local Open Shift (tried minishift but then switch to and now Red Hat CodeReady Containers), Minikube and KIND. Then we ran into an issue with a shared Open Shift container.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The same Helm 3 install that was working everywhere else was failing due to Pod policies which are not changeable due to corporate Infosec policies (so far anyway).</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Also at one point, we were told we can't use Helm 3 (some corporate policy), and we have to define our own docker containers. Now, I am not sure either of those is still true, but we learn and adapt. This has nothing to do with Open Shift. I am not an Open Shift hater. Now there could be a way to get the Helm 3 install to work. But since it does so much, I found tracking down the issues and being compliant was difficult.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The lead Infra/DevOps guy-in-charge told me to write my own StatefulSet, and do my own PV, PVC. I have always just used Helm for anything that needed PV or PVCs. I did. I learned a lot. Debugging and troubleshooting and working around pod policies not to mention differences in MiniKube, GKE, KIND, and Open Shift local vs. Open Shift shared corp gave me a new perspective. Which is why I decided to write some of this stuff down.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
My first attempt was to use Helm 3, have it spit out the manifest files and then debug it from there. But since it was doing both ZooKeeper and Kafka, it was a bit like drinking from a fire hose. I prefer the divide and conquer approach especially after the big bang approach does not work.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
I was lucky to find this <a href="https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">tutorial on managing statefulsets on the Kubernetes site using ZooKeeper</a>. The scripts as written did not work without change in any environment where I ran it except maybe GKE. Kubernetes has great documentation and an awesome community. Trial and error and troubleshooting is a good way to learn. I tried the statefulset manifest files as written locally and in the shared OSE env and it did not work. But at least it was small enough so I can follow.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
I base this tutorial from the one on the Kubernetes site on ZooKeeper and StatefulSet but I am going to deploy to MiniKube, local Open Shift, and KIND. (So far, I have written about MiniKube and OpenShift already and got it to work).</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
I have a similar version of this Kubernetes ZooKeeper deploy working on a multi-node shared, corporate locked-down environment. I had to do a lot of extra steps that don't make a lot of sense to me yet. This is a new version based on the example. I will simulate some of the issues that I encountered as I think there is a lot to learn while I went through this exercise.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
ZooKeeper is a nice tool to start StatefulSets with because it is small and lightweight, yet exhibits a lot of the same needs as many disturbed, stateful, clustered applications (Kafka, Hadoop, Cassandra, Consul, MongoDB, etc.).</div>
<blockquote style="background-color: white; border-left: 0.25em solid rgb(223, 226, 229); box-sizing: border-box; color: #6a737d; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin: 0px 0px 16px; padding: 0px 1em;">
<div style="box-sizing: border-box;">
BTW, This is not my first rodeo with <a href="https://github.com/cloudurable/zookeeper-cloud" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">ZooKeeper</a> or <a href="https://github.com/cloudurable/kafka-cloud" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Kafka</a> or even <a href="https://github.com/cloudurable/cassandra-image" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">deploying stateful clustered services (cassandra)</a> or <a href="https://www.linkedin.com/pulse/spark-cluster-metrics-influxdb-rick-hightower/" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">managing them</a> or setting up <a href="https://github.com/cloudurable/spark-cluster" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">KPIs</a>, but this is the first time I wrote about doing it with Kubernetes. I have also written <a href="https://github.com/advantageous/elekt" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">leadership election libs</a> and have done <a href="https://github.com/advantageous/qbit" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">clustering</a> with tools like ZooKeeper, namely, etcd and <a href="https://github.com/advantageous/elekt-consul" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Consul</a>.</div>
</blockquote>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#where-to-find-the-code" id="user-content-where-to-find-the-code" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Where to find the code</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
You can find the code for this project at:</div>
<ul style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><a href="https://github.com/cloudurable/kube-zookeeper-statefulsets/tree/redhat-minikube-1" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Mini-Kube</a> - branch that got the zookeeper example running in minikube and RedHat OSE installed on my local laptop.</li>
</ul>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#running-zookeeper-a-distributed-system-coordinator" id="user-content-running-zookeeper-a-distributed-system-coordinator" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Running ZooKeeper, A Distributed System Coordinator</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
This tutorial shows how to use StatefulSets in local dev environments as well as real clusters with many nodes and uses Apache Zookeeper. This tutorial will demonstrate Kubernetes StatefulSets as well as PodDisruptionBudgets, and PodAntiAffinity.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
It should augment <a href="https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Running ZooKeeper, A Distributed System Coordinator</a> but adds more details in debugging and more details regarding StatefulSets, Volumes, and PodAntiAffinity.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
This will be a standalone tutorial. You don't have to read the other one but I recommend it. If you are using MiniKube or MiniShift or Kind to learn Kubernetes, then this tutorial should work unlike the other.</div>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#objectives" id="user-content-objectives" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Objectives</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
After this tutorial, you will know the following.</div>
<ul style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">How to deploy a ZooKeeper ensemble using StatefulSet</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to deploy ZooKeeper servers on multiple nodes for availability</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to use PodDisruptionBudgets ensuring availability</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to use PodAntiAffinity to deploy to a local environment</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to use PodAntiAffinity to deploy to a production or integration environment</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to create your own Docker container that uses ZooKeeper</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to create your own liveness probes and ready probes</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to test that your ZooKeeper install really worked</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to debug common issues</li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Later follow on tutorials might show:</div>
<ul style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">How to deploy to OSE</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to write deploy scripts with Kustomize to target local vs. remote deployments</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to write deploy scripts with Helm 3 to target local vs. remote deployments</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to create your metrics gatherers and use them with Prometheus</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to install Kafka on top of ZooKeeper</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to install Istio with ZooKeeper to get dashboards</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to install Istio with ZooKeeper to get mTLS</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to write a Spring Boot app that uses Kafka, and Istio</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">How to write operators</li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
I have written these three at some level already:</div>
<ol style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">Tutorial 1: MiniKube</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Tutorial 2: Open Shift</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Tutorial 3: Using Kustomize</li>
</ol>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
I have not written tutorial 4 yet, but I already decided it will be on using config maps. Tutorial 5 will be running Kafka on top of ZooKeeper. I already wrote the code for tutorial 5.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
This tutorial should be fun and you can do it all from your laptop. :)</div>
<hr style="background: rgb(225, 228, 232); border: 0px; box-sizing: initial; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; height: 0.25em; margin: 24px 0px; padding: 0px;" />
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#before-you-begin" id="user-content-before-you-begin" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Before you begin</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Before starting this tutorial, you should be familiar with the following Kubernetes concepts.</div>
<ul style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">Pods</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Cluster DNS</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Headless Services</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">PersistentVolumes</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">PersistentVolume Provisioning</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">StatefulSets</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">PodDisruptionBudgets</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">PodAntiAffinity</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">kubectl CLI</li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
While the other tutorial required a cluster with at least four nodes (with 2 CPUs and 4 GiB of memory), this one will work with local Kubernetes dev environments. A later tutorial will show how to use <a href="https://kustomize.io/" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Kustomize</a> to target local dev and a real cluster. The default set up for minikube and Red Hat CodeReady Containers either dynamically provision PersistentVolumes or comes with enough out of the box to work.</div>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#zookeeper-basics" id="user-content-zookeeper-basics" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>ZooKeeper Basics</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<a href="https://zookeeper.apache.org/" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">ZooKeeper</a> is a distributed config system that uses a consensus algorithm. ZooKeeper is similar to Consul or etcd if you are familiar with them. It gets used by Kafka and Hadoop and quite a few others. Personally, I prefer Consul and etcd. But since a lot of projects use it, it is very common.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
ZooKeeper allows you to perform CRUD operations on config and watch for updates to the config. It uses a consensus algorithm that guarantees that all servers have the same view of the config more or less. The config can be the state of your cluster (which nodes are up or down who is the leader). Think of it like a consistent view of config data ordered in a file system like hierarchy. The major difference between a regular file system to store config is that a ZooKeeper cluster forms an ensemble so that all of the data is in-sync using a consensus algorithm. If you are familiar with these concepts from Consul, etcd or even Spark or Cassandra or MongoDB then you have a basic understanding of ZooKeeper too.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
While Consul and etcd use <a href="https://raft.github.io/" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">RAFT</a> as a consensus algorithm, ZooKeeper ensures uses the Zab consensus protocol to replicate data in a consistent state across all members of the ensemble. Both <a href="https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab+vs.+Paxos" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Zab</a>) and RAFT are faster, lighter than [Paxos](<a href="https://en.wikipedia.org/wiki/Paxos_(computer_science)" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">https://en.wikipedia.org/wiki/Paxos_(computer_science)</a> which you may have studied in school but have similar concepts.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
ZooKeeper uses Zab protocol to elect a leader. The ZooKeeper ensemble can't write data unless there is a leader. This keeps the data very consistent. The ZooKeeper ensemble replicates all writes to a quorum defined by Zab protocol before the data becomes visible to clients. As stated if you are familiar with quorums from Cassandra, MongoDB, etcd or Consul, it is really more of the same ideas behind quorums. A quorum is a majority of ZooKeeper nodes in the ensemble and the leader. "For instance, if the ensemble has three servers, a component that contains the leader and one other server constitute a quorum. If the ensemble can not achieve a quorum, the ensemble cannot write data." --<a href="https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/#zookeeper-basics" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Zookeeper Basics</a></div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
ZooKeeper servers store config in memory. The config memory is periodically written to disk. Also, every change to a ZooKeeper instance gets written to a Write Ahead Log (WAL) which is on disk.<br style="box-sizing: border-box;" />ZooKeeper nodes that crash or are updated, recover by reading the last snapshot and then replaying the WAL. After a snapshot, the WALs are deleted. This prevents the disk from filling up. --<a href="https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/#zookeeper-basics" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Zookeeper Basics</a></div>
<hr style="background: rgb(225, 228, 232); border: 0px; box-sizing: initial; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; height: 0.25em; margin: 24px 0px; padding: 0px;" />
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#set-up-kubernetes--minikube" id="user-content-set-up-kubernetes--minikube" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Set up Kubernetes / Minikube</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
This tutorial does not assume you even have Kubernetes set up at all. If you are using a <a href="http://cloudurable.com/blog/kubernetes_k8s_osx_setup_brew/index.html" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">OSX machine use this tutorial to set up minikube</a>. If you are not using OSX then first <a href="https://kubernetes.io/docs/tasks/tools/install-minikube/" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">install minikube</a> and then go through that tutorial and this <a href="http://cloudurable.com/blog/kubernetes_k8s_kubectl_cheat_sheet/index.html" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Kubernetes cheatsheet</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Once you feel comfortable with minikube, delete the cluster and create it again with this command.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#minikube" id="user-content-minikube" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Minikube</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">minikube start --kubernetes-version v1.16.0 \
--vm-driver=hyperkit \
--cpus=4 \
--disk-size=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">'</span>100000mb<span class="pl-pds" style="box-sizing: border-box;">'</span></span> \
--memory=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">'</span>6000mb<span class="pl-pds" style="box-sizing: border-box;">'</span></span></pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
We choose Kubernetes version 1.16.0, using hyperkit with 4 CPUs, 10GB of disk space and 6 GB of memory for the whole cluster.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
You should add some extra memory too and a bit extra disk space. This should all run on a modern laptop with at least 16GB of ram.</div>
<hr style="background: rgb(225, 228, 232); border: 0px; box-sizing: initial; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; height: 0.25em; margin: 24px 0px; padding: 0px;" />
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#creating-a-zookeeper-ensemble" id="user-content-creating-a-zookeeper-ensemble" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Creating a ZooKeeper Ensemble</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Just like the other example, this one has a manifest that contains a Headless Service, a Service, a PodDisruptionBudget, and a StatefulSet. --<a href="https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/#creating-a-zookeeper-ensemble" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Creating a ZooKeeper Ensemble</a></div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#zookeeperyaml-kubernetes-objects-headless-service-a-service-a-poddisruptionbudget-and-a-statefulset" id="user-content-zookeeperyaml-kubernetes-objects-headless-service-a-service-a-poddisruptionbudget-and-a-statefulset" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>zookeeper.yaml Kubernetes Objects: Headless Service, a Service, a PodDisruptionBudget, and a StatefulSet</h4>
<div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-ent" style="box-sizing: border-box; color: #22863a;">apiVersion</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">v1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">kind</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">Service</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper-headless</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">labels</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">app</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">ports</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">port</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">2888</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">server</span>
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">port</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">3888</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">leader-election</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">clusterIP</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">None</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">selector</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">app</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
---
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">apiVersion</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">v1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">kind</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">Service</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper-service</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">labels</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">app</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">ports</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">port</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">2181</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">client</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">selector</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">app</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
---
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">apiVersion</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">policy/v1beta1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">kind</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">PodDisruptionBudget</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper-pdb</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">selector</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">matchLabels</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">app</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">maxUnavailable</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1</span>
---
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">apiVersion</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">apps/v1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">kind</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">StatefulSet</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">selector</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">matchLabels</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">app</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">serviceName</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper-headless</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">replicas</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">3</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">updateStrategy</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">type</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">RollingUpdate</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">podManagementPolicy</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">OrderedReady</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">template</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">labels</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">app</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">affinity</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">podAntiAffinity</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">requiredDuringSchedulingIgnoredDuringExecution</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">labelSelector</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">matchExpressions</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">key</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>app<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">operator</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">In</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">values</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">topologyKey</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>kubernetes.io/hostname<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">containers</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">kubernetes-zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">imagePullPolicy</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">Always</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">image</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>cloudurable/kube-zookeeper:0.0.1<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">resources</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">requests</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">memory</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>1Gi<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">cpu</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>0.5<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">ports</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">containerPort</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">2181</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">client</span>
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">containerPort</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">2888</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">server</span>
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">containerPort</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">3888</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">leader-election</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">command</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">sh</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">-c</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>start.sh \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --servers=3 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --data_dir=/var/lib/zookeeper/data \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --data_log_dir=/var/lib/zookeeper/data/log \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --conf_dir=/opt/zookeeper/conf \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --client_port=2181 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --election_port=3888 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --server_port=2888 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --tick_time=2000 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --init_limit=10 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --sync_limit=5 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --heap=512M \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --max_client_cnxns=60 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --snap_retain_count=3 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --purge_interval=12 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --max_session_timeout=40000 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --min_session_timeout=4000 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --log_level=INFO<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">readinessProbe</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">exec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">command</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">sh</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">-c</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>ready_live.sh 2181<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">initialDelaySeconds</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">10</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">timeoutSeconds</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">5</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">livenessProbe</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">exec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">command</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">sh</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">-c</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>ready_live.sh 2181<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">initialDelaySeconds</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">10</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">timeoutSeconds</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">5</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">volumeMounts</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">datadir</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">mountPath</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">/var/lib/zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">securityContext</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">runAsUser</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1000</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">fsGroup</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1000</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">volumeClaimTemplates</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">datadir</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">accessModes</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">[ "ReadWriteOnce" ]</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">resources</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">requests</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">storage</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">10Gi</span>
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The above is almost the exact same as the one in the other tutorial except I changed the name of the scripts, the docker image and made the names longer so they would be easier to read. In other words, it won't work on minikube or local Open Shift (minishift or Red Hat CodeReady Containers), but how it doesn't work is a teachable moment. I learn the most when things break.</div>
<h3 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#create-the-statefulset-no-edits" id="user-content-create-the-statefulset-no-edits" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Create the statefulset no edits</h3>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
From the command line, use <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kubectl apply</code> to create the StatefulSet, services, etc.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#create-headless-service-a-service-a-poddisruptionbudget-and-a-statefulset" id="user-content-create-headless-service-a-service-a-poddisruptionbudget-and-a-statefulset" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Create Headless Service, a Service, a PodDisruptionBudget, and a StatefulSet</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl apply -f zookeeper.yaml
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>### OUTPUT</span>
service/zookeeper-headless created
service/zookeeper-service created
poddisruptionbudget.policy/zookeeper-pdb created
statefulset.apps/zookeeper created</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Next, let's check to see if the pods for the ZooKeeper StatefulSet were created.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#check-status-of-pods-in-statefulset" id="user-content-check-status-of-pods-in-statefulset" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Check status of pods in statefulset</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl get pods
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## Output</span>
NAME READY STATUS RESTARTS AGE
zookeeper-0 1/1 Running 0 63s
zookeeper-1 0/1 Pending 0 47s</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Wow. That is taking a long time. Well, go check to see if you got any texts on your phone and come back, then check again.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#check-status-of-pods-in-statefulset-again" id="user-content-check-status-of-pods-in-statefulset-again" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Check status of pods in statefulset again</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl get pods
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## Output</span>
NAME READY STATUS RESTARTS AGE
zookeeper-0 1/1 Running 0 70s
zookeeper-1 0/1 Pending 0 54s</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Wow. That is taking a long time. Well, go check to see if you got any emails and come back, then check again.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#check-status-of-pods-in-statefulset-for-the-third-time" id="user-content-check-status-of-pods-in-statefulset-for-the-third-time" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Check status of pods in statefulset for the third time.</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">% kubectl get pods
NAME READY STATUS RESTARTS AGE
zookeeper-0 1/1 Running 0 5m24s
zookeeper-1 0/1 Pending 0 5m8s
(⎈ <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span>minikube:default)richardhightower@Richards-MacBook-Pro kube-zookeeper-statefulsets %</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
It has been five minutes and the 2nd ZooKeeper node is just not getting created. Let's see why. We need a total of three ZooKeeper nodes to create an ensemble.</div>
<h3 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#debug-why-the-statefulset-did-not-work" id="user-content-debug-why-the-statefulset-did-not-work" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Debug why the statefulset did not work</h3>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
We can use the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kubectl describe</code> command to see the events for the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper-1</code> pod.</div>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl describe pod zookeeper-1
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
Name: zookeeper-1
Namespace: default
Priority: 0
Node: <span class="pl-k" style="box-sizing: border-box; color: #d73a49;"><</span>none<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">></span>
Labels: app=zookeeper
controller-revision-hash=zookeeper-7b7f6f8cb9
statefulset.kubernetes.io/pod-name=zookeeper-1
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <span class="pl-k" style="box-sizing: border-box; color: #d73a49;"><</span>unknown<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">></span> default-scheduler pod has unbound immediate PersistentVolumeClaims
Warning FailedScheduling <span class="pl-k" style="box-sizing: border-box; color: #d73a49;"><</span>unknown<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">></span> default-scheduler pod has unbound immediate PersistentVolumeClaims
Warning FailedScheduling <span class="pl-k" style="box-sizing: border-box; color: #d73a49;"><</span>unknown<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">></span> default-scheduler 0/1 nodes are available: 1 node(s) didn<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">'</span>t match pod affinity/anti-affinity, 1 node(s) didn<span class="pl-pds" style="box-sizing: border-box;">'</span></span>t satisfy existing pods anti-affinity rules. </pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Ok, we can see that we did not actually schedule this pod to run on any node because we only have one node. The message <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">0/1 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules</code>. Let's take a look at the affinity rule from our yaml file.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#zookeeperyaml---affinity-rules-requiredduringschedulingignoredduringexecution" id="user-content-zookeeperyaml---affinity-rules-requiredduringschedulingignoredduringexecution" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>zookeeper.yaml - affinity rules requiredDuringSchedulingIgnoredDuringExecution</h4>
<div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">...
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">apiVersion</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">apps/v1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">kind</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">StatefulSet</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">...</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">template</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">labels</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">app</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">affinity</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">podAntiAffinity</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">requiredDuringSchedulingIgnoredDuringExecution</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">labelSelector</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">matchExpressions</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">key</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>app<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">operator</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">In</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">values</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">topologyKey</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>kubernetes.io/hostname<span class="pl-pds" style="box-sizing: border-box;">"</span></span></pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The key here is that we set a rule via <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">requiredDuringSchedulingIgnoredDuringExecution</code> which blocks zookeeper nodes from being deployed on the same Kubernetes worker node/host (<code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">topologyKey: "kubernetes.io/hostname"</code>).</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
As I stated earlier, in later tutorials we would like to use an overlay with Kustomize to override such config for local dev vs. an industrial integration or prod cluster. But as a workaround, let's turn the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">affinity/podAntiAffinity</code> rule into more of a suggestion with <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">preferredDuringSchedulingIgnoredDuringExecution</code>. While you will likely want <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">requiredDuringSchedulingIgnoredDuringExecution</code> for production, you may get away with <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">preferredDuringSchedulingIgnoredDuringExecution</code> for development. What these statements do is really spelled out well in their name, but if you want to dive deeper, I suggest <a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Assigning Pods to Nodes</a>.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#zookeeperyaml---affinity-rules-preferredduringschedulingignoredduringexecution" id="user-content-zookeeperyaml---affinity-rules-preferredduringschedulingignoredduringexecution" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>zookeeper.yaml - affinity rules preferredDuringSchedulingIgnoredDuringExecution</h4>
<div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-ent" style="box-sizing: border-box; color: #22863a;">apiVersion</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">apps/v1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">kind</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">StatefulSet</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">selector</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">matchLabels</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">app</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">serviceName</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper-headless</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">...</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">template</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">labels</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">app</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">affinity</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">podAntiAffinity</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">preferredDuringSchedulingIgnoredDuringExecution</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">weight</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">100</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">podAffinityTerm</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">labelSelector</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">matchExpressions</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">key</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>app<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">operator</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">In</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">values</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">topologyKey</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>kubernetes.io/hostname<span class="pl-pds" style="box-sizing: border-box;">"</span></span></pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Ok. You will need to change to <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">preferredDuringSchedulingIgnoredDuringExecution</code> and remove <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">requiredDuringSchedulingIgnoredDuringExecution</code> (edit <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper.yaml</code>). Then you need to delete the Kubernetes objects from the cluster using the YAML file for the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">statefulset</code> and then recreate Kubernetes objects for <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">the statefulset</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
From the command line, delete the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">statefulset</code> objects.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#delete-statefulset-objects-using-zookeeperyaml" id="user-content-delete-statefulset-objects-using-zookeeperyaml" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Delete <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: inherit; margin: 0px; padding: 0.2em 0.4em;">statefulset</code> objects using zookeeper.yaml.</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl delete -f zookeeper.yaml
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
service <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>zookeeper-headless<span class="pl-pds" style="box-sizing: border-box;">"</span></span> deleted
service <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>zookeeper-service<span class="pl-pds" style="box-sizing: border-box;">"</span></span> deleted
poddisruptionbudget.policy <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>zookeeper-pdb<span class="pl-pds" style="box-sizing: border-box;">"</span></span> deleted
statefulset.apps <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>zookeeper<span class="pl-pds" style="box-sizing: border-box;">"</span></span> deleted
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Just to get a clean slate, go ahead and delete the persistent volume claims too. This is not required at all.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#delete-persistent-volume-claims" id="user-content-delete-persistent-volume-claims" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Delete persistent volume claims</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl delete pvc datadir-zookeeper-0
kubectl delete pvc datadir-zookeeper-1
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## Output</span>
persistentvolumeclaim <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>datadir-zookeeper-0<span class="pl-pds" style="box-sizing: border-box;">"</span></span> deleted
persistentvolumeclaim <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>datadir-zookeeper-1<span class="pl-pds" style="box-sizing: border-box;">"</span></span> deleted
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Now you have a pristine Kubernetes local dev cluster, let's create the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">statefulsets</code> objects again.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#create-statefulset-for-zookeeper-again-with-preferredduringschedulingignoredduringexecution" id="user-content-create-statefulset-for-zookeeper-again-with-preferredduringschedulingignoredduringexecution" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Create StatefulSet for ZooKeeper again with preferredDuringSchedulingIgnoredDuringExecution</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl apply -f zookeeper.yaml
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## Output </span>
service/zookeeper-headless created
service/zookeeper-service created
poddisruptionbudget.policy/zookeeper-pdb created
statefulset.apps/zookeeper created
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
As before, let's check the status of ZooKeeper pod creations. This time you can use the -w flag to watch the pod creation status change as it happens.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#use-kubectl-get-pods-to-see-zookeeper-pod-creations-status" id="user-content-use-kubectl-get-pods-to-see-zookeeper-pod-creations-status" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Use kubectl get pods to see ZooKeeper pod creations status</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">% kubectl get pods -w
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT </span>
NAME READY STATUS RESTARTS AGE
zookeeper-0 0/1 ContainerCreating 0 10s
zookeeper-0 0/1 Running 0 39s
zookeeper-0 1/1 Running 0 54s
zookeeper-1 0/1 Pending 0 1s
zookeeper-1 0/1 Pending 0 1s
zookeeper-1 0/1 Pending 0 2s
zookeeper-1 0/1 ContainerCreating 0 2s
zookeeper-1 0/1 Running 0 5s
zookeeper-1 1/1 Running 0 17s
zookeeper-2 0/1 Pending 0 0s
zookeeper-2 0/1 Pending 0 0s
zookeeper-1 0/1 Running 0 67s
zookeeper-1 1/1 Running 0 87s
...
kubectl get pods
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
NAME READY STATUS RESTARTS AGE
zookeeper-0 1/1 Running 0 4m41s
zookeeper-1 1/1 Running 0 3m47s
zookeeper-2 0/1 Pending 0 3m30s
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
It got stuck again. I wonder why. Well it looks like <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper-1</code> was created so our <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">preferredDuringSchedulingIgnoredDuringExecution</code> worked well. But, <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper-2</code>, never runs. Let's use <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kubectl describe</code> to see why</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#checking-status-of-zookeeper-2-node-with-kubectl-describe" id="user-content-checking-status-of-zookeeper-2-node-with-kubectl-describe" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Checking status of zookeeper-2 node with kubectl describe</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl describe pod zookeeper-2
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
...
Node-Selectors: <span class="pl-k" style="box-sizing: border-box; color: #d73a49;"><</span>none<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">></span>
Tolerations: node.kubernetes.io/not-ready:NoExecute <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">for</span> 300s
node.kubernetes.io/unreachable:NoExecute <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">for</span> 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <span class="pl-k" style="box-sizing: border-box; color: #d73a49;"><</span>unknown<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">></span> default-scheduler 0/1 nodes are available: 1 Insufficient memory.
Warning FailedScheduling <span class="pl-k" style="box-sizing: border-box; color: #d73a49;"><</span>unknown<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">></span> default-scheduler 0/1 nodes are available: 1 Insufficient memory.</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
You can see that there was <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">Insufficient memory</code> and thus <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper-2</code> <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">FailedScheduling</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Clearly you can see that <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kubectl describe</code> is a powerful tool to see errors.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
You can even use <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kubectl describe</code> with the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">statefulset</code> object itself. This means that you will have to recreate minikube with more memory.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#checking-status-of-zookeeper-statefulsets-with-kubectl-describe-and-get" id="user-content-checking-status-of-zookeeper-statefulsets-with-kubectl-describe-and-get" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Checking status of zookeeper statefulsets with kubectl describe and get</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl describe statefulsets zookeeper
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
Name: zookeeper
Namespace: default
CreationTimestamp: Tue, 11 Feb 2020 12:50:52 -0800
Selector: app=zookeeper
...
Replicas: 3 desired <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> 3 total
Pods Status: 2 Running / 1 Waiting / 0 Succeeded / 0 Failed
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 12m statefulset-controller create Claim datadir-zookeeper-0 Pod zookeeper-0 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">in</span> StatefulSet zookeeper success
Normal SuccessfulCreate 12m statefulset-controller create Pod zookeeper-0 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">in</span> StatefulSet zookeeper successful
Normal SuccessfulCreate 11m statefulset-controller create Claim datadir-zookeeper-1 Pod zookeeper-1 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">in</span> StatefulSet zookeeper success
Normal SuccessfulCreate 11m statefulset-controller create Pod zookeeper-1 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">in</span> StatefulSet zookeeper successful
Normal SuccessfulCreate 10m statefulset-controller create Claim datadir-zookeeper-2 Pod zookeeper-2 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">in</span> StatefulSet zookeeper success
Normal SuccessfulCreate 10m statefulset-controller create Pod zookeeper-2 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">in</span> StatefulSet zookeeper successful
...
kubectl get statefulsets zookeeper
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
NAME READY AGE
zookeeper 2/3 17m
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
You can see that it created all of the pods as it suppose to but it is forever waiting on the last pod by looking at <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">Pods Status: 2 Running / 1 Waiting / 0 Succeeded / 0 Failed</code>. Also doing a <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kubectl get statefulsets zookeeper</code> shows that only 2 of the 3 pods are ready as well (<code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper 2/3 17m</code>).</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Since the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">statefulset</code> wouldn't fit into memory on your local dev environment, let's recreate minikube with more memory. You will delete minikube and then create it again.</div>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">minikube delete
minikube start --kubernetes-version v1.16.0 \
--vm-driver=hyperkit \
--cpus=4 \
--disk-size=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">'</span>100000mb<span class="pl-pds" style="box-sizing: border-box;">'</span></span> \
--memory=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">'</span>7500mb<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
🔥 Deleting <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>minikube<span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">in</span> hyperkit ...
💔 The <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>minikube<span class="pl-pds" style="box-sizing: border-box;">"</span></span> cluster has been deleted.
...
😄 minikube v1.4.0 on Darwin 10.15.1
🔥 Creating hyperkit VM (CPUs=4, Memory=7500MB, Disk=100000MB) ...
🐳 Preparing Kubernetes v1.16.0 on Docker 18.09.9 ...
🚜 Pulling images ...
🚀 Launching Kubernetes ...
⌛ Waiting for: apiserver proxy etcd scheduler controller dns
🏄 Done<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">!</span> kubectl is now configured to use <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>minikube<span class="pl-pds" style="box-sizing: border-box;">"</span></span> </pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
It fit two ZooKeeper nodes/container/pods into 6GB so figured that Kube control plane took up some space and you just need another 1.5GB for our 3rd Zookeeper node. It was a swag. Less memory for <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">minikube</code> means more memory for your other development tools (IDE, etc.).</div>
<blockquote style="background-color: white; border-left: 0.25em solid rgb(223, 226, 229); box-sizing: border-box; color: #6a737d; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin: 0px 0px 16px; padding: 0px 1em;">
<div style="box-sizing: border-box;">
By the way, this eats up a lot of memory for just a local dev environment so later in another tutorial, we will run ZooKeeper in a single node mode. I have done this before either <a href="https://github.com/cloudurable/zookeeper-cloud/tree/master/zookeeper-image" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">here at ZooKeeper cloud</a> or <a href="https://github.com/cloudurable/kafka-cloud/tree/master/kafka-image" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">here at Kafka cloud</a>. This will allow us to create a single pod cluster for local dev which will save a lot of memory on our dev laptop.</div>
</blockquote>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Now let's create the ZooKeeper <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">statefulset</code> and check its status.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#create-zookeeper-statefulset-and-check-status-with-kubectl-get-pods--w" id="user-content-create-zookeeper-statefulset-and-check-status-with-kubectl-get-pods--w" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Create ZooKeeper <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: inherit; margin: 0px; padding: 0.2em 0.4em;">statefulset</code> and check status with kubectl get pods -w</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl apply -f zookeeper.yaml
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
service/zookeeper-headless created
service/zookeeper-service created
poddisruptionbudget.policy/zookeeper-pdb created
statefulset.apps/zookeeper created
kubectl get pods -w
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
NAME READY STATUS RESTARTS AGE
zookeeper-0 0/1 ContainerCreating 0 13s
zookeeper-0 0/1 Running 0 46s
zookeeper-0 1/1 Running 0 62s
zookeeper-1 0/1 Pending 0 0s
zookeeper-1 0/1 Pending 0 0s
zookeeper-1 0/1 Pending 0 1s
zookeeper-1 0/1 ContainerCreating 0 1s
zookeeper-1 0/1 Running 0 4s
zookeeper-1 1/1 Running 0 20s
zookeeper-2 0/1 Pending 0 0s
zookeeper-2 0/1 Pending 0 0s
zookeeper-2 0/1 Pending 0 1s
zookeeper-2 0/1 ContainerCreating 0 1s
zookeeper-2 0/1 Running 0 5s
zookeeper-2 1/1 Running 0 14s</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Now let's check the status of our ZooKeeper statefulset and the pods again.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#check-status-of-zookeeper-statefulset-and-its-pods" id="user-content-check-status-of-zookeeper-statefulset-and-its-pods" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Check status of ZooKeeper statefulset and its pods</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl get pods
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
NAME READY STATUS RESTARTS AGE
zookeeper-0 1/1 Running 1 25m
zookeeper-1 1/1 Running 0 24m
zookeeper-2 1/1 Running 0 24m
kubectl get statefulsets
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
NAME READY AGE
zookeeper 3/3 26m
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
All three are up and you can see that I got a phone call between creation and status check.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Now let's look around a bit.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#see-the-persistent-volumes" id="user-content-see-the-persistent-volumes" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>See the Persistent Volumes</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl get pv
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-20e452bf-fe67 10Gi RWO Delete Bound default/datadir-zookeeper-1 standard 26m
pvc-4e59068b-8170 10Gi RWO Delete Bound default/datadir-zookeeper-0 standard 27m
pvc-e8a57062-64b9 10Gi RWO Delete Bound default/datadir-zookeeper-2 standard 26m</pre>
</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#see-the-persistent-volumes-claims" id="user-content-see-the-persistent-volumes-claims" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>See the Persistent Volumes Claims</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl get pvc
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
datadir-zookeeper-0 Bound pvc-4e59068b-8170 10Gi RWO standard 28m
datadir-zookeeper-1 Bound pvc-20e452bf-fe67 10Gi RWO standard 27m
datadir-zookeeper-2 Bound pvc-e8a57062-64b9 10Gi RWO standard 26m
(⎈ <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span>minikube:default)richardhightower@Richards-MacBook-Pro kube-zookeeper-statefulsets
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
You can see that each instance claims a volume and you can also see that Minikube creates the volumes on the fly. It will be interesting to look at the default behavior of OpenShift later in the next article.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
How can we trust that this all works? You can look at logs for errors. You can connect to the Kubernetes console and see if we see get see errors. I mean so far it looks like it is working! Yeah!</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Let's connect to an instance and see if you can run some commands. You will use network cat (<a href="https://en.wikipedia.org/wiki/Netcat" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">nc</a>) to send <em style="box-sizing: border-box;"><span style="box-sizing: border-box; font-weight: 600;">ZooKeeper Commands</span></em> also known as "The Four Letter Words" to ZooKeeper port 2181 which was configured for the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper-service</code>'s <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">client</code> port service (see <a href="https://zookeeper.apache.org/doc/r3.4.14/zookeeperAdmin.html" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">ZooKeeper</a> admin guide for more details on ZooKeeper Commands).</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The command <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kubectl exec -it zookeeper-0</code> will run a command on a pod. The <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">-it</code> options allow us to have an interactive terminal so you will run bash and poke around with some ZooKeeper commands.</div>
<h3 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#connecting-to-zookeeper-instance-to-and-see-if-it-is-working-if-you-need-to-debug-later" id="user-content-connecting-to-zookeeper-instance-to-and-see-if-it-is-working-if-you-need-to-debug-later" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Connecting to ZooKeeper instance to and see if it is working (if you need to debug later)</h3>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> -it zookeeper-0 bash
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span># Check to see if the node is ok send ruok to port 2181 with network cat.</span>
$ <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">echo</span> <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Are you ok? <span class="pl-s" style="box-sizing: border-box;"><span class="pl-pds" style="box-sizing: border-box;">$(</span>echo ruok <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> nc 127.0.0.1 2181<span class="pl-pds" style="box-sizing: border-box;">)</span></span><span class="pl-pds" style="box-sizing: border-box;">"</span></span>
iamok
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span># Use the srvr to see if this node is the leader.</span>
$ <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">echo</span> srvr <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> nc localhost 2181 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> grep Mode
Mode: follower
$ <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exit</span>
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span># Let's try to find the leader of the ZooKeeper ensemble</span>
kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> -it zookeeper-1 bash
$ <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">echo</span> srvr <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> nc localhost 2181 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> grep Mode
Mode: leader
$ <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exit</span>
kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> -it zookeeper-2 bash
$ <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">echo</span> srvr <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> nc localhost 2181 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> grep Mode
Mode: follower
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Note: To run a zookeeper in standalone set <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">StatefulSet</code> <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">replicas</code> to 1 and the parameter <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">--servers</code> to 1. When you do this the mode will be <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">Mode: standalone</code>. This is useful for local development to save space. This will really help if you are trying to run some tests on your local laptop. Also, try adjusting the memory lower and allocate less CPU.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#zookeeperyaml---running-zookeeper-in-standalone-mode" id="user-content-zookeeperyaml---running-zookeeper-in-standalone-mode" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>zookeeper.yaml - Running ZooKeeper in standalone mode</h4>
<div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">...
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">apiVersion</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">apps/v1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">kind</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">StatefulSet</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">...</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">replicas</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">...</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">containers</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">kubernetes-zookeeper</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">...</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">resources</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">requests</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">memory</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>500Mi<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">cpu</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>0.25<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">...</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">command</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">sh</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">-c</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>start.sh \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --servers=1</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> ...<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Please note that <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper-1</code> is the ZooKeeper ensemble leader.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The ZooKeeper command <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">dump</code> lists the outstanding sessions and ephemeral nodes but you have run this from the leader. This only works on the leader. Your master is the one that returned <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">Mode: leader</code> from <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">echo srvr | nc localhost 2181 | grep Mode</code>.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#run-the-dump-command-on-the-master-and-others" id="user-content-run-the-dump-command-on-the-master-and-others" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Run the dump command on the master and others</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> -it zookeeper-1 bash
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span># Show the dump command</span>
$ <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">echo</span> dump <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> nc localhost 2181
SessionTracker dump:
Session Sets (0):
ephemeral nodes dump:
Sessions with Ephemerals (0):
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span># Get stats</span>
$ <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">echo</span> stat <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/127.0.0.1:35964[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 734
Sent: 733
Connections: 1
Outstanding: 0
Zxid: 0x100000000
Mode: leader
Node count: 4
Proposal sizes last/min/max: -1/-1/-1
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span># Notice the node count is 4</span>
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span># Get the environment for this ZooKeeper node</span>
$ <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">echo</span> envi <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> nc localhost 2181
Environment:
zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
host.name=zookeeper-1.zookeeper-headless.default.svc.cluster.local
java.version=11.0.6
java.vendor=Ubuntu
java.home=/usr/lib/jvm/java-11-openjdk-amd64
java.class.path=/usr/bin/../zookeeper-server/target/classes:/usr/bin/../build/classes:/usr/bin/../zookeeper-server/target/lib/<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">*</span>.jar:/usr/bin/../build/lib/<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">*</span>.jar:/usr/bin/...
java.io.tmpdir=/tmp
os.name=Linux
os.arch=amd64
os.version=4.15.0
user.name=zookeeper
user.home=/home/zookeeper
user.dir=/
$ <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">echo</span> conf <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> nc localhost 2181
clientPort=2181
dataDir=/var/lib/zookeeper/data/version-2
dataLogDir=/var/lib/zookeeper/data/log/version-2
tickTime=2000
maxClientCnxns=60
minSessionTimeout=4000
maxSessionTimeout=40000
serverId=2
initLimit=10
syncLimit=5
electionAlg=3
electionPort=3888
quorumPort=2888
peerType=0
zookeeper@zookeeper-1
// Monitor the cluster
$ <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">echo</span> mntr <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> nc localhost 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 827
zk_packets_sent 826
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count 4
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 27
zk_open_file_descriptor_count 27
zk_max_file_descriptor_count 1048576
zk_fsync_threshold_exceed_count 0
zk_followers 2 <span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span># <-------- Two followers for the leader. </span>
zk_synced_followers 2 <span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span># <-------- Two followers are in sync.</span>
zk_pending_syncs 0
zk_last_proposal_size -1
zk_max_proposal_size -1
zk_min_proposal_size -1</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
You could go through the <a href="https://zookeeper.apache.org/doc/r3.4.14/zookeeperAdmin.html" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">admin docs for ZooKeeper</a> to get details about each command, but you get the gist. There is a ZooKeeper ensemble and all three nodes are present. Some commands work differently depending of if they are run against a leader or a follower. All of the commands above are run by the leader.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Included in the image is a bash script called <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">metrics.sh</code> which will run <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">echo mntr | nc localhost 2181</code>.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#demonstrating-that-the-commands-run-different-per-zookeeper-node" id="user-content-demonstrating-that-the-commands-run-different-per-zookeeper-node" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Demonstrating that the commands run different per ZooKeeper Node</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span># Run against zookeeper-0</span>
kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> -it zookeeper-0 metrics.sh 2181
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 656
zk_packets_sent 655
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 4
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 27
zk_open_file_descriptor_count 25
zk_max_file_descriptor_count 1048576
zk_fsync_threshold_exceed_count 0
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span># Run against zookeeper-1</span>
kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> -it zookeeper-1 metrics.sh 2181
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 859
zk_packets_sent 858
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count 4
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 27
zk_open_file_descriptor_count 27
zk_max_file_descriptor_count 1048576
zk_fsync_threshold_exceed_count 0
zk_followers 2
zk_synced_followers 2
zk_pending_syncs 0
zk_last_proposal_size -1
zk_max_proposal_size -1
zk_min_proposal_size -1
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span># Run against zookeeper-2</span>
kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> -it zookeeper-2 metrics.sh 2181
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 850
zk_packets_sent 849
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 4
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 27
zk_open_file_descriptor_count 25
zk_max_file_descriptor_count 1048576
zk_fsync_threshold_exceed_count 0
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Notice that <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper-1</code> pod has two more attributes in its metrics namely <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zk_followers</code> and <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zk_synced_followers</code>. Only leaders have <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zk_followers</code> and <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zk_synced_followers</code> metrics.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Now let's check the log and see if everything is working.</div>
<h3 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#looking-at-the-zookeeper-logs-with-kubectl-logs" id="user-content-looking-at-the-zookeeper-logs-with-kubectl-logs" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Looking at the ZooKeeper logs with kubectl logs</h3>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl logs zookeeper-1 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> more
</pre>
</div>
<h3 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#annotated-logs-from-kubectl-logs-zookeeper-1" id="user-content-annotated-logs-from-kubectl-logs-zookeeper-1" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Annotated logs from kubectl logs zookeeper-1</h3>
<pre lang="output" style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 16px; overflow-wrap: normal; overflow: auto; padding: 16px;"><code style="background: initial; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow-wrap: normal; overflow: visible; padding: 0px; word-break: normal;">
...
### ANNOTATION JMX is enabled
ZooKeeper JMX enabled by default
Using config: /usr/bin/../etc/zookeeper/zoo.cfg
2020-02-11 21:25:24,278 [myid:] - INFO [main:QuorumPeerConfig@136] - Reading configuration from: /usr/bin/../etc/zookeeper/zoo.cfg
2020-02-11 21:25:24,285 [myid:] - INFO [main:QuorumPeer$QuorumServer@185] - Resolved hostname: zookeeper-1.zookeeper-headless.default.svc.cluster.local to address: zookeeper-1.zookeeper-headless.default.svc.cluster.local/172.17.0.14
2020-02-11 21:25:24,286 [myid:] - INFO [main:QuorumPeer$QuorumServer@185] - Resolved hostname: zookeeper-0.zookeeper-headless.default.svc.cluster.local to address: zookeeper-0.zookeeper-headless.default.svc.cluster.local/172.17.0.13
2020-02-11 21:25:24,288 [myid:] - WARN [main:QuorumPeer$QuorumServer@191] - Failed to resolve address: zookeeper-2.zookeeper-headless.default.svc.cluster.local
### ANNOTATION When zookeeper-1 started zookeeper-2 had not started yet. because the stateful set is set to start in Ordered mode
### ANNOTATION Since the yaml file uses podManagementPolicy: OrderedReady
### ANNOTATION zookeeper-0 will start and then zookeeper-1 will start and then zookeeper-1
java.net.UnknownHostException: zookeeper-2.zookeeper-headless.default.svc.cluster.local: Name or service not known
at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
...
### ANNOTATION The leadership election process is starting but the 3rd node (zookeeper-2) is not up yet (as expected)
2020-02-11 21:25:24,299 [myid:2] - INFO [main:QuorumPeerMain@130] - Starting quorum peer
2020-02-11 21:25:24,303 [myid:2] - INFO [main:ServerCnxnFactory@117] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
...
2020-02-11 21:25:24,322 [myid:2] - INFO [main:QuorumPeer@669] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2020-02-11 21:25:24,325 [myid:2] - INFO [main:QuorumPeer@684] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2020-02-11 21:25:24,329 [myid:2] - INFO [ListenerThread:QuorumCnxManager$Listener@736] - My election bind port: zookeeper-1.zookeeper-headless.default.svc.cluster.local/172.17.0.14:3888
2020-02-11 21:25:24,347 [myid:2] - INFO [QuorumPeer[myid=2]/0.0.0.0:2181:QuorumPeer@910] - LOOKING
2020-02-11 21:25:24,348 [myid:2] - INFO [QuorumPeer[myid=2]/0.0.0.0:2181:FastLeaderElection@813] - New election. My id = 2, proposed zxid=0x0
2020-02-11 21:25:24,353 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@584] - Cannot open channel to 3 at election address
...
zookeeper-2.zookeeper-headless.default.svc.cluster.local:3888 ### ANNOTATION expected
java.net.UnknownHostException: zookeeper-2.zookeeper-headless.default.svc.cluster.local
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:609)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:534)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:454)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:435)
at java.base/java.lang.Thread.run(Thread.java:834)
...
### ANNOTATION More leadership election ZAB dance
2020-02-11 21:25:24,358 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@595] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2020-02-11 21:25:24,358 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@595] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2020-02-11 21:25:24,358 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@595] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
### ANNOTATION We have a winner! zookeeper-1 is the leader .. he won the election and zookeeper-0 is the follower.
2020-02-11 21:25:24,560 [myid:2] - INFO [QuorumPeer[myid=2]/0.0.0.0:2181:QuorumPeer@992] - LEADING
...
### ANNOTATION The leadership eleciton took 230 milliseconds!
2020-02-11 21:25:24,578 [myid:2] - INFO [QuorumPeer[myid=2]/0.0.0.0:2181:Leader@380] - LEADING - LEADER ELECTION TOOK - 230
### ANNOTATION Later zookeeper-2, aka node 3 comes up, and the leader let's it know who is the boss
2020-02-11 21:25:45,403 [myid:2] - INFO [LearnerHandler-/172.17.0.15:34870:LearnerHandler@535] - Received NEWLEADER-ACK message from 3
...
### ANNOTATION You see a lot of runok commands, which are used by
2020-02-11 21:25:57,112 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@908] - Processing ruok command from /127.0.0.1:54550
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Go through the annotated output and compare that to what the <em style="box-sizing: border-box;"><span style="box-sizing: border-box; font-weight: 600;">ZooKeeper Basics</span></em> section said. Also notice that there are a lot of <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">Processing ruok command</code> in the log. This is because it has INFO level logging, and the ZooKeeper command for health check is <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">ruok</code> which should return <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">imok</code>.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#the-ruok-shows-up-a-lot-in-the-logs-beause-it-is-used-for-liveness-and-readiness-probes" id="user-content-the-ruok-shows-up-a-lot-in-the-logs-beause-it-is-used-for-liveness-and-readiness-probes" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>The ruok shows up a lot in the logs beause it is used for liveness and readiness probes</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">% kubectl logs zookeeper-1 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> grep ruok <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> wc -l
555
% kubectl logs zookeeper-0 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> grep ruok <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> wc -l
616
% kubectl logs zookeeper-2 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> grep ruok <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> wc -l
775</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">ready_live.sh</code> script uses <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">ruok</code> from the liveness and readiness probes.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#zookeeperyaml" id="user-content-zookeeperyaml" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>zookeeper.yaml</h4>
<div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">...
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">readinessProbe</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">exec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">command</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">sh</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">-c</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>ready_live.sh 2181<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">initialDelaySeconds</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">10</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">timeoutSeconds</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">5</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">livenessProbe</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">exec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">command</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">sh</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">-c</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>ready_live.sh 2181<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">initialDelaySeconds</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">10</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">timeoutSeconds</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">5</span>
...</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
This means that <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">ready_live.sh</code> gets called for the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">readinessProbe</code> and the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">livenessProbe</code>.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#ready_livesh-using-ruok-and-checking-for-imok" id="user-content-ready_livesh-using-ruok-and-checking-for-imok" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>ready_live.sh using ruok and checking for imok</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#!</span>/usr/bin/env bash</span>
OK=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">$(</span>echo ruok <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> nc 127.0.0.1 <span class="pl-smi" style="box-sizing: border-box; color: #24292e;">$1</span><span class="pl-pds" style="box-sizing: border-box;">)</span></span>
<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">if</span> [ <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span><span class="pl-smi" style="box-sizing: border-box; color: #24292e;">$OK</span><span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">==</span> <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>imok<span class="pl-pds" style="box-sizing: border-box;">"</span></span> ]<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">;</span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">then</span>
<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exit</span> 0
<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">else</span>
<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exit</span> 1
<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">fi</span></pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The results of <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">echo ruok | nc 127.0.0.1 2181</code> is stored into <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">OK</code> and if it is "imok" then the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">ready_live.sh</code> scripts return normally (0) or it signals that there was an error (1).</div>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#recap-1" id="user-content-recap-1" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Recap 1</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Let's recap what you did so far. You modified the YAML manifest for our ZooKeeper <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">statefulset</code> to use <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">requiredDuringSchedulingIgnoredDuringExecution</code> versus <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">preferredDuringSchedulingIgnoredDuringExecution</code>. You then noticed that you did not have enough memory for Minikube so you increased. Along the way you did some debugging with <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kubectl describe</code>, <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kubectl get</code>, and <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kubectl exec</code>. Then you walked through the logs can be compared what you know about <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">statefulsets</code> and ZooKeeper with the output of the logs.</div>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#facilitating-leader-election" id="user-content-facilitating-leader-election" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Facilitating Leader Election</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
This section is based on <a href="https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/#facilitating-leader-election" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Facilitating Leader Election</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Each ZooKeeper server node in the ZooKeeper ensemble has to have a unique identifier associated with a network address. These identifiers are known to every node.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
To get a list of all hostnames in our ZooKeeper ensemble use <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kubectl exec</code>.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#use-kubectl-exec-to-get-the-hostnames-of-the-pods-in-the-zookeeper-statefulset" id="user-content-use-kubectl-exec-to-get-the-hostnames-of-the-pods-in-the-zookeeper-statefulset" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Use kubectl exec to get the hostnames of the Pods in the Zookeeper StatefulSet.</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-k" style="box-sizing: border-box; color: #d73a49;">for</span> <span class="pl-smi" style="box-sizing: border-box;">i</span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">in</span> 0 1 2<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">;</span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">do</span> kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> zookeeper-<span class="pl-smi" style="box-sizing: border-box;">$i</span> -- hostname<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">;</span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">done</span>
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
zookeeper-0
zookeeper-1
zookeeper-2
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The above runs the command hostname on all three ZooKeeper pods.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The Kubernetes StatefulSet controller gives each Pod a unique hostname based on its index. The hostnames are "${statefulset_name}-${index}"". In the YAML manifest file the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">replicas</code> was set to 3. Therefore the StatefulSet controller creates three Pods with their hostnames set to <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper-0</code>, <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper-1</code>, and <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper-3</code>.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#zookeeperyaml-statefulset-replicas-is-set-to-3" id="user-content-zookeeperyaml-statefulset-replicas-is-set-to-3" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>zookeeper.yaml StatefulSet replicas is set to 3</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">...
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: zookeeper
spec:
selector:
matchLabels:
app: zookeeper
serviceName: zookeeper-headless
replicas: 3
...</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The ZooKeeper nodes store their server’s <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">id</code> in a file called myid in the zookeeper data directory (see name: datadir \ mountPath: /var/lib/zookeeper in the zookeeper.yaml file).</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Let's look at the contents of those files with <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kubectl exec</code> and cat.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#examining-each-zookeeper-nodes-id-file" id="user-content-examining-each-zookeeper-nodes-id-file" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Examining each ZooKeeper node's id file</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-k" style="box-sizing: border-box; color: #d73a49;">for</span> <span class="pl-smi" style="box-sizing: border-box;">i</span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">in</span> 0 1 2<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">;</span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">do</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">echo</span> <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>myid zookeeper-<span class="pl-smi" style="box-sizing: border-box; color: #24292e;">$i</span><span class="pl-pds" style="box-sizing: border-box;">"</span></span><span class="pl-k" style="box-sizing: border-box; color: #d73a49;">;</span> kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> zookeeper-<span class="pl-smi" style="box-sizing: border-box;">$i</span> -- cat /var/lib/zookeeper/data/myid<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">;</span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">done</span>
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>#OUTPUT</span>
myid zookeeper-0
1
myid zookeeper-1
2
myid zookeeper-2
3
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Yes that is right. Kubernetes uses 0 based indexing but ZooKeeper uses start at 1 based indexing. You are welcome.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Let's get the fully qualified domain name (<a href="https://en.wikipedia.org/wiki/Fully_qualified_domain_name" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">FQDN</a>) of each pod.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#get-fully-qualified-domain-name-with-kubectl-exec-hostname--f" id="user-content-get-fully-qualified-domain-name-with-kubectl-exec-hostname--f" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Get fully qualified domain name with kubectl exec hostname -f</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-k" style="box-sizing: border-box; color: #d73a49;">for</span> <span class="pl-smi" style="box-sizing: border-box;">i</span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">in</span> 0 1 2<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">;</span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">do</span> kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> zookeeper-<span class="pl-smi" style="box-sizing: border-box;">$i</span> -- hostname -f<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">;</span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">done</span>
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
zookeeper-0.zookeeper-headless.default.svc.cluster.local
zookeeper-1.zookeeper-headless.default.svc.cluster.local
zookeeper-2.zookeeper-headless.default.svc.cluster.local</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The zookeeper-headless service creates a domain for each pod in the statefulset.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The <a href="https://en.wikipedia.org/wiki/List_of_DNS_record_types" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">DNS A records</a> in Kubernetes DNS resolve the FQDNs to the Pods’ IP addresses. If the Pods gets reschedule or upgrading, the A records will put to new IP addresses, but the name will stay the same.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The ZooKeeper was configured to use a config file called zoo.cfg (<code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">/opt/zookeeper/conf/zoo.cfg</code> which is specified in the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">start.sh</code> file which we will cover later). You can use <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kubectl exec</code> to cat the contents of the zoo.cfg.</div>
<h3 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#optzookeeperconfzoocfg-contents-of-zookeeper-config-file" id="user-content-optzookeeperconfzoocfg-contents-of-zookeeper-config-file" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>/opt/zookeeper/conf/zoo.cfg Contents of zookeeper config file</h3>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> zookeeper-0 -- cat /opt/zookeeper/conf/zoo.cfg
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>This file was autogenerated DO NOT EDIT</span>
clientPort=2181
dataDir=/var/lib/zookeeper/data
dataLogDir=/var/lib/zookeeper/data/log
tickTime=2000
initLimit=10
syncLimit=5
maxClientCnxns=60
minSessionTimeout=4000
maxSessionTimeout=40000
autopurge.snapRetainCount=3
autopurge.purgeInteval=12
server.1=zookeeper-0.zookeeper-headless.default.svc.cluster.local:2888:3888
server.2=zookeeper-1.zookeeper-headless.default.svc.cluster.local:2888:3888
server.3=zookeeper-2.zookeeper-headless.default.svc.cluster.local:2888:3888</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
This file gets generated by <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">start.sh</code>'s <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">create_config()</code> function which gets included by the Dockerfile which we will cover later. Notice that <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">server.1</code>, <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">server.2</code>, and <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">server.3</code> properties are set to the fully qualified <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">URL:port</code> (the FDQNs) of the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper-headless</code> service. This is also done by a <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">for loop</code> in the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">start.sh</code>'s <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">create_config()</code> function. The <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">start.sh</code> is specified in the zookeeper.yaml file.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#zookeeperyaml---specifying-statefulsetspectemplatecontainers0command" id="user-content-zookeeperyaml---specifying-statefulsetspectemplatecontainers0command" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>zookeeper.yaml - specifying StatefulSet.spec.template.containers[0].command</h4>
<div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">...
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">apiVersion</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">apps/v1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">kind</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">StatefulSet</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">selector</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">matchLabels</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">app</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">...</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">template</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">labels</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">app</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">...</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">containers</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">kubernetes-zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">imagePullPolicy</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">Always</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">image</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>cloudurable/kube-zookeeper:0.0.1<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">...</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">command</span>:
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">sh</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;">-c</span>
- <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>start.sh <span class="pl-cce" style="box-sizing: border-box;">\ </span> </span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --servers=3 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --data_dir=/var/lib/zookeeper/data \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --data_log_dir=/var/lib/zookeeper/data/log \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --conf_dir=/opt/zookeeper/conf \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --client_port=2181 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --election_port=3888 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --server_port=2888 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --tick_time=2000 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --init_limit=10 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --sync_limit=5 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --heap=512M \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --max_client_cnxns=60 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --snap_retain_count=3 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --purge_interval=12 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --max_session_timeout=40000 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --min_session_timeout=4000 \</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"> --log_level=INFO<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">start.sh</code> command gets passed those arguments on startup by the Kubernetes node that starts up the ZooKeeper pod.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Using Kubernetes config maps would be a good example instead of passing all of this as hardcoded values.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#achieving-consensus" id="user-content-achieving-consensus" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Achieving Consensus</h4>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
See <a href="https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/#achieving-consensus" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Achieving Consensus</a>.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#sanity-testing-the-ensemble" id="user-content-sanity-testing-the-ensemble" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Sanity Testing the Ensemble</h4>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
This section is based on <a href="https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/#sanity-testing-the-ensemble" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Sanity Testing the Ensemble</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Let's test to see if our ZooKeeper ensemble actually works. You will use the <a href="https://www.tutorialspoint.com/zookeeper/zookeeper_cli.htm" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">zkCli.sh</a> which is the ZooKeeper command-line interface.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
With the ZooKeeper CLI operations you can:</div>
<ul style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">Create znodes</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Get data</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Watch znode for changes</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Set data into a znode</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Create children of a znode</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">List children of a znode</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Check Status</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Delete a znode</li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
A znode in ZooKeeper is like a file in that it has contents and like a folder in that, it can have children who are other znode.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Let's write the value <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">world</code> into our ZooKeeper ensemble called <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">/hello</code> into zookeeper-0.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#use-kubectl-exec-and-zkclish-to-write-world-to-znode-hello-on-zookeeper-0" id="user-content-use-kubectl-exec-and-zkclish-to-write-world-to-znode-hello-on-zookeeper-0" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Use kubectl exec and zkCli.sh to write 'world' to znode /hello on zookeeper-0</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> zookeeper-0 zkCli.sh create /hello world
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
...
2020-02-12 01:34:07,938 [myid:] - INFO [main:ZooKeeper@442] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain<span class="pl-smi" style="box-sizing: border-box;">$MyWatcher</span>@1d251891
2020-02-12 01:34:07,967 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn<span class="pl-smi" style="box-sizing: border-box;">$SendThread</span>@1025] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2020-02-12 01:34:07,985 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn<span class="pl-smi" style="box-sizing: border-box;">$SendThread</span>@879] - Socket connection established to localhost/127.0.0.1:2181, initiating session
2020-02-12 01:34:08,012 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn<span class="pl-smi" style="box-sizing: border-box;">$SendThread</span>@1299] - Session establishment <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">complete</span> on server localhost/127.0.0.1:2181, sessionid = 0x100005d64170000, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
Created /hello</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
You just wrote "world" to znode "/hello".</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Now, let's read it back but from a different zookeeper node.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#use-kubectl-exec-and-zkclish-to-read-from-znode-hello-on-zookeeper-1" id="user-content-use-kubectl-exec-and-zkclish-to-read-from-znode-hello-on-zookeeper-1" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Use kubectl exec and zkCli.sh to read from znode /hello on zookeeper-1</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> zookeeper-1 zkCli.sh get /hello
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
...
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
cZxid = 0x200000002
world
...</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The value "world" that you put into znode "/hello" is available on every server in the ZooKeeper ensemble.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#use-kubectl-exec-zookeeper-i--zkclish-to-check-every-node-has-world" id="user-content-use-kubectl-exec-zookeeper-i--zkclish-to-check-every-node-has-world" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Use kubectl exec zookeeper-$i zkCli.sh to check every node has world</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-k" style="box-sizing: border-box; color: #d73a49;">for</span> <span class="pl-smi" style="box-sizing: border-box;">i</span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">in</span> 0 1 2<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">;</span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">do</span> kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> zookeeper-<span class="pl-smi" style="box-sizing: border-box;">$i</span> zkCli.sh get /hello <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">|</span> grep world<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">;</span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">done</span></pre>
</div>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#tolerating-node-failure" id="user-content-tolerating-node-failure" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Tolerating Node Failure</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Let's show how the consensus algorithm works. You will delete one server. Then you will try to write the ZooKeeper ensemble. Since two ZooKeeper servers still exist the write to the znode will work.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#delete-server-set-value-read-it-back-----should-work" id="user-content-delete-server-set-value-read-it-back-----should-work" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Delete server, set value, read it back, --- should work.</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl delete --force=true --grace-period=0 pod zookeeper-2 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">&</span>
sleep 1<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">;</span> kubectl delete --force=true --grace-period=0 pod zookeeper-2 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">&</span>
sleep 1
kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> zookeeper-0 zkCli.sh <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">set</span> /hello world_should_work
sleep 1
kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> zookeeper-1 zkCli.sh get /hello
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The write works because the ZooKeeper ensemble has a quorum. Now if you delete two servers, you won't have a quorum. Note that you delete the servers twice and you force their deletion. Run the following commands to get a feel for how ZooKeeper works.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#delete-two-server-set-value-read-it-back-----should-not-work" id="user-content-delete-two-server-set-value-read-it-back-----should-not-work" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Delete two server, set value, read it back, --- should not work.</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl delete --force=true --grace-period=0 pod zookeeper-2 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">&</span>
kubectl delete --force=true --grace-period=0 pod zookeeper-1 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">&</span>
sleep 1<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">;</span> kubectl delete --force=true --grace-period=0 pod zookeeper-2 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">&</span>
sleep 1<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">;</span> kubectl delete --force=true --grace-period=0 pod zookeeper-1 <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">&</span>
sleep 1
kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> zookeeper-0 zkCli.sh <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">set</span> /hello world_should_not_work
sleep 1
kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> zookeeper-0 zkCli.sh get /hello
sleep 20 <span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span> If you are running manually use kubectl get pods to see status of pods restarting</span>
kubectl <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">exec</span> zookeeper-0 zkCli.sh get /hello</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The value will be the same value you wrote the last time. The ZooKeeper ensemble needs a quorum before it can write to a znode.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
For more background information on this see <a href="https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/#managing-the-zookeeper-process" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Tolerating Node Failure</a>. The two tutorials cover this concept in a completely different but complementary way.</div>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#providing-durable-storage" id="user-content-providing-durable-storage" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Providing Durable Storage</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
This section is very loosely derived from <a href="https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/#sanity-testing-the-ensemble" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Providing Durable Storage</a> tutorial. ZooKeeper stores its data in the Kubernetes volumes using the generated persistent volume claims that you specified in the YAML manifest.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#zookeeperyaml-volume-template-in-statefulsetvolumeclaimtemplates" id="user-content-zookeeperyaml-volume-template-in-statefulsetvolumeclaimtemplates" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>zookeeper.yaml volume template in StatefulSet..volumeClaimTemplates</h4>
<div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-ent" style="box-sizing: border-box; color: #22863a;">apiVersion</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">apps/v1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">kind</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">StatefulSet</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">selector</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">matchLabels</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">app</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">serviceName</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper-headless</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">replicas</span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">3</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">updateStrategy</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">type</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">RollingUpdate</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">podManagementPolicy</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">OrderedReady</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">template</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">labels</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">app</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">zookeeper</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">...</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">... </span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">volumeMounts</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">datadir</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">mountPath</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">/var/lib/zookeeper</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">volumeClaimTemplates</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">metadata</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">datadir</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">spec</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">accessModes</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">[ "ReadWriteOnce" ]</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">resources</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">requests</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">storage</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">10Gi</span></pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Notice that the volumeClaimTemplates will create persistent volume claims for the pods. If you shut down the whole ZooKeeper <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">statefulset</code>, the volumes will not get deleted. And if you recreate the ZooKeeper <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">statefuleset</code>, the same value will be present. Let's do it.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#delete-the-statefulset" id="user-content-delete-the-statefulset" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Delete the statefulset</h4>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">kubectl delete statefulset zookeeper
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
statefulset.apps <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>zookeeper<span class="pl-pds" style="box-sizing: border-box;">"</span></span> delete</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Use <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kubectl get pods</code> to ensure that all of the pods terminate. Then recreate the Zookeeper <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">statefulset</code>.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#recreate-the-the-zookeeper-statefulset-using-zookeeperyaml" id="user-content-recreate-the-the-zookeeper-statefulset-using-zookeeperyaml" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Recreate the the ZooKeeper statefulset using zookeeper.yaml</h4>
<div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-s" style="box-sizing: border-box; color: #032f62;">kubectl apply -f zookeeper.yaml</span></pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Run <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">kubectl get pods -w -l app=zookeeper</code> and hit ctrl-c once all of the pods are back online.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Now let's see if our data is still there.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#read-data-from-new-zookeeper-ensemble-using-old-volumes" id="user-content-read-data-from-new-zookeeper-ensemble-using-old-volumes" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Read data from new ZooKeeper ensemble using old volumes</h4>
<div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-s" style="box-sizing: border-box; color: #032f62;">kubectl exec zookeeper-0 zkCli.sh get /hello</span>
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"><span class="pl-c" style="box-sizing: border-box;">#</span>## OUTPUT</span>
...
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">WATCHER:</span>:
...
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">world_should_work</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">cZxid = 0x200000002</span>
<span class="pl-s" style="box-sizing: border-box; color: #032f62;">ctime = Wed Feb 12 01:34:08 GMT 2020</span>
...
</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
It works. Even though you deleted the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">statefulset</code> and all of its pods, the ensemble still has the last value you set because the volumes are still there.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
The <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">volumeClaimTemplates</code> of the ZooKeeper StatefulSet’s spec specified a PersistentVolume for each pod.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
When a pod in the ZooKeeper's StatefulSet is rescheduled or upgraded, it will have the PersistentVolume mounted to the ZooKeeper server's data directory. The persistent data will still be there. This same concept would work with Cassandra or Consul or etcd or any database.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#configuring-a-non-privileged-user" id="user-content-configuring-a-non-privileged-user" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Configuring a Non-Privileged User</h4>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
See <a href="https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/#configuring-a-non-privileged-user" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Configuring a Non-Privileged User</a> from the original tutorial.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#managing-the-zookeeper-process" id="user-content-managing-the-zookeeper-process" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Managing the ZooKeeper Process</h4>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
See <a href="https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/#managing-the-zookeeper-process" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Managing the ZooKeeper Process</a> from the original tutorial.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#using-minikube-dashboard" id="user-content-using-minikube-dashboard" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Using MiniKube Dashboard</h4>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
You can see the pods and the statefuleset with minikube dashboard.</div>
<div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">minikube dashboard
</pre>
</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#kubernetes-dashboard" id="user-content-kubernetes-dashboard" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Kubernetes Dashboard</h4>
<div>
<img height="420" src="https://github.com/cloudurable/kube-zookeeper-statefulsets/raw/master/images/kubernetes-dashboard.png" width="640" /></div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<img alt="" src="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/images/kubernetes-dashboard.png" style="border-style: none; box-sizing: initial; max-width: 100%;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Notice the pods are healthy.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
Navigate down to <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">Workloads->Stateful</code> Sets then click on <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">zookeeper</code> in the list to see the ZooKeeper Stateful Set.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#kubernetes-dashboard-showing-zookeeper-stateful-set" id="user-content-kubernetes-dashboard-showing-zookeeper-stateful-set" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Kubernetes Dashboard showing ZooKeeper Stateful Set</h4>
<div>
<img height="422" src="https://github.com/cloudurable/kube-zookeeper-statefulsets/raw/master/images/dashboard-zookeeper-statefulset.png" width="640" /></div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;">
<img alt="" src="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/images/dashboard-zookeeper-statefulset.png" style="border-style: none; box-sizing: initial; max-width: 100%;" /></div>
<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/kube-zookeeper-statefulsets/wiki/Tutorial-Part-1:--Managing-Kubernetes-StatefulSets-using-ZooKeeper-and-Minikube#we-did-it" id="user-content-we-did-it" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>We did it</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px;">
We did it. The skills we used to analyze ZooKeeper will be available for other clustered software. The biggest part to me is learning how to debug with something that goes wrong. But our example still does not run in OpenShift. In the next tutorial, we will get it running in OpenShift.</div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com0tag:blogger.com,1999:blog-6128525281101058933.post-52528776367975949682017-05-19T15:09:00.005-07:002017-05-19T15:10:26.869-07:00Writing Kafka Java Producers and Kafka Java Consumers <h3 id="kafka-tutorial-writing-a-kafka-producer-in-java" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Kafka Tutorial: Writing a Kafka Producer in Java</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
In this tutorial, we are going to create simple Java example that creates a Kafka producer. You create a new replicated Kafka topic called <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">my-example-topic</code>, then you create a Kafka producer that uses this topic to send records. You will send records with the Kafka producer. You will send records synchronously. Later, you will send records asynchronously.</div>
<h3 id="before-you-start" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Before you start</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Prerequisites to this tutorial are <a href="http://cloudurable.com/blog/kafka-tutorial-kafka-from-command-line/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka from the command line</a> and <a href="http://cloudurable.com/blog/kafka-tutorial-kafka-failover-kafka-cluster/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka clustering and failover basics</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
This tutorial is part of a series. If you are not sure what Kafka is, you should start with <a href="http://cloudurable.com/blog/what-is-kafka/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">What is Kafka?</a>. If you are unfamiliar with the architecture of Kafka then we suggest reading <a href="http://cloudurable.com/blog/kafka-architecture/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka Architecture</a>, <a href="http://cloudurable.com/blog/kafka-architecture-topics/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka Topics Architecture</a>, <a href="http://cloudurable.com/blog/kafka-architecture-producers/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka Producer Architecture</a> and <a href="http://cloudurable.com/blog/kafka-architecture-consumers/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka Consumer Architecture</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="create-replicated-kafka-topic" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Create Replicated Kafka Topic</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Next, you need to create a replicated topic.</div>
<h4 id="kafka-training-lab3-create-topic-sh" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab3/create-topic.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/usr/bin/env bash
cd ~/kafka-training
## Create topics
kafka/bin/kafka-topics.sh --create \
--replication-factor 3 \
--partitions 13 \
--topic my-example-topic \
--zookeeper localhost:2181
## List created topics
kafka/bin/kafka-topics.sh --list \
--zookeeper localhost:2181
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Above we create a topic named <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">my-example-topic</code> with 13 partitions and a replication factor of 3. Then we list the Kafka topics.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Runs <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">create-topic.sh</code> as follows.</div>
<h4 id="output-from-running-create-topic-sh" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Output from running create-topic.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab3
$ ./create-topic.sh
Created topic "my-example-topic".
__consumer_offsets
my-example-topic
my-failsafe-topic
</code></pre>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="gradle-build-script" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Gradle Build Script</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
For this example, we use gradle to build the project.</div>
<h4 id="kafka-training-lab3-solution-build-gradle" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab3/solution/build.gradle</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">group 'cloudurable-kafka'
version '1.0-SNAPSHOT'
apply plugin: 'java'
sourceCompatibility = 1.8
repositories {
mavenCentral()
}
dependencies {
compile 'org.apache.kafka:kafka-clients:0.10.2.0'
compile 'ch.qos.logback:logback-classic:1.2.2'
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that we import the jar file <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">kafka-clients:0.10.2.0</code>. Apache Kafka uses <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">sl4j</code> so to setup logging we use logback (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ch.qos.logback:logback-classic:1.2.2</code>).</div>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="construct-a-kafka-producer" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Construct a Kafka Producer</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
To create a Kafka producer, you will need to pass it a list of bootstrap servers (a list of Kafka brokers). You will also specify a <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">client.id</code> that uniquely identifies this Producer client. In this example, we are going to send messages with ids. The message body is a string, so we need a record value serializer as we will send the message body in the Kafka’s records value field. The message id (long), will be sent as the Kafka’s records key. You will need to specify a Key serializer and a value serializer, which Kafka will use to encode the message id as a Kafka record key, and the message body as the Kafka record value.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<a href="http://cloudurable.com/kafka-training/index.html" style="box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka training</a>, <a href="http://cloudurable.com/kafka-aws-consulting/index.html" style="box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka consulting</a>.</div>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="common-kafka-imports-and-constants" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Common Kafka imports and constants</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Next, we will import the Kafka packages and define a constant for the topic and a constant to define the list of bootstrap servers that the producer will connect.</div>
<h4 id="kafkaproducerexample-java-imports-and-constants" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
KafkaProducerExample.java - imports and constants</h4>
<h4 id="kafka-training-lab3-src-main-java-com-cloudurable-kafka-kafkaproducerexample-java" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab3/src/main/java/com/cloudurable/kafka/KafkaProducerExample.java</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">
package com.cloudurable.kafka;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.LongSerializer;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
public class KafkaProducerExample {
private final static String TOPIC = "my-example-topic";
private final static String BOOTSTRAP_SERVERS =
"localhost:9092,localhost:9093,localhost:9094";
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">KafkaProducerExample</code> imports <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">LongSerializer</code> which gets configured as the Kafka record key serializer, and imports <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">StringSerializer</code> which gets configured as the record value serializer. The constant <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">BOOTSTRAP_SERVERS</code> is set to<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">localhost:9092,localhost:9093,localhost:9094</code> which is the three Kafka servers that we started up in the last lesson. Go ahead and make sure all three Kafka servers are running. The constant <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">TOPIC</code> is set to the replicated Kafka topic that we just created.</div>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="create-kafka-producer-to-send-records" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Create Kafka Producer to send records</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now, that we imported the Kafka classes and defined some constants, let’s create a Kafka producer.</div>
<h4 id="kafkaproducerexample-java-create-producer-to-send-records" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
KafkaProducerExample.java - Create Producer to send Records</h4>
<h4 id="kafka-training-lab3-src-main-java-com-cloudurable-kafka-kafkaproducerexample-java-1" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab3/src/main/java/com/cloudurable/kafka/KafkaProducerExample.java</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">public class KafkaProducerExample {
...
private static Producer<Long, String> createProducer() {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
BOOTSTRAP_SERVERS);
props.put(ProducerConfig.CLIENT_ID_CONFIG, "KafkaExampleProducer");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
LongSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
StringSerializer.class.getName());
return new KafkaProducer<>(props);
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
To create a Kafka producer, you use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">java.util.Properties</code> and define certain properties that we pass to the constructor of a <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">KafkaProducer</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Above <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">KafkaProducerExample.createProducer</code> sets the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">BOOTSTRAP_SERVERS_CONFIG</code> (“bootstrap.servers) property to the list of broker addresses we defined earlier. <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">BOOTSTRAP_SERVERS_CONFIG</code> value is a comma separated list of host/port pairs that the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">Producer</code> uses to establish an initial connection to the Kafka cluster. The producer uses of all servers in the cluster no matter which ones we list here. This list only specifies the initial Kafka brokers used to discover the full set of servers of the Kafka cluster. If a server in this list is down, the producer will just go to the next broker in the list to discover the full topology of the Kafka cluster.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">CLIENT_ID_CONFIG</code> (“client.id”) is an id to pass to the server when making requests so the server can track the source of requests beyond just IP/port by passing a producer name for things like server-side request logging.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">KEY_SERIALIZER_CLASS_CONFIG</code> (“key.serializer”) is a Kafka Serializer class for Kafka record keys that implements the Kafka Serializer interface. Notice that we set this to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">LongSerializer</code> as the message ids in our example are longs.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">VALUE_SERIALIZER_CLASS_CONFIG</code> (“value.serializer”) is a Kafka Serializer class for Kafka record values that implements the Kafka Serializer interface. Notice that we set this to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">StringSerializer</code> as the message body in our example are strings.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
(<a href="http://cloudurable.com/kafka-training/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka training</a>, <a href="http://cloudurable.com/kafka-aws-consulting/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka consulting</a>.)</div>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="send-records-synchronously-with-kafka-producer" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Send records synchronously with Kafka Producer</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Kafka provides a synchronous send method to send a record to a topic. Let’s use this method to send some message ids and messages to the Kafka topic we created earlier.</div>
<h4 id="kafkaproducerexample-java-send-records-synchronously" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
KafkaProducerExample.java - Send Records Synchronously</h4>
<h4 id="kafka-training-lab3-src-main-java-com-cloudurable-kafka-kafkaproducerexample-java-2" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab3/src/main/java/com/cloudurable/kafka/KafkaProducerExample.java</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">public class KafkaProducerExample {
...
static void runProducer(final int sendMessageCount) throws Exception {
final Producer<Long, String> producer = createProducer();
long time = System.currentTimeMillis();
try {
for (long index = time; index < time + sendMessageCount; index++) {
final ProducerRecord<Long, String> record =
new ProducerRecord<>(TOPIC, index,
"Hello Mom " + index);
RecordMetadata metadata = producer.send(record).get();
long elapsedTime = System.currentTimeMillis() - time;
System.out.printf("sent record(key=%s value=%s) " +
"meta(partition=%d, offset=%d) time=%d\n",
record.key(), record.value(), metadata.partition(),
metadata.offset(), elapsedTime);
}
} finally {
producer.flush();
producer.close();
}
}
...
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The above just iterates through a for loop, creating a <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ProducerRecord</code> sending an example message (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">"Hello Mom " + index</code>) as the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">record value</code> and the for loop <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">index</code> as the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">record key</code>. For each iteration, <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">runProducer</code> calls the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">send</code> method of the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">producer</code> (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">RecordMetadata metadata = producer.send(record).get()</code>). The send method returns a Java <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">Future</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The response RecordMetadata has ‘partition’ where the record was written and the ‘offset’ of the record in that partition.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice the call to flush and close. Kafka will auto flush on its own, but you can also call flush explicitly which will send the accumulated records now. It is polite to close the connection when we are done.</div>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="running-the-kafka-producer" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Running the Kafka Producer</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Next you define the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">main</code> method.</div>
<h4 id="kafkaproducerexample-java-running-the-producer" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
KafkaProducerExample.java - Running the Producer</h4>
<h4 id="kafka-training-lab3-src-main-java-com-cloudurable-kafka-kafkaproducerexample-java-3" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab3/src/main/java/com/cloudurable/kafka/KafkaProducerExample.java</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">public static void main(String... args) throws Exception {
if (args.length == 0) {
runProducer(5);
} else {
runProducer(Integer.parseInt(args[0]));
}
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">main</code> method just calls <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">runProducer</code>.</div>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="send-records-asynchronously-with-kafka-producer" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Send records asynchronously with Kafka Producer</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Kafka provides an asynchronous send method to send a record to a topic. Let’s use this method to send some message ids and messages to the Kafka topic we created earlier. The big difference here will be that we use a lambda expression to define a callback.</div>
<h4 id="kafkaproducerexample-java-send-records-asynchronously-with-kafka-producer" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
KafkaProducerExample.java - Send Records Asynchronously with Kafka Producer</h4>
<h4 id="kafka-training-lab3-src-main-java-com-cloudurable-kafka-kafkaproducerexample-java-4" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab3/src/main/java/com/cloudurable/kafka/KafkaProducerExample.java</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">static void runProducer(final int sendMessageCount) throws InterruptedException {
final Producer<Long, String> producer = createProducer();
long time = System.currentTimeMillis();
final CountDownLatch countDownLatch = new CountDownLatch(sendMessageCount);
try {
for (long index = time; index < time + sendMessageCount; index++) {
final ProducerRecord<Long, String> record =
new ProducerRecord<>(TOPIC, index, "Hello Mom " + index);
producer.send(record, (metadata, exception) -> {
long elapsedTime = System.currentTimeMillis() - time;
if (metadata != null) {
System.out.printf("sent record(key=%s value=%s) " +
"meta(partition=%d, offset=%d) time=%d\n",
record.key(), record.value(), metadata.partition(),
metadata.offset(), elapsedTime);
} else {
exception.printStackTrace();
}
countDownLatch.countDown();
});
}
countDownLatch.await(25, TimeUnit.SECONDS);
}finally {
producer.flush();
producer.close();
}
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice the use of a CountDownLatch so we can send all N messages and then wait for them all to send.</div>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="async-interface-callback-and-async-send-method" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Async Interface Callback and Async Send Method</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Kafka defines a <a href="https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/producer/Callback.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Callback</a> interface that you use for asynchronous operations. The callback interface allows code to execute when the request is complete. The callback executes in a background I/O thread so it should be fast (don’t block it). The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">onCompletion(RecordMetadata metadata, Exception exception)</code> gets called when the asynchronous operation completes. The <a href="https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/producer/RecordMetadata.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;"><code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">metadata</code></a> gets set (not null) if the operation was a success, and the exception gets set (not null) if the operation had an error.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The async <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">send</code> method is used to send a record to a topic, and the provided callback gets called when the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">send</code> is acknowledged. The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">send</code> method is asynchronous, and when called returns immediately once the record gets stored in the buffer of records waiting to post to the Kafka broker. The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">send</code> method allows sending many records in parallel without blocking to wait for the response after each one.</div>
<blockquote style="background-color: white; border-left: 5px solid rgb(0, 191, 255); box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin: 0px 0px 20px; padding: 10px 20px;">
<div style="box-sizing: border-box;">
Since the send call is asynchronous it returns a Future for the RecordMetadata that will be assigned to this record. Invoking get() on this future will block until the associated request completes and then return the metadata for the record or throw any exception that occurred while sending the record. <a href="https://kafka.apache.org/0100/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">KafkaProducer</a></div>
</blockquote>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="conclusion-kafka-producer-example" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Conclusion Kafka Producer example</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We created a simple example that creates a Kafka Producer. First, we created a new replicated Kafka topic; then we created Kafka Producer in Java that uses the Kafka replicated topic to send records. We sent records with the Kafka Producer using async and sync send methods.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We hope you enjoyed this article. Please provide <a href="http://cloudurable.com/contact/index.html" style="box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">feedback</a>. See the <a href="http://cloudurable.com/kafka-training/index.html" style="box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka training</a>, <a href="http://cloudurable.com/kafka-aws-consulting/index.html" style="box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka consulting</a>, <a href="http://cloudurable.com/subscription_support/index.html" style="box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka support</a> and helps <a href="http://cloudurable.com/services/index.html" style="box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">setting up Kafka clusters in AWS</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
(<a href="http://cloudurable.com/kafka-training/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka training</a>, <a href="http://cloudurable.com/kafka-aws-consulting/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka consulting</a>.)</div>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="review-kafka-producer" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Review Kafka Producer</h3>
<h4 id="what-does-the-callback-lambda-do" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
What does the Callback lambda do?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The callback gets notified when the request is complete.</div>
<h4 id="what-will-happen-if-the-first-server-is-down-in-the-bootstrap-list-can-the-producer-still-connect-to-the-other-kafka-brokers-in-the-cluster" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
What will happen if the first server is down in the bootstrap list? Can the producer still connect to the other Kafka brokers in the cluster?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The producer will try to contact the next broker in the list. Any of the brokers once contacted, will let the producer know about the entire Kafka cluster. The Producer will connect as long as at least one of the brokers in the list is running. If you have 100 brokers and two of the brokers in a list of three servers in the bootstrap list are down, the producer can still use the 98 remaining brokers.</div>
<h4 id="when-would-you-use-kafka-async-send-vs-sync-send" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
When would you use Kafka async send vs. sync send?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
If you were already using an async code (Akka, QBit, Reakt, Vert.x) base, and you wanted to send records quickly.</div>
<h4 id="why-do-you-need-two-serializers-for-a-kafka-record" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Why do you need two serializers for a Kafka record?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
One of the serializers is for the Kafka record key, and the other is for the Kafka record value.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
<h3 id="kafka-tutorial-writing-a-kafka-consumer-in-java" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Kafka Tutorial: Writing a Kafka Consumer in Java</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
In this tutorial, you are going to create simple <em style="box-sizing: border-box;">Kafka Consumer</em>. This consumer consumes messages from the Kafka Producer you wrote in the last tutorial. This tutorial demonstrates how to process records from a <em style="box-sizing: border-box;">Kafka topic</em> with a <em style="box-sizing: border-box;">Kafka Consumer</em>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
This tutorial describes how <em style="box-sizing: border-box;">Kafka Consumers</em> in the same group divide up and share partitions while each <em style="box-sizing: border-box;">consumer group</em> appears to get its own copy of the same data.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
<h3 id="construct-a-kafka-consumer" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Construct a Kafka Consumer</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Just like we did with the producer, you need to specify bootstrap servers. You also need to define a group.id that identifies which consumer group this consumer belongs. Then you need to designate a Kafka record key deserializer and a record value deserializer. Then you need to subscribe the consumer to the topic you created in the producer tutorial.</div>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="kafka-consumer-imports-and-constants" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Kafka Consumer imports and constants</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Next, you import the Kafka packages and define a constant for the topic and a constant to set the list of bootstrap servers that the consumer will connect.</div>
<h4 id="kafkaconsumerexample-java-imports-and-constants" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
KafkaConsumerExample.java - imports and constants</h4>
<h4 id="kafka-training-lab4-src-main-java-com-cloudurable-kafka-kafkaconsumerexample-java" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab4/src/main/java/com/cloudurable/kafka/KafkaConsumerExample.java</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">
package com.cloudurable.kafka;
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.common.serialization.LongDeserializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Collections;
import java.util.Properties;
public class KafkaConsumerExample {
private final static String TOPIC = "my-example-topic";
private final static String BOOTSTRAP_SERVERS =
"localhost:9092,localhost:9093,localhost:9094";
...
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">KafkaConsumerExample</code> imports <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">LongDeserializer</code> which gets configured as the Kafka record key deserializer, and imports <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">StringDeserializer</code> which gets set up as the record value deserializer. The constant <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">BOOTSTRAP_SERVERS</code> gets set to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">localhost:9092,localhost:9093,localhost:9094</code> which is the three Kafka servers that we started up in the last lesson. Go ahead and make sure all three Kafka servers are running. The constant <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">TOPIC</code> gets set to the replicated Kafka topic that you created in the last tutorial.</div>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="create-kafka-consumer-using-topic-to-receive-records" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Create Kafka Consumer using Topic to Receive Records</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now, that you imported the Kafka classes and defined some constants, let’s create the Kafka consumer.</div>
<h4 id="kafkaconsumerexample-java-create-consumer-to-process-records" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
KafkaConsumerExample.java - Create Consumer to process Records</h4>
<h4 id="kafka-training-lab4-src-main-java-com-cloudurable-kafka-kafkaconsumerexample-java-1" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab4/src/main/java/com/cloudurable/kafka/KafkaConsumerExample.java</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">
public class KafkaConsumerExample {
...
private static Consumer<Long, String> createConsumer() {
final Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
BOOTSTRAP_SERVERS);
props.put(ConsumerConfig.GROUP_ID_CONFIG,
"KafkaExampleConsumer");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
LongDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class.getName());
// Create the consumer using props.
final Consumer<Long, String> consumer =
new KafkaConsumer<>(props);
// Subscribe to the topic.
consumer.subscribe(Collections.singletonList(TOPIC));
return consumer;
}
...
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
To create a Kafka consumer, you use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">java.util.Properties</code> and define certain properties that we pass to the constructor of a <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">KafkaConsumer</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Above <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">KafkaConsumerExample.createConsumer</code> sets the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">BOOTSTRAP_SERVERS_CONFIG</code> (“bootstrap.servers”) property to the list of broker addresses we defined earlier. <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">BOOTSTRAP_SERVERS_CONFIG</code> value is a comma separated list of host/port pairs that the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">Consumer</code> uses to establish an initial connection to the Kafka cluster. Just like the producer, the consumer uses of all servers in the cluster no matter which ones we list here.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">GROUP_ID_CONFIG</code> identifies the consumer group of this consumer.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">KEY_DESERIALIZER_CLASS_CONFIG</code> (“key.deserializer”) is a Kafka Deserializer class for Kafka record keys that implements the Kafka Deserializer interface. Notice that we set this to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">LongDeserializer</code> as the message ids in our example are longs.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">VALUE_DESERIALIZER_CLASS_CONFIG</code> (“value.deserializer”) is a Kafka Serializer class for Kafka record values that implements the Kafka Deserializer interface. Notice that we set this to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">StringDeserializer</code> as the message body in our example are strings.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Important notice that you need to subscribe the consumer to the topic <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">consumer.subscribe(Collections.singletonList(TOPIC));</code>. The subscribe method takes a list of topics to subscribe to, and this list will replace the current subscriptions if any.</div>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="process-messages-from-kafka-with-consumer" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Process messages from Kafka with Consumer</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now, let’s process some records with our Kafka Producer.</div>
<h4 id="kafkaconsumerexample-java-process-records-from-consumer" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
KafkaConsumerExample.java - Process records from Consumer</h4>
<h4 id="kafka-training-lab4-src-main-java-com-cloudurable-kafka-kafkaconsumerexample-java-2" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab4/src/main/java/com/cloudurable/kafka/KafkaConsumerExample.java</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">public class KafkaConsumerExample {
...
static void runConsumer() throws InterruptedException {
final Consumer<Long, String> consumer = createConsumer();
final int giveUp = 100; int noRecordsCount = 0;
while (true) {
final ConsumerRecords<Long, String> consumerRecords =
consumer.poll(1000);
if (consumerRecords.count()==0) {
noRecordsCount++;
if (noRecordsCount > giveUp) break;
else continue;
}
consumerRecords.forEach(record -> {
System.out.printf("Consumer Record:(%d, %s, %d, %d)\n",
record.key(), record.value(),
record.partition(), record.offset());
});
consumer.commitAsync();
}
consumer.close();
System.out.println("DONE");
}
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice you use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ConsumerRecords</code> which is a group of records from a Kafka topic partition. The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ConsumerRecords</code> class is a container that holds a list of ConsumerRecord(s) per partition for a particular topic. There is one <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ConsumerRecord</code> list for every topic partition returned by a the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">consumer.poll()</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice if you receive records (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">consumerRecords.count()!=0</code>), then <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">runConsumer</code> method calls <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">consumer.commitAsync()</code> which commit offsets returned on the last call to consumer.poll(…) for all the subscribed list of topic partitions.</div>
<h3 id="kafka-consumer-poll-method" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Kafka Consumer Poll method</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The poll method returns fetched records based on current partition offset. The poll method is a blocking method waiting for specified time in seconds. If no records are available after the time period specified, the poll method returns an empty ConsumerRecords.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
When new records become available, the poll method returns straight away.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
You can can control the maximum records returned by the poll() with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 100);</code>. The poll method is not thread safe and is not meant to get called from multiple threads.</div>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="running-the-kafka-consumer" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Running the Kafka Consumer</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Next you define the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">main</code> method.</div>
<h4 id="kafkaconsumerexample-java-running-the-consumer" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
KafkaConsumerExample.java - Running the Consumer</h4>
<h4 id="kafka-training-lab4-src-main-java-com-cloudurable-kafka-kafkaconsumerexample-java-3" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab4/src/main/java/com/cloudurable/kafka/KafkaConsumerExample.java</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">
public class KafkaConsumerExample {
public static void main(String... args) throws Exception {
runConsumer();
}
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">main</code> method just calls <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">runConsumer</code>.</div>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="try-running-the-consumer-and-producer" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Try running the consumer and producer</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Run the consumer from your IDE. Then run the producer from the last tutorial from your IDE. You should see the consumer get the records that the producer sent.</div>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="logging-set-up-for-kafka" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Logging set up for Kafka</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
If you don’t set up logging well, it might be hard to see the consumer get the messages.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Kafka like most Java libs these days uses <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">sl4j</code>. You can use Kafka with Log4j, Logback or JDK logging. We used logback in our gradle build (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">compile 'ch.qos.logback:logback-classic:1.2.2'</code>).</div>
<h4 id="kafka-training-lab4-solution-src-main-resources-logback-xml" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab4/solution/src/main/resources/logback.xml</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-xml" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;"><configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<logger name="org.apache.kafka" level="INFO"/>
<logger name="org.apache.kafka.common.metrics" level="INFO"/>
<root level="debug">
<appender-ref ref="STDOUT" />
</root>
</configuration>
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that we set <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">org.apache.kafka</code> to INFO, otherwise we will get a lot of log messages. You should run it set to debug and read through the log messages. It gives you a flavor of what Kafka is doing under the covers. Leave <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">org.apache.kafka.common.metrics</code> or what Kafka is doing under the covers is drowned by metrics logging.</div>
<h3 id="try-this-three-consumers-in-same-group-and-one-producer-sending-25-messages" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Try this: Three Consumers in same group and one Producer sending 25 messages</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Run the consumer example three times from your IDE. Then change Producer to send 25 records instead of 5. Then run the producer once from your IDE. What happens? The consumers should share the messages.</div>
<h4 id="producer-output" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Producer Output</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">sent record(key=1495048417121 value=..) meta(partition=6, offset=16) time=118
sent record(key=1495048417131 value=..) meta(partition=6, offset=17) time=120
sent record(key=1495048417133 value=..) meta(partition=12, offset=17) time=120
sent record(key=1495048417140 value=..) meta(partition=12, offset=18) time=121
sent record(key=1495048417143 value=..) meta(partition=12, offset=19) time=121
sent record(key=1495048417123 value=..) meta(partition=0, offset=19) time=121
sent record(key=1495048417126 value=..) meta(partition=0, offset=20) time=121
sent record(key=1495048417134 value=..) meta(partition=0, offset=21) time=122
sent record(key=1495048417122 value=..) meta(partition=3, offset=19) time=122
sent record(key=1495048417127 value=..) meta(partition=3, offset=20) time=122
sent record(key=1495048417139 value=..) meta(partition=3, offset=21) time=123
sent record(key=1495048417142 value=..) meta(partition=3, offset=22) time=123
sent record(key=1495048417136 value=..) meta(partition=10, offset=19) time=127
sent record(key=1495048417144 value=..) meta(partition=1, offset=26) time=128
sent record(key=1495048417125 value=..) meta(partition=5, offset=22) time=128
sent record(key=1495048417138 value=..) meta(partition=5, offset=23) time=128
sent record(key=1495048417128 value=..) meta(partition=8, offset=21) time=129
sent record(key=1495048417124 value=..) meta(partition=11, offset=18) time=129
sent record(key=1495048417130 value=..) meta(partition=11, offset=19) time=129
sent record(key=1495048417132 value=..) meta(partition=11, offset=20) time=130
sent record(key=1495048417141 value=..) meta(partition=11, offset=21) time=130
sent record(key=1495048417145 value=..) meta(partition=11, offset=22) time=131
sent record(key=1495048417129 value=..) meta(partition=2, offset=24) time=132
sent record(key=1495048417135 value=..) meta(partition=2, offset=25) time=132
sent record(key=1495048417137 value=..) meta(partition=2, offset=26) time=132
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice the producer sends 25 messages.</div>
<h4 id="consumer-0-in-same-group" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Consumer 0 in same group</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">Consumer Record:(1495048417121, Hello Mom 1495048417121, 6, 16)
Consumer Record:(1495048417131, Hello Mom 1495048417131, 6, 17)
Consumer Record:(1495048417125, Hello Mom 1495048417125, 5, 22)
Consumer Record:(1495048417138, Hello Mom 1495048417138, 5, 23)
Consumer Record:(1495048417128, Hello Mom 1495048417128, 8, 21)
</code></pre>
<h4 id="consumer-1-in-same-group" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Consumer 1 in same group</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">Consumer Record:(1495048417123, Hello Mom 1495048417123, 0, 19)
Consumer Record:(1495048417126, Hello Mom 1495048417126, 0, 20)
Consumer Record:(1495048417134, Hello Mom 1495048417134, 0, 21)
Consumer Record:(1495048417144, Hello Mom 1495048417144, 1, 26)
Consumer Record:(1495048417122, Hello Mom 1495048417122, 3, 19)
Consumer Record:(1495048417127, Hello Mom 1495048417127, 3, 20)
Consumer Record:(1495048417139, Hello Mom 1495048417139, 3, 21)
Consumer Record:(1495048417142, Hello Mom 1495048417142, 3, 22)
Consumer Record:(1495048417129, Hello Mom 1495048417129, 2, 24)
Consumer Record:(1495048417135, Hello Mom 1495048417135, 2, 25)
Consumer Record:(1495048417137, Hello Mom 1495048417137, 2, 26)
</code></pre>
<h4 id="consumer-2-in-same-group" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Consumer 2 in same group</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">Consumer Record:(1495048417136, Hello Mom 1495048417136, 10, 19)
Consumer Record:(1495048417133, Hello Mom 1495048417133, 12, 17)
Consumer Record:(1495048417140, Hello Mom 1495048417140, 12, 18)
Consumer Record:(1495048417143, Hello Mom 1495048417143, 12, 19)
Consumer Record:(1495048417124, Hello Mom 1495048417124, 11, 18)
Consumer Record:(1495048417130, Hello Mom 1495048417130, 11, 19)
Consumer Record:(1495048417132, Hello Mom 1495048417132, 11, 20)
Consumer Record:(1495048417141, Hello Mom 1495048417141, 11, 21)
Consumer Record:(1495048417145, Hello Mom 1495048417145, 11, 22)
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Can you answer these questions?</div>
<h4 id="which-consumer-owns-partition-10" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Which consumer owns partition 10?</h4>
<h4 id="how-many-consumerrecords-objects-did-consumer-0-get" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
How many ConsumerRecords objects did Consumer 0 get?</h4>
<h4 id="what-is-the-next-offset-from-partition-11-that-consumer-2-should-get" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
What is the next offset from Partition 11 that Consumer 2 should get?</h4>
<h4 id="why-does-each-consumer-get-unique-messages" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Why does each consumer get unique messages?</h4>
<h4 id="which-consumer-owns-partition-10-1" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Which consumer owns partition 10?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Consumer 2 owns partition 10.</div>
<h4 id="how-many-consumerrecords-objects-did-consumer-0-get-1" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
How many ConsumerRecords objects did Consumer 0 get?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
3</div>
<h4 id="what-is-the-next-offset-from-partition-11-that-consumer-2-should-get-1" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
What is the next offset from Partition 11 that Consumer 2 should get?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
22</div>
<h4 id="why-does-each-consumer-get-unique-messages-1" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Why does each consumer get unique messages?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Each gets its share of partitions for the topic.</div>
<h3 id="try-this-three-consumers-in-different-consumer-group-and-one-producer-sending-5-messages" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Try this: Three Consumers in different Consumer group and one Producer sending 5 messages</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Modify the consumer, so each consumer processes will have a unique group id.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Stop all consumers and producers processes from the last run.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Then execute the consumer example three times from your IDE. Then change producer to send five records instead of 25. Then run the producer once from your IDE. What happens? The consumers should each get a copy of the messages.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
First, let’s modify the Consumer to make their group id unique as follows:</div>
<h4 id="kafkaconsumerexample-make-the-consumer-group-id-unique" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
KafkaConsumerExample - Make the Consumer group id unique</h4>
<h4 id="kafka-training-lab4-src-main-java-com-cloudurable-kafka-kafkaconsumerexample-java-4" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab4/src/main/java/com/cloudurable/kafka/KafkaConsumerExample.java</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">public class KafkaConsumerExample {
private final static String TOPIC = "my-example-topic";
private final static String BOOTSTRAP_SERVERS =
"localhost:9092,localhost:9093,localhost:9094";
private static Consumer<Long, String> createConsumer() {
final Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
BOOTSTRAP_SERVERS);
props.put(ConsumerConfig.GROUP_ID_CONFIG,
"KafkaExampleConsumer" +
System.currentTimeMillis());
...
}
...
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice, to make the group id unique you just add <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">System.currentTimeMillis()</code> to it.</div>
<h4 id="producer-output-1" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Producer Output</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">sent record(key=1495049585396 value=..) meta(partition=7, offset=30) time=134
sent record(key=1495049585392 value=..) meta(partition=4, offset=24) time=138
sent record(key=1495049585393 value=..) meta(partition=4, offset=25) time=139
sent record(key=1495049585395 value=..) meta(partition=4, offset=26) time=139
sent record(key=1495049585394 value=..) meta(partition=11, offset=25) time=140
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice the producer sends 25 messages.</div>
<h4 id="consumer-0-in-own-group" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Consumer 0 in own group</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">Consumer Record:(1495049585396, Hello Mom 1495049585396, 7, 30)
Consumer Record:(1495049585394, Hello Mom 1495049585394, 11, 25)
Consumer Record:(1495049585392, Hello Mom 1495049585392, 4, 24)
Consumer Record:(1495049585393, Hello Mom 1495049585393, 4, 25)
Consumer Record:(1495049585395, Hello Mom 1495049585395, 4, 26)
</code></pre>
<h4 id="consumer-1-in-unique-consumer-group" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Consumer 1 in unique consumer group</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">Consumer Record:(1495049585396, Hello Mom 1495049585396, 7, 30)
Consumer Record:(1495049585394, Hello Mom 1495049585394, 11, 25)
Consumer Record:(1495049585392, Hello Mom 1495049585392, 4, 24)
Consumer Record:(1495049585393, Hello Mom 1495049585393, 4, 25)
Consumer Record:(1495049585395, Hello Mom 1495049585395, 4, 26)
</code></pre>
<h4 id="consumer-2-in-its-own-consumer-group" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Consumer 2 in its own consumer group</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">Consumer Record:(1495049585396, Hello Mom 1495049585396, 7, 30)
Consumer Record:(1495049585394, Hello Mom 1495049585394, 11, 25)
Consumer Record:(1495049585392, Hello Mom 1495049585392, 4, 24)
Consumer Record:(1495049585393, Hello Mom 1495049585393, 4, 25)
Consumer Record:(1495049585395, Hello Mom 1495049585395, 4, 26)
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Can you answer these questions?</div>
<h4 id="which-consumer-owns-partition-10-2" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Which consumer owns partition 10?</h4>
<h4 id="how-many-consumerrecords-objects-did-consumer-0-get-2" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
How many ConsumerRecords objects did Consumer 0 get?</h4>
<h4 id="what-is-the-next-offset-from-partition-11-that-consumer-2-should-get-2" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
What is the next offset from Partition 11 that Consumer 2 should get?</h4>
<h4 id="why-does-each-consumer-get-unique-messages-2" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Why does each consumer get unique messages?</h4>
<h4 id="which-consumer-owns-partition-10-3" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Which consumer owns partition 10?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
They all do! Since they are all in a unique consumer group, and there is only one consumer in each group, then each consumer we ran owns all of the partitions.</div>
<h4 id="how-many-consumerrecords-objects-did-consumer-0-get-3" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
How many ConsumerRecords objects did Consumer 0 get?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
3</div>
<h4 id="what-is-the-next-offset-from-partition-11-that-consumer-2-should-get-3" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
What is the next offset from Partition 11 that Consumer 2 should get?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
26</div>
<h4 id="why-does-each-consumer-get-the-same-messages" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Why does each consumer get the same messages?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
They do because they are each in their own consumer group, and each consumer group is a subscription to the topic.</div>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="conclusion-kafka-consumer-example" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Conclusion Kafka Consumer example</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
You created a simple example that creates a <em style="box-sizing: border-box;">Kafka consumer</em> to consume messages from the Kafka Producer you created in the last tutorial. We used the replicated Kafka topic from producer lab. You created a <em style="box-sizing: border-box;">Kafka Consumer</em> that uses the topic to receive messages. The <em style="box-sizing: border-box;">Kafka consumer</em> uses the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">poll</code> method to get N number of records.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<em style="box-sizing: border-box;">Consumers</em> in the same group divide up and share partitions as we demonstrated by running three consumers in the same group and one producer. Each consumer groups gets a copy of the same data. More precise, each consumer group really has a unique set of offset/partition pairs per.</div>
<hr style="background-color: white; border-bottom: 0px; border-image: initial; border-left: 0px; border-right: 0px; border-top-color: rgb(238, 238, 238); border-top-style: solid; box-sizing: content-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; height: 0px; margin-bottom: 20px; margin-top: 20px;" />
<h3 id="review-kafka-consumer" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Review Kafka Consumer</h3>
<h4 id="how-did-we-demonstrate-consumers-in-a-consumer-group-dividing-up-topic-partitions-and-sharing-them" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
How did we demonstrate Consumers in a Consumer Group dividing up topic partitions and sharing them?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We ran three consumers in the same consumer group, and then sent 25 messages from the producer. We saw that each consumer owned a set of partitions.</div>
<h4 id="how-did-we-demonstrate-consumers-in-different-consumer-groups-each-getting-their-own-offsets" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
How did we demonstrate Consumers in different Consumer Groups each getting their own offsets?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We ran three consumers each in its own unique consumer group, and then sent 5 messages from the producer. We saw that each consumer owned every partition.</div>
<h4 id="how-many-records-does-poll-get" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
How many records does poll get?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
However many you set in with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 100);</code> in the properties that you pass to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">KafkaConsumer</code>.</div>
<h4 id="does-a-call-to-poll-ever-get-records-from-two-different-partitions" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Does a call to poll ever get records from two different partitions?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
No</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="related-content" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Related content</h4>
<ul style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px; margin-top: 0px;">
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/blog/kafka-tutorial-kafka-producer/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka Tutorial: Kafka Producer</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/blog/kafka-architecture/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka Architecture</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/blog/what-is-kafka/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">What is Kafka?</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/blog/kafka-architecture-topics/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka Topic Architecture</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/blog/kafka-architecture-consumers/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka Consumer Architecture</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/blog/kafka-architecture-producers/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka Producer Architecture</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/blog/kafka-avro-schema-registry/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka and Schema Registry</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/blog/avro/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka and Avro</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/ppt/kafka-tutorial-cloudruable-v2.pdf" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka Tutorial Slides</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/blog/kafka-tutorial-kafka-from-command-line/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka from the command line</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/blog/kafka-tutorial-kafka-failover-kafka-cluster/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka clustering and failover basics</a></li>
</ul>
<div>
<span style="background-color: white; color: #555555; font-family: "roboto" , "helvetica" , "arial" , sans-serif; font-size: 14px;">We hope you enjoyed this article. Please provide </span><a href="http://cloudurable.com/contact/index.html" style="background-color: white; box-sizing: border-box; color: deepskyblue; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; text-decoration-line: none; transition: all 0.2s ease-out;">feedback</a><span style="background-color: white; color: #555555; font-family: "roboto" , "helvetica" , "arial" , sans-serif; font-size: 14px;">. See the </span><a href="http://cloudurable.com/kafka-training/index.html" style="background-color: white; box-sizing: border-box; color: deepskyblue; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka training</a><span style="background-color: white; color: #555555; font-family: "roboto" , "helvetica" , "arial" , sans-serif; font-size: 14px;">, </span><a href="http://cloudurable.com/kafka-aws-consulting/index.html" style="background-color: white; box-sizing: border-box; color: deepskyblue; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka consulting</a><span style="background-color: white; color: #555555; font-family: "roboto" , "helvetica" , "arial" , sans-serif; font-size: 14px;">, </span><a href="http://cloudurable.com/subscription_support/index.html" style="background-color: white; box-sizing: border-box; color: deepskyblue; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka support</a><span style="background-color: white; color: #555555; font-family: "roboto" , "helvetica" , "arial" , sans-serif; font-size: 14px;"> and helps </span><a href="http://cloudurable.com/services/index.html" style="background-color: white; box-sizing: border-box; color: deepskyblue; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; text-decoration-line: none; transition: all 0.2s ease-out;">setting up Kafka clusters in AWS</a><span style="background-color: white; color: #555555; font-family: "roboto" , "helvetica" , "arial" , sans-serif; font-size: 14px;">.</span></div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com0tag:blogger.com,1999:blog-6128525281101058933.post-8915882928419786862017-05-16T08:02:00.003-07:002017-05-19T15:05:07.951-07:00Kafka Tutorial: Kafka Clustering, Consumer Failover, Broker Failover<div style="color: #333333; font-family: "Lucida Grande", "Lucida Sans", Verdana, Arial, sans-serif; font-size: 13.3333px;">
This tutorial covers Kafka clustering and replicated topic. It demonstrates consumer failover and broker failover. It also demonstrates load balancing Kafka consumers. The article shows how, with many groups, Kafka acts like a Publish/Subscribe MOM. But, when you put all of our consumers in the same group, Kafka will load share the messages to the consumers in the same group like a MOM queue. This Kafka tutorial demonstrates how Kafka consumer failover and Kafka broker failover. </div>
<div style="color: #333333; font-family: "Lucida Grande", "Lucida Sans", Verdana, Arial, sans-serif; font-size: 13.3333px;">
<br /></div>
<div style="color: #333333; font-family: "Lucida Grande", "Lucida Sans", Verdana, Arial, sans-serif; font-size: 13.3333px;">
See <a href="http://cloudurable.com/blog/kafka-tutorial-kafka-failover-kafka-cluster/index.html" style="background-attachment: initial; background-clip: initial; background-image: url("bg_link.gif"); background-origin: initial; background-position: initial; background-repeat: repeat-x; background-size: initial; color: #666666; cursor: text; font-weight: bold; padding: 2px;" target="">Kafka Tutorial: Kafka Clustering, Consumer Failover, Broke Failover</a>.</div>
<div style="color: #333333; font-family: "Lucida Grande", "Lucida Sans", Verdana, Arial, sans-serif; font-size: 13.3333px;">
<br /></div>
<div style="color: #333333; font-family: "Lucida Grande", "Lucida Sans", Verdana, Arial, sans-serif; font-size: 13.3333px;">
Reposted with permission.</div>
<div style="color: #333333; font-family: "Lucida Grande", "Lucida Sans", Verdana, Arial, sans-serif; font-size: 13.3333px;">
<br /></div>
<div style="color: #333333; font-family: "Lucida Grande", "Lucida Sans", Verdana, Arial, sans-serif; font-size: 13.3333px;">
<br /></div>
<div style="color: #333333; font-family: "Lucida Grande", "Lucida Sans", Verdana, Arial, sans-serif; font-size: 13.3333px;">
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
f you are not sure what Kafka is, start here <a href="http://cloudurable.com/blog/what-is-kafka/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">“What is Kafka?”</a>.</div>
<h2 id="getting-started-with-kafka-cluster-tutorial" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 30px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Getting started with Kafka cluster tutorial</h2>
<h2 id="understanding-kafka-failover" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 30px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Understanding Kafka Failover</h2>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
This Kafka tutorial picks up right where the first <a href="http://cloudurable.com/blog/kafka-tutorial-kafka-from-command-line/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka tutorial from the command line</a> left off. The first tutorial has instructions on how to run ZooKeeper and use Kafka utils.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
In this tutorial, we are going to run many Kafka Nodes on our development laptop so that you will need at least 16 GB of RAM for local dev machine. You can run just two servers if you have less memory than 16 GB. We are going to create a replicated topic. We then demonstrate consumer failover and broker failover. We also demonstrate load balancing Kafka consumers. We show how, with many groups, Kafka acts like a Publish/Subscribe. But, when we put all of our consumers in the same group, Kafka will load share the messages to the consumers in the same group (more like a queue than a topic in a traditional MOM sense).</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
If not already running, then start up ZooKeeper (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">./run-zookeeper.sh</code> from the first tutorial). Also, shut down Kafka from the first tutorial.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Next, you need to copy server properties for three brokers (detailed instructions to follow). Then we will modify these Kafka server properties to add unique Kafka ports, Kafka log locations, and unique Broker ids. Then we will create three scripts to start these servers up using these properties, and then start the servers. Lastly, we create replicated topic and use it to demonstrate Kafka consumer failover, and Kafka broker failover.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h3 id="create-three-new-kafka-server-n-properties-files" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Create three new Kafka server-n.properties files</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
In this section, we will copy the existing Kafka <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">server.properties</code> to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">server-0.properties</code>, <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">server-1.properties</code>, and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">server-2.properties</code>. Then we change <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">server-0.properties</code> to set <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">log.dirs</code> to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">“./logs/kafka-0</code>. Then we modify <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">server-1.properties</code> to set <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">port</code> to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">9093</code>, broker <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">id</code> to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">1</code>, and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">log.dirs</code> to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">“./logs/kafka-1”</code>. Lastly modify <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">server-2.properties</code>to use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">port</code> 9094, broker <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">id</code> 2, and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">log.dirs</code> <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">“./logs/kafka-2”</code>.</div>
<h4 id="copy-server-properties-file" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Copy server properties file</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ~/kafka-training
$ mkdir -p lab2/config
$ cp kafka/config/server.properties kafka/lab2/config/server-0.properties
$ cp kafka/config/server.properties kafka/lab2/config/server-1.properties
$ cp kafka/config/server.properties kafka/lab2/config/server-2.properties
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
With your favorite text editor change server-0.properties so that <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">log.dirs</code> is set to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">./logs/kafka-0</code>. Leave the rest of the file the same. Make sure <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">log.dirs</code> is only defined once.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="kafka-training-lab2-config-server-0-properties" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab2/config/server-0.properties</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">broker.id=0
port=9092
log.dirs=./logs/kafka-0
...
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
With your favorite text editor change <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">log.dirs</code>, <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">broker.id</code> and and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">log.dirs</code> of <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">server-1.properties</code> as follows.</div>
<h4 id="kafka-training-lab2-config-server-1-properties" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab2/config/server-1.properties</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">broker.id=1
port=9093
log.dirs=./logs/kafka-1
...
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
With your favorite text editor change <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">log.dirs</code>, <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">broker.id</code> and and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">log.dirs</code> of <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">server-2.properties</code> as follows.</div>
<h4 id="kafka-training-lab2-config-server-2-properties" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab2/config/server-2.properties</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">broker.id=2
port=9094
log.dirs=./logs/kafka-2
...
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h3 id="create-startup-scripts-for-three-kafka-servers" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Create Startup scripts for three Kafka servers</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The startup scripts will just run <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">kafka-server-start.sh</code> with the corresponding properties file.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="kafka-training-lab2-start-1st-server-sh" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab2/start-1st-server.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/usr/bin/env bash
CONFIG=`pwd`/config
cd ~/kafka-training
## Run Kafka
kafka/bin/kafka-server-start.sh \
"$CONFIG/server-0.properties"
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="kafka-training-lab2-start-2nd-server-sh" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab2/start-2nd-server.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/usr/bin/env bash
CONFIG=`pwd`/config
cd ~/kafka-training
## Run Kafka
kafka/bin/kafka-server-start.sh \
"$CONFIG/server-1.properties"
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="kafka-training-lab2-start-3rd-server-sh" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab2/start-3rd-server.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/usr/bin/env bash
CONFIG=`pwd`/config
cd ~/kafka-training
## Run Kafka
kafka/bin/kafka-server-start.sh \
"$CONFIG/server-2.properties"
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice we are passing the Kafka server properties files that we created in the last step.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now run all three in separate terminals/shells.</div>
<h4 id="run-kafka-servers-each-in-own-terminal-from-kafka-training-lab2" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Run Kafka servers each in own terminal from ~/kafka-training/lab2</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-1st-server.sh
...
$ ./start-2nd-server.sh
...
$ ./start-3rd-server.sh
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Give the servers a minute to startup and connect to ZooKeeper.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h3 id="create-kafka-replicated-topic-my-failsafe-topic" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Create Kafka replicated topic my-failsafe-topic</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now we will create a replicated topic that the console producers and console consumers can use.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="kafka-training-lab2-create-replicated-topic-sh" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab2/create-replicated-topic.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/usr/bin/env bash
cd ~/kafka-training
kafka/bin/kafka-topics.sh --create \
--zookeeper localhost:2181 \
--replication-factor 3 \
--partitions 13 \
--topic my-failsafe-topic
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that the replication factor gets set to 3, and the topic name is <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">my-failsafe-topic</code>, and like before it has 13 partitions.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Then we just have to run the script to create the topic.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="run-create-replicated-topic-sh" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Run create-replicated-topic.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./create-replicated-topic.sh
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h3 id="start-kafka-consumer-that-uses-replicated-topic" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Start Kafka Consumer that uses Replicated Topic</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Next, create a script that starts the consumer and then start the consumer with the script.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="kafka-training-lab2-start-consumer-console-replicated-sh" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab2/start-consumer-console-replicated.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/usr/bin/env bash
cd ~/kafka-training
kafka/bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9094,localhost:9092 \
--topic my-failsafe-topic \
--from-beginning
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that a list of Kafka servers is passed to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">--bootstrap-server</code> parameter. Only, two of the three servers get passed that we ran earlier. Even though only one broker is needed, the consumer client will learn about the other broker from just one server. Usually, you list multiple brokers in case there is an outage so that the client can connect.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now we just run this script to start the consumer.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="run-start-consumer-console-replicated-sh" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Run start-consumer-console-replicated.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-console-replicated.sh
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h3 id="start-kafka-producer-that-uses-replicated-topic" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Start Kafka Producer that uses Replicated Topic</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Next, we create a script that starts the producer. Then launch the producer with the script you create.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="kafka-training-lab2-start-consumer-producer-replicated-sh" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab2/start-consumer-producer-replicated.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/usr/bin/env bash
cd ~/kafka-training
kafka/bin/kafka-console-producer.sh \
--broker-list localhost:9092,localhost:9093 \
--topic my-failsafe-topic
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice we start Kafka producer and pass it a list of Kafka Brokers to use via the parameter <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">--broker-list</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now use the start producer script to launch the producer as follows.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="run-start-producer-console-replicated-sh" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Run start-producer-console-replicated.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-producer-replicated.sh
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h3 id="now-send-messages" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Now send messages</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now send some message from the producer to Kafka and see those messages consumed by the consumer.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="producer-console" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Producer Console</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-producer-replicated.sh
Hi Mom
How are you?
How are things going?
Good!
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="consumer-console" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Consumer Console</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-console-replicated.sh
Hi Mom
How are you?
How are things going?
Good!
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h3 id="now-start-two-more-consumers-and-send-more-messages" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Now Start two more consumers and send more messages</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now Start two more consumers in their own terminal window and send more messages from the producer.</div>
<h4 id="producer-console-1" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Producer Console</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-producer-replicated.sh
Hi Mom
How are you?
How are things going?
Good!
message 1
message 2
message 3
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="consumer-console-1st" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Consumer Console 1st</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-console-replicated.sh
Hi Mom
How are you?
How are things going?
Good!
message 1
message 2
message 3
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="consumer-console-2nd-in-new-terminal" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Consumer Console 2nd in new Terminal</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-console-replicated.sh
Hi Mom
How are you?
How are things going?
Good!
message 1
message 2
message 3
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="consumer-console-2nd-in-new-terminal-1" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Consumer Console 2nd in new Terminal</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-console-replicated.sh
Hi Mom
How are you?
How are things going?
Good!
message 1
message 2
message 3
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that the messages are sent to all of the consumers because each consumer is in a different consumer group.</div>
<h3 id="change-consumer-to-be-in-their-own-consumer-group" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Change consumer to be in their own consumer group</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Stop the producers and the consumers from before, but leave Kafka and ZooKeeper running.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now let’s modify the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">start-consumer-console-replicated.sh</code> script to add a Kafka <em style="box-sizing: border-box;">consumer group</em>. We want to put all of the consumers in same <em style="box-sizing: border-box;">consumer group</em>. This way the consumers will share the messages as each consumer in the <em style="box-sizing: border-box;">consumer group</em> will get its share of partitions.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="kafka-training-lab2-start-consumer-console-replicated-sh-1" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab2/start-consumer-console-replicated.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/usr/bin/env bash
cd ~/kafka-training
kafka/bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9094,localhost:9092 \
--topic my-failsafe-topic \
--consumer-property group.id=mygroup
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that the script is the same as before except we added <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">--consumer-property group.id=mygroup</code> which will put every consumer that runs with this script into the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">mygroup</code> consumer group.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now we just run the producer and three consumers.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="run-this-three-times-start-consumer-console-replicated-sh" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Run this three times - start-consumer-console-replicated.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-console-replicated.sh
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="run-producer-console" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Run Producer Console</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-producer-replicated.sh
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now send seven messages from the Kafka producer console.</div>
<h4 id="producer-console-2" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Producer Console</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-producer-replicated.sh
m1
m2
m3
m4
m5
m6
m7
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that the messages are spread evenly among the consumers.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="1st-kafka-consumer-gets-m3-m5" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
1st Kafka Consumer gets m3, m5</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-console-replicated.sh
m3
m5
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice the first consumer gets messages m3 and m5.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="2nd-kafka-consumer-gets-m2-m6" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
2nd Kafka Consumer gets m2, m6</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-console-replicated.sh
m2
m6
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice the second consumer gets messages m2 and m6.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="3rd-kafka-consumer-gets-m1-m4-m7" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
3rd Kafka Consumer gets m1, m4, m7</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-console-replicated.sh
m1
m4
m7
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice the third consumer gets messages m1, m4 and m7.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that each consumer in the group got a share of the messages.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h3 id="kafka-consumer-failover" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Kafka Consumer Failover</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Next, let’s demonstrate consumer failover by killing one of the consumers and sending seven more messages. Kafka should divide up the work to the consumers that are running.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
First, kill the third consumer (CTRL-C in the consumer terminal does the trick).</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now send seven more messages with the Kafka console-producer.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="producer-console-send-seven-more-messages-m8-through-m14" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Producer Console - send seven more messages m8 through m14</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-producer-replicated.sh
m1
...
m8
m9
m10
m11
m12
m13
m14
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that the messages are spread evenly among the remaining consumers.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="1st-kafka-consumer-gets-m8-m9-m11-m14" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
1st Kafka Consumer gets m8, m9, m11, m14</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-console-replicated.sh
m3
m5
m8
m9
m11
m14
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The first consumer got m8, m9, m11 and m14.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="2nd-kafka-consumer-gets-m10-m12-m13" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
2nd Kafka Consumer gets m10, m12, m13</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-console-replicated.sh
m2
m6
m10
m12
m13
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The second consumer got m10, m12, and m13.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We killed one consumer, sent seven more messages, and saw Kafka spread the load to remaining consumers. <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">Kafka consumer failover works!</em></span></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h3 id="create-kafka-describe-topic-script" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Create Kafka Describe Topic Script</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
You can use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">kafka-topics.sh</code> to see how the Kafka topic is laid out among the Kafka brokers. The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">---describe</code> will show partitions, ISRs, and broker partition leadership.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="kafka-training-lab2-describe-topics-sh" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/kafka-training/lab2/describe-topics.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/usr/bin/env bash
cd ~/kafka-training
# List existing topics
kafka/bin/kafka-topics.sh --describe \
--topic my-failsafe-topic \
--zookeeper localhost:2181
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s run <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">kafka-topics.sh --describe</code> and see the topology of our <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">my-failsafe-topic</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h3 id="run-describe-topics" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Run describe-topics</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We are going to lists which broker owns (leader of) which partition, and list replicas and ISRs of each partition. ISRs are replicas that are up to date. Remember there are 13 topics.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="topology-of-kafka-topic-partition-ownership" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Topology of Kafka Topic Partition Ownership</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./describe-topics.sh
Topic: my-failsafe-topic PartitionCount: 13 ReplicationFactor: 3 Configs:
Topic: my-failsafe-topic Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: my-failsafe-topic Partition: 1 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
Topic: my-failsafe-topic Partition: 2 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Topic: my-failsafe-topic Partition: 3 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
Topic: my-failsafe-topic Partition: 4 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
Topic: my-failsafe-topic Partition: 5 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
Topic: my-failsafe-topic Partition: 6 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: my-failsafe-topic Partition: 7 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
Topic: my-failsafe-topic Partition: 8 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Topic: my-failsafe-topic Partition: 9 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
Topic: my-failsafe-topic Partition: 10 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
Topic: my-failsafe-topic Partition: 11 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
Topic: my-failsafe-topic Partition: 12 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice how each broker gets a share of the partitions as leaders and followers. Also, see how Kafka replicates the partitions on each broker.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h3 id="test-broker-failover-by-killing-1st-server" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Test Broker Failover by killing 1st server</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s kill the first broker, and then test the failover.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="kill-the-first-broker" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Kill the first broker</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;"> $ kill `ps aux | grep java | grep server-0.properties | tr -s " " | cut -d " " -f2`
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
You can stop the first broker by hitting CTRL-C in the broker terminal or by running the above command.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now that the first Kafka broker has stopped, let’s use Kafka <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">topics describe</code> to see that new leaders were elected!</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="run-describe-topics-again-to-see-leadership-change" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Run describe-topics again to see leadership change</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2/solution
$ ./describe-topics.sh
Topic:my-failsafe-topic PartitionCount:13 ReplicationFactor:3 Configs:
Topic: my-failsafe-topic Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 2,1
Topic: my-failsafe-topic Partition: 1 Leader: 1 Replicas: 0,1,2 Isr: 1,2
Topic: my-failsafe-topic Partition: 2 Leader: 1 Replicas: 1,2,0 Isr: 1,2
Topic: my-failsafe-topic Partition: 3 Leader: 2 Replicas: 2,1,0 Isr: 2,1
Topic: my-failsafe-topic Partition: 4 Leader: 2 Replicas: 0,2,1 Isr: 2,1
Topic: my-failsafe-topic Partition: 5 Leader: 1 Replicas: 1,0,2 Isr: 1,2
Topic: my-failsafe-topic Partition: 6 Leader: 2 Replicas: 2,0,1 Isr: 2,1
Topic: my-failsafe-topic Partition: 7 Leader: 1 Replicas: 0,1,2 Isr: 1,2
Topic: my-failsafe-topic Partition: 8 Leader: 1 Replicas: 1,2,0 Isr: 1,2
Topic: my-failsafe-topic Partition: 9 Leader: 2 Replicas: 2,1,0 Isr: 2,1
Topic: my-failsafe-topic Partition: 10 Leader: 2 Replicas: 0,2,1 Isr: 2,1
Topic: my-failsafe-topic Partition: 11 Leader: 1 Replicas: 1,0,2 Isr: 1,2
Topic: my-failsafe-topic Partition: 12 Leader: 2 Replicas: 2,0,1 Isr: 2,1
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" />
Notice how Kafka spreads the leadership over the 2nd and 3rd Kafka brokers.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h3 id="show-broker-failover-worked" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Show Broker Failover Worked</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s prove that failover worked by sending two more messages from the producer console.<br />
Then notice if the consumers still get the messages.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Send the message m15 and m16.</div>
<h4 id="producer-console-send-m15-and-m16" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Producer Console - send m15 and m16</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-producer-replicated.sh
m1
...
m15
m16
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that the messages are spread evenly among the remaining live consumers.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="1st-kafka-consumer-gets-m16" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
1st Kafka Consumer gets m16</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-console-replicated.sh
m3
m5
m8
m9
m11
m14
...
m16
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The first Kafka broker gets m16.</div>
<h4 id="2nd-kafka-consumer-gets-m15" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
2nd Kafka Consumer gets m15</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">~/kafka-training/lab2
$ ./start-consumer-console-replicated.sh
m2
m6
m10
m12
m13
...
m15
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The second Kafka broker gets m15.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Kafka broker Failover WORKS!</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h3 id="kafka-cluster-failover-review" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Kafka Cluster Failover Review</h3>
<h4 id="why-did-the-three-consumers-not-load-share-the-messages-at-first" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Why did the three consumers not load share the messages at first?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
They did not load share at first because they were each in a different consumer group. Consumer groups each subscribe to a topic and maintain their own offsets per partition in that topic.</div>
<h4 id="how-did-we-demonstrate-failover-for-consumers" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
How did we demonstrate failover for consumers?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We shut a consumer down. Then we sent more messages. We observed Kafka spreading messages to the remaining cluster.</div>
<h4 id="how-did-we-show-failover-for-producers" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
How did we show failover for producers?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We didn’t. We showed failover for Kafka brokers by shutting one down, then using the producer console to send two more messages. Then we saw that the producer used the remaining Kafka brokers. Those Kafka brokers then delivered the messages to the live consumers.</div>
<h4 id="what-tool-and-option-did-we-use-to-show-ownership-of-partitions-and-the-isrs" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
What tool and option did we use to show ownership of partitions and the ISRs?</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We used <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">kafka-topics.sh</code> using the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">--describe</code> option.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h3 id="more-about-kafka" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
More about Kafka</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
To learn about Kafka see <a href="http://cloudurable.com/blog/kafka-architecture/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka architecture</a>, <a href="http://cloudurable.com/blog/kafka-architecture-topics/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka topic architecture</a> and <a href="http://cloudurable.com/blog/kafka-architecture-producers/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka producer architecture</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h3 id="related-content" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Related content</h3>
<ul style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px; margin-top: 0px;">
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/blog/kafka-avro-schema-registry/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka and Schema Registry</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/blog/avro/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka and Avro</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/ppt/kafka-tutorial-cloudruable-v2.pdf" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka Tutorial Slides</a></li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br style="box-sizing: border-box;" /></div>
<h4 id="about-cloudurable" style="background-color: white; box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
About Cloudurable</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We hope you enjoyed this article. Please provide <a href="http://cloudurable.com/contact/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">feedback</a>. Cloudurable provides <a href="http://cloudurable.com/kafka-training/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka training</a>, <a href="http://cloudurable.com/kafka-aws-consulting/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka consulting</a>, <a href="http://cloudurable.com/subscription_support/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka support</a> and helps <a href="http://cloudurable.com/services/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">setting up Kafka clusters in AWS</a>.</div>
</div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com1tag:blogger.com,1999:blog-6128525281101058933.post-5528295178079775962017-05-09T09:53:00.002-07:002017-05-09T10:20:30.687-07:00Apache Avro Tutorial <h2 id="avro" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 30px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Apache Avro Tutorial</h2>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Apache Avro™ is a data serialization system. Avro provides data structures, binary data format, container file for storing persistent data and provides RPC capabilities. Avro does not require code generation to use. Avro is polyglot like you would expect and integrates well with JavaScript, Python, Ruby, C, C#, C++ and Java. Avro gets used in the <em style="box-sizing: border-box;">Hadoop ecosystem</em> as well as by <em style="box-sizing: border-box;">Kafka</em>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Avro is similar to Thrift, Protocol Buffers, JSON, etc. Unlike Thrift and Protocol Buf, Avro does not require code generation. Avro needs less encoding as part of the data since it stores names and types in the schema reducing duplication. Avro supports the evolution of schemas. <b><i>Kafka</i></b> uses <b><i>Avro</i></b> with its <a href="https://github.com/confluentinc/schema-registry">Schema Registry</a>. </div>
<h2 id="avro-schema" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 30px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Avro Schema</h2>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Avro data format (wire format and file format) is defined by Avro schemas. When deserializing data, the schema is used. Data is serialized based on the schema, and schema is sent with data or in the case of files stored with the data. Avro data plus schema is fully self-describing data format.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
When Avro files store data it also stores schema. Avro RPC is also based on schema, and IDL. Part of the RPC protocol exchanges schemas as part of the handshake. Avro schemas and IDL are written in JSON.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s take a look at an example Avro schema.</div>
<h4 id="src-main-avro-com-cloudurable-phonebook-employee-avsc" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
./src/main/avro/com/cloudurable/phonebook/Employee.avsc</h4>
<h4 id="example-schema-for-an-employee-record" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Example schema for an Employee record</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-javascript" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">{"namespace": "com.cloudurable.phonebook",
"type": "record",
"name": "Employee",
"fields": [
{"name": "firstName", "type": "string"},
{"name": "lastName", "type": "string"},
{"name": "age", "type": "int"},
{"name": "phoneNumber", "type": "string"}
]
}
</code></pre>
<h2 id="avro-schema-generation-tools" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 30px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Avro schema generation tools</h2>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Avro comes with a set of tools for generating Java classes for Avro types that you define in Avro schema. There are plugins for Maven and Gradle to generate code based on Avro schemas.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
This <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">gradle-avro-plugin</code> is a Gradle plugin that uses Avro tools to do Java code generation for Apache Avro. This plugin supports Avro schema files (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">.avsc</code>), and Avro RPC IDL (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">.avdl</code>). For <a href="http://cloudurable.com/kafka-training/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka</a> you only need <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">avsc</code> schema files.</div>
<h4 id="build-gradle-example-using-gradle-avro-plugin" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
build.gradle - example using gradle-avro-plugin</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">plugins {
id "com.commercehub.gradle.plugin.avro" version "0.9.0"
}
group 'cloudurable'
version '1.0-SNAPSHOT'
apply plugin: 'java'
sourceCompatibility = 1.8
dependencies {
compile "org.apache.avro:avro:1.8.1"
testCompile group: 'junit', name: 'junit', version: '4.11'
}
repositories {
jcenter()
mavenCentral()
}
avro {
createSetters = false
fieldVisibility = "PRIVATE"
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that we did not generate setter methods, and we made the fields private. This makes the instances somewhat immutable.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Running <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">gradle build</code> will generate the Employee.java.</div>
<h4 id="build-generated-main-avro-java-com-cloudurable-phonebook-employee-java" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
./build/generated-main-avro-java/com/cloudurable/phonebook/Employee.java</h4>
<h4 id="generated-avro-code" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Generated Avro code</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">
package com.cloudurable.phonebook;
import org.apache.avro.specific.SpecificData;
@SuppressWarnings("all")
@org.apache.avro.specific.AvroGenerated
public class Employee extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {
private static final long serialVersionUID = -6112285611684054927L;
public static final org.apache.avro.Schema SCHEMA$ = new
org.apache.avro.Schema.Parser().parse("{\"type\":\"record\",\"name\":\"Employee\"...");
public static org.apache.avro.Schema getClassSchema() { return SCHEMA$; }
private java.lang.String firstName;
private java.lang.String lastName;
private int age;
private java.lang.String phoneNumber;
...
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The gradle plugin calls the Avro utilities which generates the files and puts them under <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">build/generated-main-avro-java</code></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s use the generated class as follows to construct an Employee instance.</div>
<h4 id="using-the-new-employee-class" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Using the new Employee class</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">Employee bob = Employee.newBuilder().setAge(35)
.setFirstName("Bob")
.setLastName("Jones")
.setPhoneNumber("555-555-1212")
.build();
assertEquals("Bob", bob.getFirstName());
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The Employee class has a constructor and has a builder. We can use the builder to build a new Employee instance.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Next we want to write the Employees to disk.</div>
<h4 id="writing-a-list-of-employees-to-an-avro-file" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Writing a list of employees to an Avro file</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">final List<Employee> employeeList = ...
final DatumWriter<Employee> datumWriter = new SpecificDatumWriter<>(Employee.class);
final DataFileWriter<Employee> dataFileWriter = new DataFileWriter<>(datumWriter);
try {
dataFileWriter.create(employeeList.get(0).getSchema(),
new File("employees.avro"));
employeeList.forEach(employee -> {
try {
dataFileWriter.append(employee);
} catch (IOException e) {
throw new RuntimeException(e);
}
});
} finally {
dataFileWriter.close();
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The above shoes serializing an Employee list to disk. In Kafka, we will not be writing to disk directly. We are just showing how so you have a way to test Avro serialization, which is helpful when debugging schema incompatibilities. Note we create a <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">DatumWriter</code>, which converts Java instance into an in-memory serialized format. <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">SpecificDatumWriter</code> is used with generated classes like Employee.<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">DataFileWriter</code> writes the serialized records to the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">employee.avro</code> file.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now let’s demonstrate how to read data from an Avro file.</div>
<h4 id="reading-a-list-of-employees-from-an-avro-file" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Reading a list of employees from an avro file</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">final File file = new File("employees.avro");
final List<Employee> employeeList = new ArrayList<>();
final DatumReader<Employee> empReader = new SpecificDatumReader<>(Employee.class);
final DataFileReader<Employee> dataFileReader = new DataFileReader<>(file, empReader);
while (dataFileReader.hasNext()) {
employeeList.add(dataFileReader.next(new Employee()));
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The above deserializes employees from the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">employees.avro</code> file into a <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">java.util.List</code> of Employee instances. Deserializing is similar to serializing but in reverse. We create a <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">SpecificDatumReader</code> to converts in-memory serialized items into instances of our generated <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">Employee</code> class. The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">DatumReader</code> reads records from the file by calling next. Another way to read is using forEach as follows:</div>
<h4 id="reading-a-list-of-employees-from-an-avro-file-using-foreach" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Reading a list of employees from an avro file using forEach</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">final DataFileReader<Employee> dataFileReader = new DataFileReader<>(file, empReader);
dataFileReader.forEach(employeeList::add);
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
You can use a <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">GenericRecord</code> instead of generating an Employee class as follows.</div>
<h4 id="using-genericrecord-to-create-an-employee-record" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Using GenericRecord to create an Employee record</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">final String schemaLoc = "src/main/avro/com/cloudurable/phonebook/Employee.avsc";
final File schemaFile = new File(schemaLoc);
final Schema schema = new Schema.Parser().parse(schemaFile);
GenericRecord bob = new GenericData.Record(schema);
bob.put("firstName", "Bob");
bob.put("lastName", "Smith");
bob.put("age", 35);
assertEquals("Bob", bob.get("firstName"));
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
You can write to Avro files using GenericRecords as well.</div>
<h4 id="writing-genericrecords-to-an-avro-file" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Writing GenericRecords to an Avro file</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">final List<GenericRecord> employeeList = new ArrayList<>();
final DatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<>(schema);
final DataFileWriter<GenericRecord> dataFileWriter = new DataFileWriter<>(datumWriter);
try {
dataFileWriter.create(employeeList.get(0).getSchema(),
new File("employees2.avro"));
employeeList.forEach(employee -> {
try {
dataFileWriter.append(employee);
} catch (IOException e) {
throw new RuntimeException(e);
}
});
} finally {
dataFileWriter.close();
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
You can read from Avro files using <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">GenericRecord</code>s as well.</div>
<h4 id="reading-genericrecords-from-an-avro-file" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Reading GenericRecords from an Avro file</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">final File file = new File("employees2.avro");
final List<GenericRecord> employeeList = new ArrayList<>();
final DatumReader<GenericRecord> empReader = new GenericDatumReader<>();
final DataFileReader<GenericRecord> dataFileReader = new DataFileReader<>(file, empReader);
while (dataFileReader.hasNext()) {
employeeList.add(dataFileReader.next(null));
}
employeeList.forEach(System.out::println);
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Avro will validate the data types when it serializes and deserializes the data.</div>
<h4 id="using-the-wrong-type" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Using the wrong type</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">GenericRecord employee = new GenericData.Record(schema);
employee.put("firstName", "Bob" + index);
employee.put("lastName", "Smith"+ index);
//employee.put("age", index % 35 + 25);
employee.put("age", "OLD");
</code></pre>
<h4 id="stack-trace-from-above" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Stack trace from above</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">
org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.ClassCastException:
java.lang.String cannot be cast to java.lang.Number
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
at com.cloudurable.phonebook.EmployeeTestNoGen.lambda$testWrite$1(EmployeeTestNoGen.java:71)
at java.util.ArrayList.forEach(ArrayList.java:1249)
at com.cloudurable.phonebook.EmployeeTestNoGen.testWrite(EmployeeTestNoGen.java:69)
...
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Number
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:117)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153)
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:302)
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
If you left out a required field like <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">firstName</code>, then you would get this.</div>
<h4 id="stack-trace-from-leaving-out-firstname" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Stack trace from leaving out firstName</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">Caused by: java.lang.NullPointerException: null of string in field firstName of com.cloudurable.phonebook.Employee
at org.apache.avro.generic.GenericDatumWriter.npe(GenericDatumWriter.java:132)
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:126)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <a href="https://avro.apache.org/docs/current/spec.html#Protocol+Declaration" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Avro schema and IDL specification document</a> describes all of the supported types.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br />
<div style="-webkit-text-stroke-width: 0px; background-color: white; box-sizing: border-box; color: #555555; font-family: roboto, helvetica, arial, sans-serif; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: normal; letter-spacing: normal; margin-bottom: 10px; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">
</div>
<br />
<div style="-webkit-text-stroke-width: 0px; background-color: white; box-sizing: border-box; color: #555555; font-family: roboto, helvetica, arial, sans-serif; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: normal; letter-spacing: normal; margin-bottom: 10px; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">
<div style="margin: 0px;">
With Avro schema, you can define records, arrays, enums, unions, maps and you can use primitive types like Strings, Int, Boolean, Decimal, Timestamp, Date, and more.</div>
</div>
Next, Let’s add to the Employee schema and show some of the different types that Avro supports.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<b><u>Employe Schema</u></b></div>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;"> {"namespace": "com.cloudurable.phonebook",
"type": "record",
"name": "Employee",
"fields": [
{"name": "firstName", "type": "string"},
{"name": "nickName", "type": ["null", "string"], "default" : null},
{"name": "lastName", "type": "string"},
{"name": "age", "type": "int"},
{"name": "emails", "default":[], "type":{"type": "array", "items": "string"}},
{"name": "phoneNumber", "type":
[ "null",
{ "type": "record", "name": "PhoneNumber",
"fields": [
{"name": "areaCode", "type": "string"},
{"name": "countryCode", "type": "string", "default" : ""},
{"name": "prefix", "type": "string"},
{"name": "number", "type": "string"}
]
}
]
},
{"name":"status", "default" :"SALARY", "type": { "type": "enum", "name": "Status",
"symbols" : ["RETIRED", "SALARY", "HOURLY", "PART_TIME"]}
}
]
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The Employee schema uses default values, arrays, primitive types, Records within records, enums, and more. It also uses a Union type to represent a value that is optional.<br />
<br />
What follows are some classes that are generated from the above schema. </div>
<h4 id="phonenumber-record" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
PhoneNumber record</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">
package com.cloudurable.phonebook;
import org.apache.avro.specific.SpecificData;
@SuppressWarnings("all")
@org.apache.avro.specific.AvroGenerated
public class PhoneNumber extends org.apache.avro.specific.SpecificRecordBase ...{
private static final long serialVersionUID = -3138777939618426199L;
public static final org.apache.avro.Schema SCHEMA$ =
new org.apache.avro.Schema.Parser().parse("{\"type\":\"record\",\"name\":...
public static org.apache.avro.Schema getClassSchema() { return SCHEMA$; }
private java.lang.String areaCode;
private java.lang.String countryCode;
private java.lang.String prefix;
private java.lang.String number;
</code></pre>
<h4 id="status-enum" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Status enum</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-java" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">package com.cloudurable.phonebook;
@SuppressWarnings("all")
@org.apache.avro.specific.AvroGenerated
public enum Status {
RETIRED, SALARY, HOURLY, PART_TIME ;
...
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br />
<b><i><u>Conclusiton</u></i></b><br />
<a href="http://cloudurable.com/blog/avro/index.html">Avro</a> provides fast data serialization. It supports data structures like Supports Records, Maps, Array, and basic types. You can use it directly or use Code Generation. Avro allows schema support to Kafka which we will demonstrate in another article. <b><i>Kafka</i></b> uses <b><i>Avro</i></b> with its Schema Registry. </div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Enjoy this <a href="http://cloudurable.com/ppt/avro.pdf" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">slide deck about Avro</a> or this SlideShare by Jean-Paul on <a href="https://www.slideshare.net/JeanPaulAzar1/avro-tutorial-records-with-schema-for-kafka-and-hadoop">Avro/Kafka</a>. If you like this article check out my friends <a href="http://cloudurable.com/kafka-training/index.html">Kafka training course</a>.</div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com4tag:blogger.com,1999:blog-6128525281101058933.post-52364799479179508392017-05-07T17:50:00.002-07:002017-05-07T17:50:16.866-07:00Reakt Kafka Example<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; line-height: 1.25; margin-bottom: 16px; margin-top: 0px !important; padding-bottom: 0.3em;">
Reakt Kafka Example</h2>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
Reakt-Kakfa adapts <a href="http://cloudurable.com/kafka-training/index.html" style="background-color: transparent; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Kafka</a> to <a href="http://advantageous.github.io/reakt/" style="background-color: transparent; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Reakt promises and streams</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
Reakt has promise libraries for Vert.x, Netty, Guava, and Cassandra.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/reakt-kafka/wiki/Full-Reakt-Kafka-Example#using-promises-with-kafka-producers" id="user-content-using-promises-with-kafka-producers" style="background-color: transparent; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Using Promises with Kafka Producers</h4>
<div class="highlight highlight-source-java" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">AsyncProducer<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Long</span>, <span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span>></span> producer <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">AsyncProducer<></span>(createProducer());
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
producer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>send(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">TOPIC</span>, key, value)
.catchError(throwable <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>err<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>println(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Trouble sending record <span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">+</span> throwable<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getLocalizedMessage());
throwable<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>printStackTrace(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>err);
})
.then(recordMetadata <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>out<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>printf(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>%d %d %s <span class="pl-cce" style="box-sizing: border-box;">\n</span><span class="pl-pds" style="box-sizing: border-box;">"</span></span>, recordMetadata<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>offset(),
recordMetadata<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>partition(), recordMetadata<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>topic());
})<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>invoke();
</pre>
</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/reakt-kafka/wiki/Full-Reakt-Kafka-Example#using-streams-with-kafka-consumers" id="user-content-using-streams-with-kafka-consumers" style="background-color: transparent; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Using Streams with Kafka Consumers</h4>
<div class="highlight highlight-source-java" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">StreamConsumer<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Long</span>, <span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span>></span> stream <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">StreamConsumer</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>subscribe(createConsumer(), <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">TOPIC</span>, result <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
result<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>then(consumerRecords <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>out<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>println(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Got message <span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">+</span> consumerRecords<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>count());
consumerRecords<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>forEach(record <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
countDownLatch<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>countDown();
});
result<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>request(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1</span>); <span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span>calls commitAsync</span>
})<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>catchError(throwable <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>err<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>println(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Trouble Getting record <span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">+</span> throwable<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getLocalizedMessage());
throwable<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>printStackTrace(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>err);
result<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>cancel();
});
});
stream<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>close();
</pre>
</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/reakt-kafka/wiki/Full-Reakt-Kafka-Example#full-integration-test-and-example-for-reakt-kafka" id="user-content-full-integration-test-and-example-for-reakt-kafka" style="background-color: transparent; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Full integration test and example for Reakt Kafka</h4>
<div class="highlight highlight-source-java" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">io.advantageous.reakt.kafka</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">org.apache.kafka.clients.consumer.Consumer</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">org.apache.kafka.clients.consumer.ConsumerConfig</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">org.apache.kafka.clients.consumer.KafkaConsumer</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">org.apache.kafka.clients.producer.KafkaProducer</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">org.apache.kafka.clients.producer.Producer</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">org.apache.kafka.clients.producer.ProducerConfig</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">org.apache.kafka.common.serialization.LongDeserializer</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">org.apache.kafka.common.serialization.LongSerializer</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">org.apache.kafka.common.serialization.StringDeserializer</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">org.apache.kafka.common.serialization.StringSerializer</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">org.junit.Test</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">java.util.Properties</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">java.util.concurrent.CountDownLatch</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">java.util.concurrent.ExecutorService</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">java.util.concurrent.Executors</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">java.util.concurrent.TimeUnit</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import static</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">org.junit.Assert.assertEquals</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">class</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">IntegrationTest</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">static</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">TOPIC</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>my-test-topic<span class="pl-pds" style="box-sizing: border-box;">"</span></span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">static</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">BOOTSTRAP_SERVERS</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>localhost:9092<span class="pl-pds" style="box-sizing: border-box;">"</span></span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">static</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">int</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">SEND_RECORD_COUNT</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">10_000</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@Test</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">test</span>() <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">throws</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Exception</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">KafkaServer</span> kafkaServer <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">KafkaServer</span>();
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>out<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>println(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Starting server<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Thread</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>sleep(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">10_000</span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">ExecutorService</span> executorService <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Executors</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>newSingleThreadExecutor();
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">AsyncProducer<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Long</span>, <span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span>></span> producer <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">AsyncProducer<></span>(createProducer());
executorService<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>execute(() <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">for</span> (<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">int</span> i <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>; i <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;"><</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">SEND_RECORD_COUNT</span>; i<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">++</span>) {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">if</span> (i <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">%</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1000</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">==</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>) <span class="pl-smi" style="box-sizing: border-box; color: #333333;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>out<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>println(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Sending message <span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">+</span> i);
producer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>send(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">TOPIC</span>, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1L</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">*</span> i, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>value <span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">+</span> i)
.catchError(throwable <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>err<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>println(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Trouble sending record <span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">+</span> throwable<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getLocalizedMessage());
throwable<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>printStackTrace(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>err);
})
.then(recordMetadata <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">if</span> (recordMetadata<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>offset() <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">%</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1000</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">==</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>)
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>out<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>printf(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>%d %d %s <span class="pl-cce" style="box-sizing: border-box;">\n</span><span class="pl-pds" style="box-sizing: border-box;">"</span></span>, recordMetadata<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>offset(),
recordMetadata<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>partition(), recordMetadata<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>topic());
})
.invoke();
}
producer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>flush();
});
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">CountDownLatch</span> countDownLatch <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">CountDownLatch</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">SEND_RECORD_COUNT</span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">StreamConsumer<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Long</span>, <span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span>></span> stream <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">StreamConsumer</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>subscribe(createConsumer(), <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">TOPIC</span>, result <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
result<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>then(consumerRecords <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>out<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>println(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Got message <span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">+</span> consumerRecords<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>count());
consumerRecords<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>forEach(record <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
countDownLatch<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>countDown();
});
result<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>request(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1</span>);
})<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>catchError(throwable <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>err<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>println(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Trouble Getting record <span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">+</span> throwable<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getLocalizedMessage());
throwable<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>printStackTrace(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>err);
result<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>cancel();
});
});
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Thread</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>sleep(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">3_000</span>);
countDownLatch<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>await(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">10</span>, <span class="pl-smi" style="box-sizing: border-box; color: #333333;">TimeUnit</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>SECONDS</span>);
assertEquals(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>, countDownLatch<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getCount());
stream<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>close();
producer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>close();
executorService<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>shutdown();
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Thread</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>sleep(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">3_000</span>);
kafkaServer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>shutdown();
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Thread</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>sleep(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">3_000</span>);
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">static</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Producer<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Long</span>, <span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">createProducer</span>() {
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Properties</span> props <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Properties</span>();
props<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>put(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">ProducerConfig</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>BOOTSTRAP_SERVERS_CONFIG</span>, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">BOOTSTRAP_SERVERS</span>);
props<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>put(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">ProducerConfig</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>CLIENT_ID_CONFIG</span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>KafkaExampleProducer<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
props<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>put(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">ProducerConfig</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>KEY_SERIALIZER_CLASS_CONFIG</span>, <span class="pl-smi" style="box-sizing: border-box; color: #333333;">LongSerializer</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getName());
props<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>put(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">ProducerConfig</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>VALUE_SERIALIZER_CLASS_CONFIG</span>, <span class="pl-smi" style="box-sizing: border-box; color: #333333;">StringSerializer</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getName());
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">KafkaProducer<></span>(props);
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">static</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Consumer<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Long</span>, <span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">createConsumer</span>() {
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Properties</span> props <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Properties</span>();
props<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>put(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">ConsumerConfig</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>BOOTSTRAP_SERVERS_CONFIG</span>, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">BOOTSTRAP_SERVERS</span>);
props<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>put(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">ConsumerConfig</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>GROUP_ID_CONFIG</span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>KafkaExampleConsumer<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
props<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>put(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">ConsumerConfig</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>KEY_DESERIALIZER_CLASS_CONFIG</span>,
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">LongDeserializer</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getName());
props<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>put(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">ConsumerConfig</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>VALUE_DESERIALIZER_CLASS_CONFIG</span>,
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">StringDeserializer</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getName());
props<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>put(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">ConsumerConfig</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>MAX_POLL_RECORDS_CONFIG</span>, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1000</span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">KafkaConsumer<></span>(props);
}
}</pre>
</div>
<div style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
Notice that we use an embedded version of Kafka.</div>
<h4 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/reakt-kafka/wiki/Full-Reakt-Kafka-Example#kafka-embedded" id="user-content-kafka-embedded" style="background-color: transparent; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Kafka embedded</h4>
<div class="highlight highlight-source-java" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">io.advantageous.reakt.kafka</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">org.apache.zookeeper.server.ServerConfig</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">org.apache.zookeeper.server.ZooKeeperServerMain</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">org.apache.zookeeper.server.quorum.QuorumPeerConfig</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">java.io.IOException</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">java.util.Properties</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">kafka.server.KafkaConfig</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">kafka.server.KafkaServerStartable</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">class</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">KafkaServer</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">ZooKeeperServerMain</span> zooKeeperServer;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">KafkaServerStartable</span> kafkaServer;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">KafkaServer</span>() {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Properties</span> zkProperties <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Properties</span>();
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Properties</span> kafkaProperties <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Properties</span>();
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">try</span> {
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span>load properties</span>
kafkaProperties<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>load(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Class</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getResourceAsStream(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/io/advantageous/reakt/kafka/kafka.properties<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
zkProperties<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>load(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Class</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getResourceAsStream(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/io/advantageous/reakt/kafka/zookeeper.properties<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
} <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">catch</span> (<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Exception</span> e){
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">throw</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">RuntimeException</span>(e);
}
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">QuorumPeerConfig</span> quorumConfiguration <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">QuorumPeerConfig</span>();
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">try</span> {
quorumConfiguration<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>parseProperties(zkProperties);
} <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">catch</span>(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Exception</span> e) {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">throw</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">RuntimeException</span>(e);
}
zooKeeperServer <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">ZooKeeperServerMain</span>();
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">ServerConfig</span> configuration <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">ServerConfig</span>();
configuration<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>readFrom(quorumConfiguration);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Thread</span>(() <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">try</span> {
zooKeeperServer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>runFromConfig(configuration);
} <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">catch</span> (<span class="pl-smi" style="box-sizing: border-box; color: #333333;">IOException</span> e) {
e<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>printStackTrace(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>err);
}
})<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>start();
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span>start local kafka broker</span>
kafkaServer <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">KafkaServerStartable</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">KafkaConfig</span>(kafkaProperties));
kafkaServer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>startup();
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">shutdown</span>() {
kafkaServer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>shutdown();
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">static</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">main</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">String</span>[] <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">args</span>) {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">KafkaServer</span>();
}
}</pre>
</div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com1tag:blogger.com,1999:blog-6128525281101058933.post-60192518357699721372017-04-27T09:41:00.003-07:002017-04-27T09:41:58.342-07:00Radom thoughts by Rick Hightower: DR and multi-region<div class="p1">
DR based on region is silly for most apps and services. It is an expensive bet.</div>
<div class="p1">
<br /></div>
<div class="p1">
Multi-region is great for reducing latency for sure and DR for sure, but multi-region hot standbys is silly for most apps.</div>
<div class="p1">
<br /></div>
<div class="p1">
Mutli-AZs deployments are enough for DR IMO for 99% of use cases. </div>
<div class="p1">
If your app/service can survive a single-AZ outage, it is better than 99.999% of apps out there. </div>
<div class="p1">
<br /></div>
<div class="p1">
I am not saying to not do multi-region deploys (hot standbys), but merely that it has a cost, and your app may not need it. </div>
<div class="p1">
<br /></div>
<div class="p1">
<br /></div>
<div class="p1">
If you have a regular backup and a way to restore from another region, you are ahead of the game.</div>
<div class="p1">
</div>
<ul>
<li>frequent EBS snapshots sent to another region, </li>
<li>back things up to S3, replicate S3 bucket to S3 bucket, </li>
<li>read replicas for DBs in another region if you must</li>
</ul>
<br />
<div class="p2">
<br /></div>
<div class="p1">
For many services and applications, you don’t have to run a hot standby if you are spread across three AZs. </div>
<div class="p1">
Focus on surviving a single AZ failure. Get that right. Then focus on how to recover in another region from backups:</div>
<div class="p1">
<br /></div>
<div class="p1">
</div>
<ul>
<li>snapshot, AMIs, etc. ready to go, ready to be spun up, </li>
<li>backups to S3 with S3 bucket replication. Cheap and easy.</li>
</ul>
<br />
<div class="p1">
If all hell breaks loose, and it takes you 15 minutes to 1 hour to spin up in a new region that is a lot cheaper than running hot-standby in a second region 24/7 365 days a year. The probability of a complete region failure and the cost to your business being down for 15 minutes to an hour vs. the cost of running a second set of servers all of the time. </div>
<div class="p1">
<br /></div>
<div class="p1">
Engineers love to over engineer (especially bad ones). Hot standbys are expensive. Unless you need to run in multiple regions to reduce latency. </div>
<div class="p1">
<br /></div>
<div class="p1">
If CA falls into the ocean, no one is going to care if your app serving virtual tractors is down for a few hours. </div>
<div class="p1">
If Ohio is nuked, and your app is down for an hour, no one will care that they saw the same ad twice.</div>
<div class="p1">
We can serve a default ad without personalization for an hour. </div>
<style type="text/css"> p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Helvetica Neue'; color: #454545} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Helvetica Neue'; color: #454545; min-height: 14.0px} </style>
<br />
<div class="p2">
<br /></div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com2tag:blogger.com,1999:blog-6128525281101058933.post-46192860829304108682017-04-17T09:39:00.002-07:002017-04-17T09:39:38.671-07:00Cassandra AWS Cluster with CloudFormation, bastion host, Ansible, ssh and the aws-command line<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
This Cassandra tutorial is useful for developers and <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">DevOps/DBA</em></span> staff who want to launch a Cassandra cluster in AWS.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <a href="https://github.com/cloudurable/cassandra-image" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">cassandra-image</a> project has been using Vagrant and Ansible to set up a Cassandra Cluster for local testing. Then we used <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">Packer</em></span>, <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">Ansible</em></span> and <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">EC2</em></span>. We used Packer to create AWS images in the last tutorial. In this tutorial, we will use <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">CloudFormation</em></span> to create a VPC, Subnets, security groups and more to launch a Cassandra cluster in EC2 using the AWS AMI image we created with Packer in the last article. The next two tutorials after this one, will set up Cassandra to work in multiple AZs and multiple regions using custom snitches for Cassandra.</div>
<h3 id="overview" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Overview</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
This article covers the following:</div>
<ul style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px; margin-top: 0px;">
<li style="box-sizing: border-box;">CloudFormation</li>
<li style="box-sizing: border-box;">CloudFormer</li>
<li style="box-sizing: border-box;">Setting up VPC, NAT, Subnets, CIDRs, and more</li>
<li style="box-sizing: border-box;">AWS command line tools to launch CloudFormations</li>
<li style="box-sizing: border-box;">Setting up a bastion server for ssh and ansible in AWS</li>
<li style="box-sizing: border-box;">Setting up ansible to tunnel through our bastion server to manage AWS Cassandra instances</li>
<li style="box-sizing: border-box;">Using ansible to install Oracle 8 JDK instead of OpenJDK</li>
</ul>
<h3 id="getting-started" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Getting started</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We will create a VPC, subnets, security groups and more. Then we will expand the CloudFormation as we need to set up <span style="box-sizing: border-box; font-weight: 700;">EC2Snitch</span> and <span style="box-sizing: border-box; font-weight: 700;">EC2MultiRegionSnitch</span> in later tutorials. We also set up a bastion host in our new public subnet of our new VPC. The bastion host allows us to tunnel ansible commands to our Cassandra or Kafka cluster.</div>
<h3 id="retrospective-past-articles-in-this-cassandra-cluster-devops-dba-series" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Retrospective - Past Articles in this Cassandra Cluster DevOps/DBA series</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The first tutorial in this series was about setting up a Cassandra <a href="http://cloudurable.com/blog/cassandra-image-vagrant-cluster-example/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">cluster with Vagrant</a> (also appeared on DZone with some additional content <a href="https://dzone.com/articles/setting-up-a-cassandra-cluster-with-vagrant" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">DZone Setting up a Cassandra Cluster with Vagrant</a>. The second tutorial in this series was about <a href="http://cloudurable.com/blog/cassandra-image-vagrant-cluster-example/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">setting up SSL for a Cassandra cluster using Vagrant</a> (which also appeared with more content as <a href="https://dzone.com/articles/setting-up-a-cassandra-cluster-with-ssl" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">DZone Setting up a Cassandra Cluster with SSL</a>). The <a href="http://cloudurable.com/blog/ansible-cassandra-vagrant-devops/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">third article</a> in this series was about configuring and using Ansible (building on the first two articles). This last article (the 4th) <a href="http://cloudurable.com/blog/aws-ansible-packer-ssh-for-devops/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Cassandra Tutorial: AWS Ansible Packer and the AWS command line</a> covered applying the tools and techniques from the first three articles to produce an image (EC2 AMI to be precise) that we can deploy to AWS/EC2. This article uses that AWS AMI image and deploys it into a VPC that we create with CloudFormation.</div>
<h3 id="where-do-you-go-if-you-have-a-problem-or-get-stuck" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Where do you go if you have a problem or get stuck?</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We set up a <a href="https://groups.google.com/forum/#!forum/cassandra-image-project" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">google group</a> for this project and set of articles. If you just can’t get something to work or you are getting an error message, please report it <a href="https://github.com/cloudurable/cassandra-image/issues" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">here</a>. Between the mailing list and the github issues, we can support you with quite a few questions and issues. You can also find new articles in this series by following Cloudurable™ at our <a href="https://www.linkedin.com/company/17964258/" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">LinkedIn page</a>, <a href="https://www.facebook.com/cloudurable/" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Facebook page</a>, <a href="https://plus.google.com/116648719730180908239" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Google plus</a> or <a href="https://twitter.com/cloudurable" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Twitter</a>.</div>
<h3 id="creating-a-simple-vpc-with-one-private-subnet-and-one-public-subnet-for-our-cassandra-cluster" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Creating a simple VPC with one private subnet and one public subnet for our <span style="box-sizing: border-box;"><em style="box-sizing: border-box;">Cassandra Cluster</em></span></h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We describe the process here to create a <a href="http://cloudurable.com/blog/aws-vpc/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">VPC</a>, but we have a script (CloudFormation template) to save you the trouble.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Recall that an <span style="box-sizing: border-box; font-weight: 700;">AWS VPC</span> is a virtual private cloud. You can create multiple Amazon VPCs within a region that spans multiple availability zones, which is useful for <span style="box-sizing: border-box; font-weight: 700;">Amazon Cassandra deploys</span> and <span style="box-sizing: border-box; font-weight: 700;">Amazon Kafka deploys</span>. A VPC is an isolated area to deploy EC2 instances.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s create a new VPC for our cluster (Kafka or Cassandra). To start things off, you can use the <a href="http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/AmazonVPC.CreatingVPC.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">AWS VPC creation wizard</a>. Before you do that, create an elastic IP, which you will use for the <a href="http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-nat-gateway.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">NatGateway</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Recall that Amazon EC2 instances launched in a private subnet cannot access the Internet to do updates unless there is a NAT gateway. A NAT is a network address translator. Even if you wanted to update your Cassandra or Kafka EC2 instances with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">yum install foo</code>, you could not do it because they have no route to the public Internet. AWS provides NAT gateways which are similar to IGW, but unlike IGWs they do not allow incoming traffic, but rather only allow responses to outgoing traffic from your Amazon EC2 instances.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Before we create the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">NatGateway</code>, we need to create an EIP. First created a new EIP to associate with the new VPC. The wizard will ask you to select a VPC template, pick the one with one private network and one public network. It will ask you for the EIP id for the NatGateway.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Recall that an EIP is an Elastic IP Address which is a public IP address. AWS has a pool of public IP addresses available to rent per region, and an EIP is taken from this pool.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Don’t worry, we did all of this and created a <a href="https://aws.amazon.com/cloudformation/" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">CloudFormation</a> template which we will cover in a bit, you can use the CloudFormation script instead of the Wizard, but we want to describe how we created the CloudFormation template.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
When using the VPC wizard, it says it is waiting for a Nat Gateway, but the NatGateway seems to be waiting for a subnet, but it is not. All you need is the EIP to give the VPC wizard; then it creates the NatGateway for you.</div>
<h2 id="using-cloudformer" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 30px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Using CloudFormer</h2>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
After you are done creating something in AWS/EC2 that you want to automate, do the following. Tag all of the resources, VPC, NAT gateway, etc. For this, we used <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">cloudgen=cassandra-test</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Then you want to start up the <a href="http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-using-cloudformer.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">AWS CloudFormer</a>. To do this in the AWS Console, go to <span style="box-sizing: border-box; font-weight: 700;">CloudFormation</span>, you should see a wizard, pick the 3rd option down on the home page of CloudFormation (or pick create-stack and choose CloudFormer from templates examples dropdown). Then select the CloudFormer from the list of templates. Run this CloudFormation template to create a CloudFormer. Give a username and password that you will have to use later. Run the CloudFormation for <span style="box-sizing: border-box; font-weight: 700;">CloudFormer</span> (very meta). After the CloudFormation completes, go to the link provided in the CloudFormation stack output. Enter your username and password. Launch the CloudFormer, then use the filter on <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">cloudgen=cassandra-test</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Walk through the wizard, and it will create a CloudFormation template that you can run (it won’t work, but it gets you 99% of the way there as CloudFormer came out before NatGateways). First time I created it the username and password did not work. I had to pick a shorter password. After you are done creating your <span style="box-sizing: border-box; font-weight: 700;">CloudFormation</span> stack, then you can shutdown the <span style="box-sizing: border-box; font-weight: 700;">CloudFormer</span></div>
<h2 id="cloudformation-template-created-from-cloudformer" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 30px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
CloudFormation template created from CloudFormer</h2>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Here is the CloudFormation template that we derived from the above process with some edits to make it more readable.</div>
<h4 id="cloudformation-template" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
CloudFormation Template</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-javascript" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Setup VPC for Cassandra",
"Resources": {
"vpcMain": {
"Type": "AWS::EC2::VPC",
"Properties": {
"CidrBlock": "10.0.0.0/16",
"InstanceTenancy": "default",
"EnableDnsSupport": "true",
"EnableDnsHostnames": "true",
"Tags": [
{
"Key": "cloudgen",
"Value": "cassandra-test"
},
{
"Key": "Name",
"Value": "CassandraTestCluster"
}
]
}
},
"subnetPublic": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"CidrBlock": "10.0.0.0/24",
"AvailabilityZone": "us-west-2a",
"VpcId": {
"Ref": "vpcMain"
},
"Tags": [
{
"Key": "cloudgen",
"Value": "cassandra-test"
},
{
"Key": "Name",
"Value": "Public subnet"
}
]
}
},
"subnetPrivate": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"CidrBlock": "10.0.1.0/24",
"AvailabilityZone": "us-west-2a",
"VpcId": {
"Ref": "vpcMain"
},
"Tags": [
{
"Key": "cloudgen",
"Value": "cassandra-test"
},
{
"Key": "Name",
"Value": "Private subnet"
}
]
}
},
"internetGateway": {
"Type": "AWS::EC2::InternetGateway",
"Properties": {
}
},
"dhcpOptions": {
"Type": "AWS::EC2::DHCPOptions",
"Properties": {
"DomainName": "us-west-2.compute.internal",
"DomainNameServers": [
"AmazonProvidedDNS"
]
}
},
"networkACL": {
"Type": "AWS::EC2::NetworkAcl",
"Properties": {
"VpcId": {
"Ref": "vpcMain"
},
"Tags": [
{
"Key": "cloudgen",
"Value": "cassandra-test"
},
{
"Key": "Name",
"Value": "CassandraTestNACL"
}
]
}
},
"routeTableMain": {
"Type": "AWS::EC2::RouteTable",
"Properties": {
"VpcId": {
"Ref": "vpcMain"
},
"Tags": [
{
"Key": "cloudgen",
"Value": "cassandra-test"
}
]
}
},
"routeTablePublic": {
"Type": "AWS::EC2::RouteTable",
"Properties": {
"VpcId": {
"Ref": "vpcMain"
},
"Tags": [
{
"Key": "cloudgen",
"Value": "cassandra-test"
}
]
}
},
"eipForNatGateway": {
"Type": "AWS::EC2::EIP",
"Properties": {
"Domain": "vpc"
}
},
"natGateway": {
"Type": "AWS::EC2::NatGateway",
"Properties": {
"AllocationId": {
"Fn::GetAtt": [
"eipForNatGateway",
"AllocationId"
]
},
"SubnetId": {
"Ref": "subnetPublic"
}
}
},
"securityGroupDefault": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": "default VPC security group",
"VpcId": {
"Ref": "vpcMain"
},
"Tags": [
{
"Key": "cloudgen",
"Value": "cassandra-test"
},
{
"Key": "Name",
"Value": "CassandraTestSG"
}
]
}
},
"aclEntryAllowAllEgress": {
"Type": "AWS::EC2::NetworkAclEntry",
"Properties": {
"CidrBlock": "0.0.0.0/0",
"Egress": "true",
"Protocol": "-1",
"RuleAction": "allow",
"RuleNumber": "100",
"NetworkAclId": {
"Ref": "networkACL"
}
}
},
"aclEntryAllowAllIngress": {
"Type": "AWS::EC2::NetworkAclEntry",
"Properties": {
"CidrBlock": "0.0.0.0/0",
"Protocol": "-1",
"RuleAction": "allow",
"RuleNumber": "100",
"NetworkAclId": {
"Ref": "networkACL"
}
}
},
"subnetAclAssociationPublic": {
"Type": "AWS::EC2::SubnetNetworkAclAssociation",
"Properties": {
"NetworkAclId": {
"Ref": "networkACL"
},
"SubnetId": {
"Ref": "subnetPublic"
}
}
},
"subnetAclAssociationPrivate": {
"Type": "AWS::EC2::SubnetNetworkAclAssociation",
"Properties": {
"NetworkAclId": {
"Ref": "networkACL"
},
"SubnetId": {
"Ref": "subnetPrivate"
}
}
},
"vpcGatewayAttachment": {
"Type": "AWS::EC2::VPCGatewayAttachment",
"Properties": {
"VpcId": {
"Ref": "vpcMain"
},
"InternetGatewayId": {
"Ref": "internetGateway"
}
}
},
"subnetRouteTableAssociationPublic": {
"Type": "AWS::EC2::SubnetRouteTableAssociation",
"Properties": {
"RouteTableId": {
"Ref": "routeTablePublic"
},
"SubnetId": {
"Ref": "subnetPublic"
}
}
},
"routeNatGateway": {
"Type": "AWS::EC2::Route",
"Properties": {
"DestinationCidrBlock": "0.0.0.0/0",
"NatGatewayId": {
"Ref": "natGateway"
},
"RouteTableId": {
"Ref": "routeTableMain"
}
}
},
"routeInternetGateway": {
"Type": "AWS::EC2::Route",
"Properties": {
"DestinationCidrBlock": "0.0.0.0/0",
"RouteTableId": {
"Ref": "routeTablePublic"
},
"GatewayId": {
"Ref": "internetGateway"
}
},
"DependsOn": "vpcGatewayAttachment"
},
"vpcDHCPOptionsAssociation": {
"Type": "AWS::EC2::VPCDHCPOptionsAssociation",
"Properties": {
"VpcId": {
"Ref": "vpcMain"
},
"DhcpOptionsId": {
"Ref": "dhcpOptions"
}
}
},
"securityGroupIngressDefault": {
"Type": "AWS::EC2::SecurityGroupIngress",
"Properties": {
"GroupId": {
"Ref": "securityGroupDefault"
},
"IpProtocol": "-1",
"SourceSecurityGroupId": {
"Ref": "securityGroupDefault"
}
}
},
"securityGroupEgressDefault": {
"Type": "AWS::EC2::SecurityGroupEgress",
"Properties": {
"GroupId": {
"Ref": "securityGroupDefault"
},
"IpProtocol": "-1",
"CidrIp": "0.0.0.0/0"
}
}
}
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We define the following resources in the above <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">CloudFormation</em></span> which was generated with <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">CloudFormer</em></span>.</div>
<ul style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px; margin-top: 0px;">
<li style="box-sizing: border-box;"><code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">vpcMain</code> which is the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">VPC</code> with CIDR <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">10.0.0.0/16</code></li>
<li style="box-sizing: border-box;"><code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">subnetPublic</code> which is the public <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">Subnet</code> with CIDR <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">10.0.0.0/24</code></li>
<li style="box-sizing: border-box;"><code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">subnetPrivate</code> which is the private <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">Subnet</code> with CIDR <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">10.0.1.0/24</code></li>
<li style="box-sizing: border-box;"><code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">internetGateway</code> of type <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">InternetGateway</code></li>
<li style="box-sizing: border-box;"><code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">dhcpOptions</code> of type <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">DHCPOptions</code></li>
<li style="box-sizing: border-box;"><code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">networkACL</code> of type <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">NetworkAcl</code></li>
<li style="box-sizing: border-box;"><code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">natGateway</code> of type <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">NatGateway</code></li>
<li style="box-sizing: border-box;"><code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">routeTableMain</code> of type <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">RouteTable</code></li>
<li style="box-sizing: border-box;"><code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">routeTablePublic</code> of type <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">RouteTable</code></li>
<li style="box-sizing: border-box;"><code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">eipForNatGateway</code> of type <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">NatGateway</code></li>
<li style="box-sizing: border-box;"><code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">securityGroupDefault</code> of type <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">SecurityGroup</code></li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">vpcMain</code> which is the AWS <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">VPC</code> which is the VPC we use to deploy instances. The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">subnetPublic</code> (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">Subnet</code>) with CIDR <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">10.0.0.0/24</code> is a part of a VPC’s IP address range. Just like an AWS VPC you need to specify CIDR blocks for the subnets. Subnets are associated with availability zones (independent power source and network). Subnets can be public or private. A private subnet is one that is not routable from the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">internetGateway</code>. The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">subnetPrivate</code> (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">Subnet</code>) does not have a route to the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">internetGateway</code>. The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">internetGateway</code> (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">InternetGateway</code>) enables traffic from the public Internet to the mainVPC VPC. The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">internetGateway</code> (IGW) does network address translation from public IPs of EC2 instances to their private IP for incoming traffic. When an EC2 instance sends IP traffic from a public subnet, the IGW acts as the NAT for the public subnet and translates the reply address to the EC2 instance’s public IP (EIP). The IGW keep track of the mappings of EC2 instances private IP address and their public IP address. AWS ensures that the IGW is highly available and handles the horizontal scale, redundancy as needed. The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">dhcpOptions</code> (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">DHCPOptions</code>) is associated with with the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">mainVPC</code> and is used for Dynamic Host Configuration Protocol (DHCP) config and provides a standard for configuring TCP/IP networks. The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">networkACL</code>(<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">NetworkAcl</code>) - Network ACL Control List (NACL) - is a stateless layer of security for subnets. NACLs act as a stateless firewall. The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">natGateway</code> (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">NatGateway</code>) are similar to IGW, but unlike IGWs they do not allow incoming traffic, but rather only allow responses to outgoing traffic from your Amazon EC2 instances. NAT gateways are simple to manage and highly available. The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">securityGroupDefault</code> (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">SecurityGroup</code>) provides a stateful firewall that is applied directly to EC2 instance, ELBs and Autoscale group launches. For more details of what the above CloudFormation creates and why see this <a href="http://cloudurable.com/blog/aws-vpc/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">short guide to VPC</a> or this <a href="http://cloudurable.com/ppt/amazon-cassandra-notes.pdf" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">AWS Cassandra deployment guide</a>.</div>
<h4 id="using-the-new-cloudformation" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Using the new CloudFormation</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We use the <a href="http://docs.aws.amazon.com/cli/latest/reference/cloudformation/index.html#cli-aws-cloudformation" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">AWS CloudFormation CommandLine</a> to create the VPC, subnets, Network ACL, etc. for our <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">Cassandra Cluster</em></span>.</div>
<h4 id="using-aws-cloudformation-command-line-to-create-vpc-for-cassandra-cluster-or-kafka-cluster" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Using aws cloudformation command line to create VPC for Cassandra Cluster or Kafka Cluster</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/usr/bin/env bash
set -e
source bin/ec2-env.sh
aws --region ${REGION} s3 cp cloud-formation/vpc.json s3://$CLOUD_FORMER_BUCKET
aws --region ${REGION} cloudformation create-stack --stack-name ${ENV}-vpc-cassandra \
--template-url "https://s3-us-west-2.amazonaws.com/$CLOUD_FORMER_BUCKET/vpc.json" \
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that we upload the <span style="box-sizing: border-box; font-weight: 700;">CloudFormation</span> template to <span style="box-sizing: border-box; font-weight: 700;">S3</span> using the AWS command-line. Then, we call <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">create-stack</code> to run the <span style="box-sizing: border-box; font-weight: 700;">CloudFormation</span> stack. This is our base VPC setup. We will add to it as we continue.</div>
<h2 id="modifying-ec2-env-sh" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 30px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Modifying ec2-env.sh</h2>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We added three more variables to our <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ec2-env.sh</code> file as follows:</div>
<h4 id="ec2-env-sh-key-name-aws-region-name-of-s3-bucket-to-store-cloudformation-templates" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
ec2-env.sh - KEY name, aws REGION, name of S3 bucket to store CloudFormation templates</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">export KEY=KEY_NAME_CASSANDRA
export REGION=us-west-2
export CLOUD_FORMER_BUCKET=cloudurable-cloudformer-templates
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
You might recall that our <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ec2-env.sh</code> file specifies security groups id, subnets id, IAM profile name, etc.</div>
<h3 id="adding-outputs-to-cloudformation" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Adding outputs to CloudFormation.</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We were using the default VPC, but now we want to use the VPC, subnet, etc. that we just created. We could just look that up using the console, but a better way would be to add <a href="http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/outputs-section-structure.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;"><span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">Outputs</em></span> to our VPC CloudFormation</a> for our <em style="box-sizing: border-box;">Cassandra Cluster</em>. The <em style="box-sizing: border-box;">CloudFormation Outputs</em> section declares values that can be imported into other CloudFormation stacks or queried by the command line, or just displayed in the <em style="box-sizing: border-box;">AWS CloudFormation</em> console.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
CloudFormation templates once deployed are called CloudFormation stacks. CloudFormation stacks can depend on outputs from other CloudFormation stacks.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Here are the updates we make to create output variables from our <span style="box-sizing: border-box; font-weight: 700;">CloudFormation</span>.</div>
<h4 id="cloud-formation-vpc-json-adding-output-variables" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
cloud-formation/vpc.json - adding output variables</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Setup VPC for Cassandra",
"Outputs": {
"subnetPublicOut": {
"Description": "Subnet Public Id",
"Value": {
"Ref": "subnetPublic"
},
"Export": {
"Name": {
"Fn::Sub": "${AWS::StackName}-subnetPublic"
}
}
},
"subnetPrivateOut": {
"Description": "Subnet Private Id",
"Value": {
"Ref": "subnetPrivate"
},
"Export": {
"Name": {
"Fn::Sub": "${AWS::StackName}-subnetPrivate"
}
}
}
},
...
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice we put our output variable under the key <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">"Outputs"</code> which is a map of output variables.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We define <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">subnetPublicOut</code> and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">subnetPrivateOut</code> which get exported to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">${AWS::StackName}-subnetPublic</code> and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">${AWS::StackName}-subnetPrivate</code></div>
<h3 id="setting-up-bastion-security-group" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Setting up Bastion Security Group</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Next, we need to setup the Security Group for our Bastion Host. A Bastion host will allow us to manage the Cassandra/Kafka EC2 instances from ssh/ansible. It is the bridge to out private subnet where we will keep our Cassandra/Kafka EC2 instances.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The security group needs to open up port 22 as follows.</div>
<h4 id="cloud-formation-vpc-json-bastion-security-group-for-ansible-mgmt-of-cassandra-database-nodes" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
cloud-formation/vpc.json - Bastion Security Group for Ansible Mgmt of Cassandra Database Nodes</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-javascript" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;"> "securityGroupBastion": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": "Security group for bastion server.",
"VpcId": {
"Ref": "vpcMain"
},
"SecurityGroupIngress": [
{
"IpProtocol": "tcp",
"FromPort": "22",
"ToPort": "22",
"CidrIp": "0.0.0.0/0"
}
],
"SecurityGroupEgress": [
{
"IpProtocol": "-1",
"CidrIp": "0.0.0.0/0"
}
],
"Tags": [
{
"Key": "Name",
"Value": "bastionSecurityGroup"
},
{
"Key": "cloudgen",
"Value": "cassandra-test"
}
]
}
},
</code></pre>
<h3 id="setting-up-security-group-for-cassandra-nodes" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Setting up Security Group for Cassandra Nodes</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
This example will focus on Cassandra nodes, not Kafka, but the ideas are similar. This security group uses the CIDR of the VPC to open up all traffic to all subnets in this VPC.</div>
<h4 id="cloud-formation-vpc-json-security-group-for-cassandra-database-nodes-in-cassandra-cluster" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
cloud-formation/vpc.json - Security group for Cassandra Database nodes in Cassandra Cluster</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-javascript" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;"> "securityGroupCassandraNodes": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": "Security group for Cassandra Database nodes in Cassandra Cluster",
"VpcId": {
"Ref": "vpcMain"
},
"SecurityGroupIngress": [
{
"IpProtocol": "-1",
"CidrIp": "10.0.0.0/8"
}
],
"SecurityGroupEgress": [
{
"IpProtocol": "-1",
"CidrIp": "0.0.0.0/0"
}
],
"Tags": [
{
"Key": "Name",
"Value": "cassandraSecurityGroup"
},
{
"Key": "cloudgen",
"Value": "cassandra-test"
}
]
}
}
</code></pre>
<h3 id="output-new-security-groups-as-cloudformation-outputs" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Output new security groups as CloudFormation outputs.</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We will want to add the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">securityGroupCassandraNodes</code> and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">securityGroupBastion</code> to the output of the <span style="box-sizing: border-box; font-weight: 700;">CloudFormation</span> so we can use it from our AWS EC2 scripts.</div>
<h4 id="cloud-formation-vpc-json-output-new-security-groups" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
cloud-formation/vpc.json - output new security groups</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Setup VPC for Cassandra Cluster for Cassandra Database",
"Outputs": {
"subnetPublicOut": {
"Description": "Subnet Public Id",
"Value": {
"Ref": "subnetPublic"
},
"Export": {
"Name": {
"Fn::Sub": "${AWS::StackName}-subnetPublic"
}
}
},
"subnetPrivateOut": {
"Description": "Subnet Private Id",
"Value": {
"Ref": "subnetPrivate"
},
"Export": {
"Name": {
"Fn::Sub": "${AWS::StackName}-subnetPrivate"
}
}
},
"securityGroupBastionOutput": {
"Description": "Security Group Bastion for managing Cassandra Cluster Nodes with Ansible",
"Value": {
"Ref": "securityGroupBastion"
},
"Export": {
"Name": {
"Fn::Sub": "${AWS::StackName}-securityGroupBastion"
}
}
},
"securityGroupCassandraNodesOutput": {
"Description": "Cassandra Database Node security group for Cassandra Cluster",
"Value": {
"Ref": "securityGroupCassandraNodes"
},
"Export": {
"Name": {
"Fn::Sub": "${AWS::StackName}-securityGroupCassandraNodes"
}
}
}
},
...
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that we added <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">securityGroupBastionOutput</code> and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">securityGroupCassandraNodesOutput</code> to the above <span style="box-sizing: border-box; font-weight: 700;">CloudFormation</span>.</div>
<h4 id="cloudurable-specialize-in-aws-devops-automation-for-cassandra-spark-and-kafka" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Cloudurable specialize in <span style="box-sizing: border-box;">AWS DevOps Automation for Cassandra, Spark and Kafka</span></h4>
<blockquote style="background-color: white; border-left: 5px solid rgb(0, 191, 255); box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin: 0px 0px 20px; padding: 10px 20px;">
<div style="box-sizing: border-box;">
We hope you find this Cassandra tutorial useful. We also provide <a href="http://cloudurable.com/spark-aws-emr-consulting/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Spark consulting</a>, <a href="http://cloudurable.com/cassandra-aws-consulting/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Casandra consulting</a> and <a href="http://cloudurable.com/kafka-aws-consulting/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka consulting</a> to get you setup fast in AWS with CloudFormation and CloudWatch. Support us by checking out our <a href="http://cloudurable.com/spark-aws-emr-training/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Spark Training</a>, <a href="http://cloudurable.com/cassandra-course/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Casandra training</a> and <a href="http://cloudurable.com/kafka-training/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka training</a>.</div>
</blockquote>
<h3 id="updating-cloudformation" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Updating CloudFormation</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
As we iterative develop our CloudFormation, like add new security groups, we do not have to rebuild everything. Instead, we can update the CloudFormation stack. CloudFormation is smart enough to see what has changed and only add/update those areas.</div>
<h4 id="bin-update-vpc-cloudformation-sh" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
bin/update-vpc-cloudformation.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/usr/bin/env bash
set -e
source bin/ec2-env.sh
aws --region ${REGION} s3 cp cloud-formation/vpc.json s3://$CLOUD_FORMER_BUCKET
aws --region ${REGION} cloudformation update-stack --stack-name ${ENV}-vpc-cassandra \
--template-url "https://s3-us-west-2.amazonaws.com/$CLOUD_FORMER_BUCKET/vpc.json" \
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The above uses the <a href="http://docs.aws.amazon.com/cli/latest/reference/cloudformation/update-stack.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">CloudFormation Update Stack</a> to update a stack as specified by the template. After the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">update stack</code> completes successfully, the stack update starts.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We can see our output variable from our CloudFormation template from the command line as follows.</div>
<h4 id="list-the-output-variables-of-cloudformation-with-the-aws-command-line" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
List the output variables of CloudFormation with the aws command-line</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">
$ aws cloudformation describe-stacks --stack-name dev-vpc-cassandra | jq .Stacks[].Outputs[]
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Output</div>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-javascript" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">{
"Description": "Subnet Private Id",
"OutputKey": "subnetPrivateOut",
"OutputValue": "subnet-XXe5453a"
}
{
"Description": "Cassandra Database Node security group for Cassandra Cluster",
"OutputKey": "securityGroupCassandraNodesOutput",
"OutputValue": "sg-XX527048"
}
{
"Description": "Subnet Public Id",
"OutputKey": "subnetPublicOut",
"OutputValue": "subnet-XXe5453c"
}
{
"Description": "Security Group Bastion for managing Cassandra Cluster Nodes with Ansible",
"OutputKey": "securityGroupBastionOutput",
"OutputValue": "sg-XX527040"
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Then we just modify our <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ec2-env.sh</code> script to use these values. Now we can modify the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bin/ec2-env.sh</code>.</div>
<h4 id="env-file" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
env file</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/bin/bash
set -e
export REGION=us-west-2
export ENV=dev
export KEY_PAIR_NAME="cloudurable-$REGION"
export PEM_FILE="${HOME}/.ssh/${KEY_PAIR_NAME}.pem"
export SUBNET_PUBLIC=subnet-XXe5453a
export SUBNET_PRIVATE=subnet-XXe5453b
export CLOUD_FORMER_S3_BUCKET=cloudurable-cloudformer-templates
export HOSTED_ZONE_ID="XXNXWWXWZXEXHJ-NOT-REAL"
export BASTION_NODE_SIZE=t2.small
export BASTION_SECURITY_GROUP=sg-XX527040
export BASTION_AMI=ami-XXb3310e
export BASTION_EC2_INSTANCE_NAME="bastion.${ENV}.${REGION}"
export BASTION_DNS_NAME="bastion.${ENV}.${REGION}.cloudurable.com."
export CASSANDRA_NODE_SIZE=m4.large
export CASSANDRA_AMI=ami-XXb3310f
export CASSANDRA_SECURITY_GROUP=sg-XX527048
export CASSANDRA_IAM_PROFILE=IAM_PROFILE_CASSANDRA
export CASSANDRA_EC2_INSTANCE_NAME="cassandra-node.${ENV}.${REGION}"
export CASSANDRA_DNS_NAME="node0.${ENV}.${REGION}.cloudurable.com."
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Just like a war plan does not survive the first battle, variable names do not survive the first refactor to add a feature. We could also use the CloudFormation outputs as input variables to another CloudFormation as input variables.</div>
<blockquote style="background-color: white; border-left: 5px solid rgb(0, 191, 255); box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin: 0px 0px 20px; padding: 10px 20px;">
<div style="box-sizing: border-box;">
CloudFormation is the AWS way to create immutable infrastructure.</div>
</blockquote>
<h3 id="why-a-bastion-server" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Why a Bastion server</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
A <a href="https://en.wikipedia.org/wiki/Bastion_host" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">bastion host</a> is a computer that is locked down and fully exposed to attack, but in our case the Bastion has a firewall so that only port 22 is open (SSH), and in fact when we only run the bastion host when we want to ssh into our private subnet or run asnbile playbooks. The bastion host is on the public side of the DMZ.</div>
<h3 id="creating-bastion-server" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Creating bastion server</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We updated the <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">log into server bash script</em></span>, the <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">associate DNS with IP bash scripts</em></span> and the <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">get IP address of ec2 instance bash scripts</em></span> to take arguments and renamed them to work with the bastion EC2 instance and Cassandra Database instances. Then we created a new script called <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bin/create-ec2-instance-bastion.sh</code> to use the new scripts and the appropriate environment variables.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Here is the create bastion script.</div>
<h4 id="create-ec2-instance-bastion-sh-bastion-for-ansible-and-ssh-bridge" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
create-ec2-instance-bastion.sh - bastion for ansible and ssh bridge</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/bin/bash
set -e
source bin/ec2-env.sh
instance_id=$(aws ec2 run-instances --image-id "$BASTION_AMI" --subnet-id "$SUBNET_PUBLIC" \
--instance-type "$BASTION_NODE_SIZE" --iam-instance-profile "Name=$CASSANDRA_IAM_PROFILE" \
--associate-public-ip-address --security-group-ids "$BASTION_SECURITY_GROUP" \
--key-name "$KEY_PAIR_NAME" | jq --raw-output .Instances[].InstanceId)
echo "bastion ${instance_id} is being created"
aws ec2 wait instance-exists --instance-ids "$instance_id"
aws ec2 create-tags --resources "${instance_id}" --tags Key=Name,Value="${BASTION_EC2_INSTANCE_NAME}" \
Key=Role,Value="Bastion" Key=Env,Value="DEV"
echo "${instance_id} was tagged waiting to login"
aws ec2 wait instance-status-ok --instance-ids "$instance_id"
bin/associate-route53-DNS-with-IP.sh ${BASTION_EC2_INSTANCE_NAME} ${BASTION_DNS_NAME}
bin/login-ec2-instance.sh ${BASTION_EC2_INSTANCE_NAME}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
If you followed along with the previous tutorials, the above will all make sense. Essentially, we are just launching and EC2 instance using the <a href="http://cloudurable.com/blog/aws-ansible-packer-ssh-for-devops/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">AMI/image we created with packer from Packer/Ansible/Cassandra Tutorial</a>. Then the script waits for the image to become active, then it associates the bastion DNS name with the IP of this image.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Note since we are launching the Cassandra node in a private subnet, we will not be able to log into it direct any longer. We will have to log into the bastion server first.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now let’s run this script.</div>
<h4 id="running-bin-create-ec2-instance-bastion-sh-to-create-ansible-bastion-aws-ec2-instance" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Running bin/create-ec2-instance-bastion.sh - to create ansible bastion AWS ec2 instance</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ bin/create-ec2-instance-bastion.sh
bastion i-069819c22bbd379ab is being created
i-069819c22bbd379ab was tagged waiting to login
IP ADDRESS 55.222.33.66 bastion.dev.us-west-2.cloudurable.com.
{
"Changes":[
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Type": "A",
"Name": "bastion.dev.us-west-2.cloudurable.com.",
"TTL": 300,
"ResourceRecords": [{
"Value": "55.222.33.66"
}]
}
}
]
}
IP ADDRESS 55.222.33.66
ECDSA key fingerprint is SHA256:DzyRqdhPPUlTf8ZPAH6XtGe0SRNthSoMXK4cZCpGGME.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '54.202.31.60' (ECDSA) to the list of known hosts.
...
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s also startup the Cassandra Database Node which will be a seed server in the Cassandra Cluster. We do this with the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">create-ec2-instance-cassandra.sh</code> as follows.</div>
<h4 id="run-bin-create-ec2-instance-cassandra-sh-create-cassandra-instance" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Run bin/create-ec2-instance-cassandra.sh - Create Cassandra instance</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;"> $ bin/create-ec2-instance-cassandra.sh
Cassandra Database: Cassandra Cluster Node i-0602e8b4d75020438 is being created
Cassandra Node i-0602e8b4d75020438 was tagged waiting for status ready
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now we can log into the bastion.</div>
<h3 id="logging-into-bastion-running-in-our-public-subnet" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Logging into Bastion running in our public subnet</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We will log into Bastion so we can then log into our Cassandra or Kafka nodes.</div>
<h4 id="using-login-ec2-instance-sh" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Using login-ec2-instance.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ bin/login-ec2-instance.sh bastion.dev.us-west-2
centos@ip-10-0-0-220 ~]$
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The private IP address of the Cassandra instance is 10.0.1.10. To log into that server, we would first need to log into bastion (private IP 10-0-0-220) as follows.</div>
<h4 id="logging-into-cassandra-node-from-bastion" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Logging into Cassandra node from bastion</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">centos@ip-10-0-0-220 ~]$ ssh -i ~/resources/server/certs/test_rsa ansible@10.0.1.138
[ansible@ip-10-0-1-10 ~]$
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice we are using the <a href="http://cloudurable.com/blog/ansible-cassandra-vagrant-devops/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;"><code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">test_rsa</code> that we setup for ssh access from our first Ansible/Cassandra Tutorial</a>.</div>
<h3 id="ssh-setup-for-ansible-ssh-to-managed-cassandra-nodes" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
SSH Setup for Ansible/SSH to managed Cassandra nodes</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Before we setup, let’s make sure we can log into the bastion host.</div>
<h3 id="try-to-connect-to-cassandra-seed-node-via-ssh-bastion-tunnel" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Try to connect to Cassandra seed node via ssh bastion tunnel</h3>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ssh -t bastion ssh -t -i /home/ansible/.ssh/test_rsa ansible@10.0.1.10
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
This trick of ssh bastion tunneling from the cmd line was described <a href="https://10mi2.wordpress.com/2015/01/14/using-ssh-through-a-bastion-host-transparently/" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Using SSH through a Bastion tunnel</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We modify our <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/.ssh/config</code> to tunnel requests for <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">10.0.1.10</code> (our Cassandra node), through the bastion <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bastion.dev.us-west-2.cloudurable.com</code>.</div>
<h4 id="ssh-config-configure-ssh-bridge-from-bastion-to-main-cassandra-node" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/.ssh/config - configure ssh bridge from bastion to main cassandra node</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">Host *.us-west-2.compute.amazonaws.com
ForwardAgent yes
IdentityFile ~/.ssh/test_rsa
User ansible
Host bastion
Hostname bastion.dev.us-west-2.cloudurable.com
ForwardAgent yes
IdentityFile ~/.ssh/test_rsa
User ansible
Host cassandra.node0
Hostname 10.0.1.10
ForwardAgent yes
IdentityFile ~/.ssh/test_rsa
ProxyCommand ssh bastion -W %h:%p
User ansible
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
First, we create <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">Host bastion</code> alias that sets up access to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bastion.dev.us-west-2.cloudurable.com</code>, your DNS name will vary. Then we use this <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bastion</code> to tunnel ssh to the Cassandra instance using the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ProxyCommand</code>. This <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ProxyCommand</code> of the <a href="https://linux.die.net/man/5/ssh_config" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">ssh client config</a> runs the command <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh bastion -W host:port</code> and then talks to the standard in/out of that command as if the remote connection (specified by <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">-W</code>). The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">%h</code> is for the hostname and the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">%p</code> is port.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now, <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">cassandra.node0</code> is a bit special because it is a seed server, but other Cassandra Nodes will be a bit nameless so to speak. We want a way to configure them without naming each one, and while we are at for speed we want to use <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">SSH multiplexing</em></span>.</div>
<h4 id="ssh-config-create-the-ssh-bridge-for-the-rest-of-the-cassandra-nodes" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/.ssh/config - create the ssh bridge for the rest of the Cassandra nodes</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">Host 10.0.1.*
ForwardAgent yes
IdentityFile ~/.ssh/test_rsa
ProxyCommand ssh bastion -W %h:%p
User ansible
ControlMaster auto
ControlPath ~/.ssh/ansible-%r@%h:%p
ControlPersist 5m
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Ideas for this setup came from the <a href="http://blog.scottlowe.org/2015/12/24/running-ansible-through-ssh-bastion-host/" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">running ansible through SSH bastion host</a> on Scott’s WebLog.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We make the following changes to get ansible to work through bastion.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We add this workaround to ansible.cfg.</div>
<h4 id="ansible-cfg" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
ansible.cfg -</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">...
[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=30m
control_path = %(directory)s/%%h-%%p-%%r
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Then we setup our inventory file with our new ssh aliases that we defined in <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/.ssh/config</code> earlier.</div>
<h4 id="inventory-ini" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
inventory.ini -</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">[cassandra-nodes]
cassandra.node0
[bastion]
bastion
</code></pre>
<h3 id="side-note-local-to-project-ssh-config-so-we-can-check-it-in" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Side Note: Local to project ssh config so we can check it in.</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
If we use the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">-F</code> parameter as an <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh_args</code> in <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible.cfg/ssh_connection</code> then we can specify . Now we can keep all these files in source control by adding this <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">[ssh_connection] \n ssh_args = -F ssh/ssh.config -o ControlMaster=auto -o ControlPersist=30m</code> to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible.cfg</code> in the project directory of this Ansible/Cassandra tutorial. This is another trick from <a href="http://blog.scottlowe.org/2015/12/24/running-ansible-through-ssh-bastion-host/" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">blog post: running ansible through a bastion</a></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The next thing we want to do is ping our servers with ansible just to show that it is working.</div>
<h3 id="running-ansible-commands-via-a-bastion-server" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Running ansible commands via a bastion server</h3>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">
$ ansible cassandra.node0 -m ping
cassandra.node0 | SUCCESS => {
"changed": false,
"ping": "pong"
}
$ ansible bastion -m ping
bastion | SUCCESS => {
"changed": false,
"ping": "pong"
}
</code></pre>
<h3 id="installing-more-than-one-node-to-the-cluster" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Installing more than one node to the cluster</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
To automate the configuration of the Cassandra instances, we will use <a href="https://github.com/cloudurable/cassandra-cloud" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Cassandra Cloud</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">CassandraCloud</em></span> is a tool that helps you configure Cassandra for clustered environments. It works well in <em style="box-sizing: border-box;">Docker</em>, <em style="box-sizing: border-box;">AWS</em>, <em style="box-sizing: border-box;">Mesos</em>, <em style="box-sizing: border-box;">EC2</em>, and <em style="box-sizing: border-box;">VirtualBox</em> environments (and similar environments). It allows you to configure Cassandra easily. For example, it could be kicked off as a <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">USER_DATA</code> script in Amazon EC2 (AWS EC2). <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">CassandraCloud</em></span> usually runs once when an instance is first launched and then never again. <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">CassandraCloud</em></span> allows you to override values via <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">OS ENVIRONMENT</em></span> variables. There is an HCL config file, and there are command line arguments. The HCL config file can be overridden with <span style="box-sizing: border-box; font-weight: 700;">ENVIRONMENT</span> which can be overridden with command line arguments.<span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">CassandraCloud</em></span> will generate <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">${CASSANDRA_HOME}/conf/cassandra.yaml</code> file. You can specify a custom template (usually found in <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">${CASSANDRA_HOME}/conf/cassandra-yaml.template</code>).</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Here is the EC2 user-data script where we invoke <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">Cassandra Cloud</em></span>.</div>
<h4 id="resources-user-data-cassandra-aws-ec2-user-data-for-cassandra-database-node" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
resources/user-data/cassandra - AWS EC2 User-Data for Cassandra Database Node</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/bin/bash
set -e
export BIND_IP=`curl http://169.254.169.254/latest/meta-data/local-ipv4`
/opt/cloudurable/bin/cassandra-cloud -cluster-name test \
-client-address ${BIND_IP} \
-cluster-address ${BIND_IP} \
-cluster-seeds 10.0.1.10
/bin/systemctl restart cassandra
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice we are passing the client <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">BIND_IP</code> which we get form the <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">EC2 meta-data</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We also added an extra param to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bin/create-ec2-instance-cassandra.sh</code> so we can pin a deployment to a certain IP in the CIDR range of our AWS VPC private subnet.</div>
<h4 id="bin-create-ec2-instance-cassandra-sh" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
bin/create-ec2-instance-cassandra.sh -</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/bin/bash
set -e
source bin/ec2-env.sh
if [ -z "$1" ]
then
PRIVATE_IP_ADDRESS=10.0.1.10
else
PRIVATE_IP_ADDRESS=$1
fi
instance_id=$(aws ec2 run-instances --image-id "$CASSANDRA_AMI" --subnet-id "$SUBNET_PRIVATE" \
--instance-type ${CASSANDRA_NODE_SIZE} --private-ip-address ${PRIVATE_IP_ADDRESS} \
--iam-instance-profile "Name=$CASSANDRA_IAM_PROFILE" \
--security-group-ids "$CASSANDRA_SECURITY_GROUP" \
--user-data file://resources/user-data/cassandra \
--key-name "$KEY_PAIR_NAME" | jq --raw-output .Instances[].InstanceId)
echo "Cassandra Database: Cassandra Cluster Node ${instance_id} is being created"
aws ec2 wait instance-exists --instance-ids "$instance_id"
aws ec2 create-tags --resources "${instance_id}" --tags Key=Name,Value="${CASSANDRA_EC2_INSTANCE_NAME}" \
Key=Cluster,Value="Cassandra" Key=Role,Value="Cassandra_Database_Cluster_Node" Key=Env,Value="DEV"
echo "Cassandra Node ${instance_id} was tagged waiting for status ready"
aws ec2 wait instance-status-ok --instance-ids "$instance_id"
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
If you run it with no IP address, then it creates the Cassandra Seed Node EC2 instance. If you run it with an IP address, then it creates and instance with that private IP. (Note that the IP must be in the range of the private subnet CIDR that we created earlier.)</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s use this to create a second instance.</div>
<h4 id="running-bin-create-ec2-instance-cassandra-sh-10-0-1-11" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Running bin/create-ec2-instance-cassandra.sh 10.0.1.11</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ bin/create-ec2-instance-cassandra.sh 10.0.1.11
Cassandra Database: Cassandra Cluster Node i-0e9939e9f62ae33d4 is being created
Cassandra Node i-0e9939e9f62ae33d4 was tagged waiting for status ready
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now we need to make sure it is working. We can do this by ssh-ing into <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">cassandra.node0</code> and checking the status and describing the cassandra cluster with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">nodetool</code>.</div>
<h4 id="check-cassandra-node-status-with-nodetool" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Check <em style="box-sizing: border-box;">Cassandra node</em> status with nodetool</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ssh cassandra.node0
...
$ /opt/cassandra/bin/nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.0.1.10 132.06 KiB 32 100.0% 95794596-7dbe-4ec9-8c35-f4f49a5bb999 rack1
UN 10.0.1.11 94.97 KiB 32 100.0% eb35bb65-c582-4fa9-9069-fd5222830c99 rack1
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that both <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">10.0.1.10</code> (the seed server), and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">10.0.1.11</code> are seen. We will setup a seed server per availability zone.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s also see the Cassandra cluster with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">nodetool status</code>.</div>
<h4 id="check-cassandra-node-status-with-nodetool-1" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Check <em style="box-sizing: border-box;">Cassandra node</em> status with nodetool</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">[ansible@ip-10-0-1-10 ~]$ /opt/cassandra/bin/nodetool describecluster
Cluster Information:
Name: test
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
86afa796-d883-3932-aa73-6b017cef0d19: [10.0.1.10, 10.0.1.11]
</code></pre>
<h3 id="up-next" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Up next</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
In the next article we will use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">Ec2Snitch</code> and setup a second subnet and availability zone.</div>
<h3 id="bonus-lap-using-an-ansible-playbook-to-install-oracle-jdk-8" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Bonus lap using an ansible playbook to install Oracle JDK 8</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
A big mandate came down from the corporate home office, switch all instances from OpenJDK to the Oracle JDK. We protest that we ran benchmarks and burn-ins with the OpenJDK and there is no difference. The home office has silenced our pleas. No remorse.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We are using the OpenJDK and Cassandra gives warning messages not to. Let’s use Ansible to fix that. We will use a <a href="https://gist.github.com/andershedstrom/7c7d0bb5b9450c54a907" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">ansible playbook for installing jdk-8 on CentOS</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
First let’s add our extra Cassandra node <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">10.0.1.11</code> to the list of <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">cassandra-nodes</code> that we are managing.</div>
<h4 id="inventory-ini-add-new-cassandra-node-to-file" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
inventory.ini - add new Cassandra node to file</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">[cassandra-nodes]
cassandra.node0
10.0.1.11
...
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Then let’s create an ansible playbook that installs the Oracle JDK on our Cassandra nodes.</div>
<h4 id="playbooks-oracle-8-jdk-yml" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
playbooks-oracle-8.jdk.yml</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">---
- hosts: cassandra-nodes
gather_facts: no
become: true
remote_user: ansible
vars:
download_url: http://download.oracle.com/otn-pub/java/jdk/8u121-b13/e9e7ea248e2c4826b92b3f075a80e441/jdk-8u121-linux-x64.tar.gz
download_folder: /opt
java_name: "{{download_folder}}/jdk1.8.0_121"
java_archive: "{{download_folder}}/jdk-8u121-linux-x64.tar.gz"
tasks:
- name: Download Java
command: "curl -L -b 'oraclelicense=a' {{download_url}} -o {{java_archive}} creates={{java_archive}}"
- name: Unpack archive
command: "tar -zxf {{java_archive}} -C {{download_folder}} creates={{java_name}}"
- name: Fix ownership
file: state=directory path={{java_name}} owner=root group=root recurse=yes
- name: Remove previous
command: 'alternatives --remove "java" /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/jre/bin/java'
- name: Make Java available for system
command: 'alternatives --install "/usr/bin/java" "java" "{{java_name}}/bin/java" 2000'
- name: Clean up
file: state=absent path={{java_archive}}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Next up we just need to run the playbook.</div>
<h4 id="running-playbook" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Running playbook</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ansible-playbook playbooks/install-oracle-8-jdk.yml
PLAY [cassandra-nodes] *********************************************************
TASK [Download Java] ***********************************************************
changed: [cassandra.node0]
[WARNING]: Consider using get_url or uri module rather than running curl
changed: [10.0.1.11]
TASK [Unpack archive] **********************************************************
changed: [cassandra.node0]
[WARNING]: Consider using unarchive module rather than running tar
changed: [10.0.1.11]
TASK [Fix ownership] ***********************************************************
changed: [10.0.1.11]
changed: [cassandra.node0]
TASK [Remove previous] *********************************************************
changed: [cassandra.node0]
changed: [10.0.1.11]
TASK [Make Java available for system] ******************************************
changed: [10.0.1.11]
changed: [cassandra.node0]
TASK [Clean up] ****************************************************************
changed: [cassandra.node0]
changed: [10.0.1.11]
PLAY RECAP *********************************************************************
10.0.1.11 : ok=6 changed=6 unreachable=0 failed=0
cassandra.node0 : ok=6 changed=6 unreachable=0 failed=0
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now imagine that we did not just have two servers but 50. This playbook is a lot nicer.</div>
<h2 id="conclusion" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 30px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Conclusion</h2>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We used <span style="box-sizing: border-box; font-weight: 700;">CloudFormer</span> to create a starter <span style="box-sizing: border-box; font-weight: 700;">CloudFormation</span> template which created an AWS VPC, NAT, Subnets, InternetGateway, CIDRs, and more. Then we used <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">AWS command line tools</em></span> to launch CloudFormations as a stack. Then we added some additional security groups. Then we used <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">AWS command line tools</em></span> to update the CloudFormations stack that we set up earlier. We set up a <span style="box-sizing: border-box; font-weight: 700;">bastion host</span> which allows us to tunnel ansible commands. We then used the <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">AWS command line</em></span>. We then set up a Cassandra cluster using <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">Cassandra Cloud</em></span>, EC2 USER DATA script, and EC2 instance meta-data to generate Cassandra YAML config. Then as a bonus lap, we used ansible to run a playbook to replace our OpenJDK usage with the Oracle JDK.</div>
<h4 id="more-about-cloudurable-trade" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
More about Cloudurable™</h4>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Cloudurable is focused on AWS DevOps automation for Cassandra and Kafka. We also focus on supporting the SMACK stack in AWS. All of our training and mentoring has a special focus on AWS deployments. Our Apache Spark course, for example, covers running Spark on EMR and using Spark SQL with data from S3 and DynamoDB as well as using Spark Streaming with Kinesis. We also cover how to use Cassandra from Spark (for example).</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">Consulting</em></span></div>
<ul style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px; margin-top: 0px;">
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/cassandra-aws-consulting/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Cassandra consulting</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/kafka-aws-consulting/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka consulting</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/dynamodb-consulting/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">DynamoDB consulting</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/kinesis-consulting/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kinesis consulting</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/spark-aws-emr-consulting/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Apache Spark on AWS/EMR consulting</a></li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">Training</em></span></div>
<ul style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px; margin-top: 0px;">
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/cassandra-course/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Cassandra training</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/kafka-training/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kafka training</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/dynamodb-training/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">DynamoDB training</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/kinesis-training/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Kinesis training</a></li>
<li style="box-sizing: border-box;"><a href="http://cloudurable.com/spark-aws-emr-training/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Apache Spark and EMR training</a></li>
</ul>
<h3 id="resources" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Resources</h3>
<ul style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px; margin-top: 0px;">
<li style="box-sizing: border-box;"><a href="http://blog.scottlowe.org/2015/12/24/running-ansible-through-ssh-bastion-host/" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Running ansible through an ssh bastion host</a> on Scott’s Weblog</li>
<li style="box-sizing: border-box;"><a href="https://alexbilbie.com/2014/07/using-ansible-with-a-bastion-host/" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Using ansible with a bastion host</a> by Alex Bilbie</li>
<li style="box-sizing: border-box;"><a href="https://medium.com/@paulskarseth/ansible-bastion-host-proxycommand-e6946c945d30#.u9j1nowx2" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Ansible Bastion Host Proxy Command</a> by Paul Skarseth</li>
<li style="box-sizing: border-box;"><a href="http://blog.scottlowe.org/2015/11/21/using-ssh-bastion-host/" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Using SSH Bastion Host</a> by Scott Lowe</li>
<li style="box-sizing: border-box;"><a href="https://10mi2.wordpress.com/2015/01/14/using-ssh-through-a-bastion-host-transparently/" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">Using SSH through a Bastion Host transparently</a></li>
<li style="box-sizing: border-box;"><a href="http://www.ipaddressguide.com/cidr" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration-line: none; transition: all 0.2s ease-out;">IP Address CIDR guide</a></li>
</ul>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com0tag:blogger.com,1999:blog-6128525281101058933.post-85851619954984590962017-03-04T14:50:00.001-08:002017-03-04T14:50:32.829-08:00Setting up Ansible for our Cassandra Database Cluster to do DevOps/DBA tasks<h2 id="setting-up-ansible-for-our-cassandra-database-cluster-to-do-devops-dba-tasks" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 30px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Setting up Ansible for our Cassandra Database Cluster to do DevOps/DBA tasks</h2>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<a href="https://www.ansible.com/" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">Ansible</a> is an essential <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">DevOps/DBA</em></span> tool for managing backups and rolling upgrades to the Cassandra cluster in AWS/EC2. An excellent aspect of Ansible is that it uses ssh, so you do not have to install an agent to use Ansible.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
This article series centers on how DevOps/DBA tasks with the Cassandra Database. However the use of Ansible for DevOps/DBA transcends its use with the Cassandra Database so this article is good information for any DevOps/DBA or Developer that needs to manage groups of instances, boxes, hosts whether they be on-prem bare-metal, dev boxes, or in the Cloud. You don’t need to be setting up Cassandra to get use of this article.</div>
<h3 id="article-series-on-devops-dba-cassandra-database" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Cassandra Tutorial Series on DevOps/DBA Cassandra Database</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The first article in this series was about setting up a Cassandra <a href="http://cloudurable.com/blog/cassandra-image-vagrant-cluster-example/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">cluster with Vagrant</a> (also appeared on DZone with some additional content <a href="https://dzone.com/articles/setting-up-a-cassandra-cluster-with-vagrant" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">DZone Setting up a Cassandra Cluster with Vagrant</a>. The second article in this series was about <a href="http://cloudurable.com/blog/cassandra-image-vagrant-cluster-example/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">setting up SSL for a Cassandra cluster using Vagrant</a> (which also appeared with more content as <a href="https://dzone.com/articles/setting-up-a-cassandra-cluster-with-ssl" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">DZone Setting up a Cassandra Cluster with SSL</a>). You don’t need those articles to follow along, but they might provide a lot of contexts. You can find the source for the first and second article at our <a href="https://github.com/cloudurable/cassandra-image" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">Cloudurable Cassandra Image for Docker, AWS, and Vagrant</a>. In later articles, we will use Ansible to create more complicated playbooks like doing a rolling Cassandra upgrade, and we will cover using Ansible/ssh with AWS EC2.</div>
<h3 id="source-code-for-vagrant-and-ansbile" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Source code for Vagrant, and ansbile</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We continue to evolve the <a href="https://github.com/cloudurable/cassandra-image" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">cassandra-image GitHub project</a>. In an effort for the code to match the listings in the article, we created a new branch where the code was when this article was written (more or less): <a href="https://github.com/cloudurable/cassandra-image/tree/article_3_ansible_cassandra_vagrant" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">Article 3 Ansible Cassandra Vagrant</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s get to it. Let’s start by creating a key for our DevOps/DBA test <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">Cassandra cluster</em></span>.</div>
<h2 id="create-key-for-test-cluster-to-do-cassandra-database-devops-dba-tasks-with-ansible" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 30px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Create key for test cluster to do Cassandra Database DevOps/DBA tasks with Ansible</h2>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
To use Ansible for DevOps/DBA, we will need to setup <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh keys</code> as Ansible uses <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh</code> instead of running an agent on each server like Chef and Puppet.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The tool <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">ssh-keygen</em></span> manages authentication keys for <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">ssh</em></span>(secure shell). The utility <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">ssh-keygen</em></span> generates RSA or DSA keys for SSH (secure shell) protocol version 1 and 2. You can specify the key type with the -t option.</div>
<h4 id="setup-key-script-bin-setupkeys-cassandra-security-sh" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
setup key script bin/setupkeys-cassandra-security.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">CLUSTER_NAME=test
...
ssh-keygen -t rsa -C "your_email@example.com" -N "" -C "setup for cloud" \
-f "$PWD/resources/server/certs/${CLUSTER_NAME}_rsa"
chmod 400 "$PWD/resources/server/certs/"*
cp "$PWD/resources/server/certs/"* ~/.ssh
...
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s break that down.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We use <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">ssh-keygen</em></span> to create a private key that we will use to log into our boxes.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
In this article those boxes are Vagrant boxes (VirtualBox), but in the next <a href="http://cloudurable.com/blog/aws-ansible-packer-ssh-for-devops/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">article</a>, we will use the same key to manage EC2 instances.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
<h4 id="use-ssh-keygen-to-create-private-key-for-ssh" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Use ssh-keygen to create private key for ssh to log into Cassandra Database nodes</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">ssh-keygen -t rsa -C "your_email@example.com" -N "" -C "setup for cloud" \
-f "$PWD/resources/server/certs/${CLUSTER_NAME}_rsa"
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Then we restrict the access to the file of the key otherwise, ansible, <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh</code> and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">scp</code> (secure copy) will not let us use it.</div>
<h4 id="change-the-access-of-the-key" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Change the access of the key</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">
chmod 400 "$PWD/resources/server/certs/"*
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The above <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">chmod 400</code> changes the cert files so only the owner can read the file. This file change mod makes sense. The certification file should be private to the user (and that is what <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">400</code> does).</div>
<h4 id="copy-keys-to-area-where-it-will-be-copied-by-provisioning" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Copy keys to area where it will be copied by Cassandra node provisioning</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">cp "$PWD/resources/server/certs/"* ~/.ssh
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The above just puts the files where our provisioners (Packer and Vagrant) can pick them up and deploy them with the image.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Locally we are using Vagrant to launch a cluster to do some tests on our laptop.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We also use Packer and <em style="box-sizing: border-box;">aws</em> command line tools to create EC2 AMIs (and Docker images), but we don’t cover aws in this article (it is in the very next which is sort of part 2 to this article).</div>
<h2 id="create-a-bastion-server-to-do-ansible-devops-dba-tasks" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 30px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Create a bastion server to do ansible DevOps/DBA tasks for Cassandra Cluster</h2>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Eventually, we would like to use a <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">bastion server</em></span> that is a <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">public subnet</em></span> to send commands to our Cassandra Database nodes that are in a <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">private subnet</em></span> in EC2. For local testing, we set up a bastion server, which is well explained in this <a href="https://sysadmincasts.com/episodes/45-learning-ansible-with-vagrant-part-2-4" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">guide to Vagrant and Ansible</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We used <a href="https://sysadmincasts.com/episodes/45-learning-ansible-with-vagrant-part-2-4" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">Learning Ansible with Vagrant (Part <span style="box-sizing: border-box; font-size: 10.5px; line-height: 0; position: relative; top: -0.5em; vertical-align: baseline;">2</span>⁄<span style="bottom: -0.25em; box-sizing: border-box; font-size: 10.5px; line-height: 0; position: relative; vertical-align: baseline;">4</span>)</a> as a guide for some of the setup performed in this article. It is a reliable source of Ansible and Vagrant knowledge for DevOps/DBA. Their <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">mgmt</em></span> node corresponds to what we call a <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">bastion</em></span> server. A notable difference is we are using CentOS 7 not Ubuntu, and we made some slight syntax updates to some of the Ansible commands that we are using (we use a later version of Ansible).</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We added a <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">bastion server</em></span> to our Vagrant config as follows:</div>
<h4 id="vagrantfile-to-set-up-the-bastion-for-our-cassandra-cluster" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Vagrantfile to set up the bastion for our Cassandra Cluster</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-ruby" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">
# Define Bastion Node
config.vm.define "bastion" do |node|
node.vm.network "private_network", ip: "192.168.50.20"
node.vm.provider "virtualbox" do |vb|
vb.memory = "256"
vb.cpus = 1
end
node.vm.provision "shell", inline: <<-SHELL
yum install -y epel-release
yum update -y
yum install -y ansible
mkdir /home/vagrant/resources
cp -r /vagrant/resources/* /home/vagrant/resources/
mkdir -p ~/resources
cp -r /vagrant/resources/* ~/resources/
mkdir -p /home/vagrant/.ssh/
cp /vagrant/resources/server/certs/* /home/vagrant/.ssh/
sudo /vagrant/scripts/002-hosts.sh
ssh-keyscan node0 node1 node2 >> /home/vagrant/.ssh/known_hosts
mkdir ~/playbooks
cp -r /vagrant/playbooks/* ~/playbooks/
sudo cp /vagrant/resources/home/inventory.ini /etc/ansible/hosts
chown -R vagrant:vagrant /home/vagrant
SHELL
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The bastion server which could be on a public subnet in AWS in a VPC uses the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-keyscan</code> to add nodes that we setup in the host file into <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">/home/vagrant/.ssh/known_hosts</code>.</div>
<h4 id="running-ssh-keyscan" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Running ssh-keyscan</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">ssh-keyscan node0 node1 node2 >> /home/vagrant/.ssh/known_hosts
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
This utility is to get around the problem of needing to verify nodes, and getting this error message: <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">The authenticity of host ... can't be established. ... Are you sure you want to continue connecting (yes/no)? no</code> when we are trying to run ansible command line tools.</div>
<h2 id="modify-the-vagrant-provision-script" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 30px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Modify the Vagrant provision script</h2>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Since we are using provision files to create different types of images (Docker, EC2 AMI, Vagrant/VirtualBox), then we use a provisioning script specific to <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">vagrant</em></span>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
In this vagrant provision script, we call another provision script to setup a hosts file.</div>
<h4 id="000-vagrant-provision-sh" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
000-vagrant-provision.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">mkdir -p /home/vagrant/.ssh/
cp /vagrant/resources/server/certs/* /home/vagrant/.ssh/
...
scripts/002-hosts.sh
echo RUNNING TUNE OS
</code></pre>
<h3 id="setting-up-sshd-on-our-cassandra-database-nodes-in-our-devops-cassandra-cluster" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Setting up sshd on our Cassandra Database nodes in our DevOps Cassandra Cluster</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The provision script <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">002-hosts.sh</code> configures <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">/etc/ssh/sshd_config/sshd_config</code> to allow public key auth. Then it restarts the daemon for ssh communication <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">sshd</code>. (The other provisioning scripts it invokes was covered in the first two articles).</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s look at the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">002-hosts.sh</code> provision script. You can see some remnants from the last article where we setup <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">csqlsh</code>, and then it gets to business setting up <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">sshd</code> (secure server shell daemon).</div>
<h4 id="scripts-002-hosts-sh-sets-up-sshd-and-hosts-file" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
scripts/002-hosts.sh - sets up sshd and hosts file</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/bin/bash
set -e
## Copy csqlshrc file that controls csqlsh connections to ~/.cassandra/cqlshrc.
mkdir ~/.cassandra
cp ~/resources/home/.cassandra/cqlshrc ~/.cassandra/cqlshrc
## Allow pub key login to ssh.
sed -ie 's/#PubkeyAuthentication no/PubkeyAuthentication yes/g' /etc/ssh/sshd_config
## System control restart sshd daemon to take sshd_config into effect.
systemctl restart sshd
# Create host file so it is easier to ssh from box to box
cat >> /etc/hosts <<EOL
192.168.50.20 bastion
192.168.50.4 node0
192.168.50.5 node1
192.168.50.6 node2
192.168.50.7 node3
192.168.50.8 node4
192.168.50.9 node5
EOL
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
This setup is fairly specific to our Vagrant setup at this point. To simplify access to the servers that hold the different Cassandra Database nodes, the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">002-hosts.sh</code> creates an <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">\etc\hosts\</code> file on the bastion server.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
With our certification keys added to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">sshd config</code> and our hosts configured (and our <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">inventory.ini</code> file shipped), we can start using <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">ansible</em></span> from our bastion server.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
This reminds me, we did not talk about the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible inventory.ini</code> file.</div>
<h3 id="ansible-config-on-bastion-for-cassandra-database-cluster" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Ansible config on bastion for Cassandra Database Cluster</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Ansible has a <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible.cfg</code> file, and an <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">inventory.ini</code> file. When you run <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible</code>, it checks for <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible.cfg</code> in the current working directory, then your home directory, and then for a master config file (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">/etc/ansible</code>). We created an <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">inventory.ini</code> file which lives under <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~\github\cassandra-image\resources\home</code>, which gets mapped to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">\vagrant\resources\home</code> on the virtual machines (node0, bastion, node1, and node2). A provision script moves the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">inventory.ini</code> file to its proper location (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">sudo cp /vagrant/resources/home/inventory.ini /etc/ansible/hosts</code>).</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">inventory.ini</code> contains servers that you want to manage with Ansible. A couple of things are going on here, we have a <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bastion</code>group, this is for our <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bastion</code> server, next we have the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">nodes</code> group, and it is made up of <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">node0</code>, <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">node1</code>, and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">node2</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s see what the inventory.ini file actually looks like.</div>
<h4 id="inventory-ini-that-gets-copied-to-ansible-master-list-on-bastion" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
inventory.ini that gets copied to Ansible master list on Bastion</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">[bastion]
bastion
[nodes]
node0
node1
node2
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Once we provision our cluster, we can log into bastion and start executing <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">ansible</em></span> commands.</div>
<h3 id="installing-cert-key-for-test-devops-dba-cassandra-cluster-on-all-nodes-using-an-ansible-playbook" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Installing cert key for test DevOps/DBA Cassandra Cluster on all nodes using an ansible playbook</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
To make this happen, we had to tell the other servers about our certification keys.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We did this with an ansible playbook as follows:</div>
<h4 id="ansible-playbook-getting-invoked-from-vagrant-on-each-new-cassandra-database-node" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Ansible playbook getting invoked from Vagrant on each new Cassandra Database node</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">
Vagrant.configure("2") do |config|
config.vm.box = "centos/7"
# Define Cassandra Nodes
(0..numCassandraNodes-1).each do |i|
port_number = i + 4
ip_address = "192.168.50.#{port_number}"
seed_addresses = "192.168.50.4,192.168.50.5,192.168.50.6"
config.vm.define "node#{i}" do |node|
node.vm.network "private_network", ip: ip_address
node.vm.provider "virtualbox" do |vb|
vb.memory = "2048"
vb.cpus = 4
end
...
node.vm.provision "ansible" do |ansible|
ansible.playbook = "playbooks/ssh-addkey.yml"
end
end
end
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice the line <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">node.vm.provision "ansible" do |ansible|</code> and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible.playbook = "playbooks/ssh-addkey.yml"</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
If you are new to Vagrant and the above just is not making sense, please watch <a href="https://sysadmincasts.com/episodes/42-crash-course-on-vagrant-revised" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">Vagrant Crash Course</a>. It is by the same folks (guy) who created the Ansible series.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Ansible playbooks are like configuration playbooks. You can perform tons of operations that are important for DevOps (like yum installing software, specific tasks to Cassandra, etc.).</div>
<blockquote style="background-color: white; border-left: 5px solid rgb(0, 191, 255); box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin: 0px 0px 20px; padding: 10px 20px;">
<div style="box-sizing: border-box;">
Playbooks are Ansible’s configuration, deployment, and orchestration language. They can describe a policy you want your remote systems to enforce, or a set of steps in a general IT process. –<a href="http://docs.ansible.com/ansible/playbooks.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">Ansible Playbook documentation</a>.</div>
</blockquote>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Here is the ansible playbook to add the RSA public key to the cassandra nodes as follows.</div>
<h4 id="ansible-playbook-ssh-addkey-yml-to-add-test-rsa-pub-to-all-cassandra-database-node-servers" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Ansible playbook ssh-addkey.yml to add test_rsa.pub to all Cassandra Database node servers</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">---
- hosts: all
become: true
gather_facts: no
remote_user: vagrant
tasks:
- name: install ssh key
authorized_key: user=vagrant
key="{{ lookup('file', '../resources/server/certs/test_rsa.pub') }}"
state=present
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The trick here is that <a href="https://www.vagrantup.com/docs/provisioning/ansible.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">Vagrant supports running Ansible playbooks as well</a>.</div>
<blockquote style="background-color: white; border-left: 5px solid rgb(0, 191, 255); box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin: 0px 0px 20px; padding: 10px 20px;">
<div style="box-sizing: border-box;">
The Vagrant Ansible provisioner allows you to provision the guest using Ansible playbooks by executing ansible-playbook from the Vagrant host. –(Vagrant Ansible documentation)[<a href="https://www.vagrantup.com/docs/provisioning/ansible.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">https://www.vagrantup.com/docs/provisioning/ansible.html</a>]</div>
</blockquote>
<h3 id="for-users-who-did-not-read-any-of-the-first-articles-on-setting-up-the-cassandra-cluster" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
For users who did not read any of the first articles on setting up the Cassandra Cluster</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
If you have not done so already navigate to the project root dir (which is ~/github/cassandra-image on my dev box), download the binaries. The source code is at <a href="https://github.com/cloudurable/cassandra-image" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">Github Cassandra Image project</a>.</div>
<h4 id="running-setup-scripts" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Running setup scripts</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">## cd ~; mkdir github; cd github; git clone https://github.com/cloudurable/cassandra-image
$ cd ~/github/cassandra-image
$ pwd
~/github/cassandra-image
## Setup keys
$ bin/setupkeys-cassandra-security.sh
## Download binaries
$ bin/prepare_binaries.sh
## Bring Vagrant cluster up
$ vagrant up
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Even if you read the first article note that <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bin/prepare_binaries.sh</code> is something we added after the first two articles. It downloads the binaries needed for the provisioning, does a checksum of the files and then installs them as part of the provisioning process.</div>
<h3 id="where-do-you-go-if-you-have-a-problem-or-get-stuck" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Where do you go if you have a problem or get stuck?</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We set up a <a href="https://groups.google.com/forum/#!forum/cassandra-image-project" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">google group</a> for this project and set of articles. If you just can’t get something to work or you are getting an error message, please report it <a href="https://github.com/cloudurable/cassandra-image/issues" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">here</a>. Between the mailing list and the github issues, we can support you with quite a few questions and issues.</div>
<h3 id="running-ansible-commands-from-bastion" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Running ansible commands from bastion</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s log into bastion and run ansible commands against the cassandra nodes.</div>
<h4 id="working-with-ansible-from-bastion-and-using-ssh-agent" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Working with ansible from bastion and using ssh-agent</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ vagrant ssh bastion
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
So we don’t have to keep logging in, and passing our cert key, let’s start up an <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code> and add our cert key <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-add ~/.ssh/test_rsa</code> to the agent.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code> is a utility to hold private keys used for public key authentication (RSA, DSA, ECDSA, Ed25519) so you don’t have to keep passing the keys around. The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code> is usually started in the beginning of a login session. Other programs (scp, ssh, ansible) are started as clients to the ssh-agent utility.</div>
<blockquote style="background-color: white; border-left: 5px solid rgb(0, 191, 255); box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin: 0px 0px 20px; padding: 10px 20px;">
<div style="box-sizing: border-box;">
Mastering ssh is essential for DevOps and needed for ansible.</div>
</blockquote>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
First set up <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code> and add keys to it with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-add</code>.</div>
<h4 id="start-ssh-agent-and-add-keys" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Start ssh-agent and add keys</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ssh-agent bash
$ ssh-add ~/.ssh/test_rsa
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now that the agent is running and our keys are added, we can use <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">ansible</em></span> without passing it the RSA private key.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s verify connectivity, by pinging some of these machines. Let’s ping the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">node0</code> machine. Then let’s ping all of the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">nodes</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s use the ansible ping module to ping the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">node0</code> server.</div>
<h4 id="ansible-ping-the-cassandra-database-node" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Ansible Ping the Cassandra Database node</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ansible node0 -m ping
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Output</div>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">node0 | SUCCESS => {
"changed": false,
"ping": "pong"
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
To learn more about DevOps with ansible see this video on <a href="https://sysadmincasts.com/episodes/43-19-minutes-with-ansible-part-1-4" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">Ansible introduction</a>. It covers a lot of the basics of ansible.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now let’s ping all of the nodes.</div>
<h4 id="ansible-ping-all-cassandra-database-cluster-nodes" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Ansible Ping all Cassandra Database Cluster nodes</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ansible nodes -m ping
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Output</div>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">node0 | SUCCESS => {
"changed": false,
"ping": "pong"
}
node2 | SUCCESS => {
"changed": false,
"ping": "pong"
}
node1 | SUCCESS => {
"changed": false,
"ping": "pong"
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Looks like bastion can run ansible against all of the servers.</div>
<h2 id="setting-up-my-macosx-to-run-ansible-against-cassandra-database-cluster-nodes" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 30px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Setting up my MacOSX to run Ansible against Cassandra Database Cluster nodes</h2>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The script <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/github/cassandra-image/bin/setupkeys-cassandra-security.sh</code> copies the test cluster key for ssh (secure shell) over to ~/.ssh/ (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">cp "$PWD/resources/server/certs/"* ~/.ssh</code>). It was Run from the project root folder which is <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/github/cassandra-image</code> on my box.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Move to the where you checked out the <a href="https://github.com/cloudurable/cassandra-image" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">project</a>.</div>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">cd ~/github/cassandra-image
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
In this folder is an <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible.cfg</code> file and an inventory.ini file for local dev. Before you use these first modify your <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">/etc/hosts</code> file to configure entries for <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bastion</code>, <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">node0</code>, <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">node1</code>, <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">node2</code> servers.</div>
<h4 id="add-bastion-node0-etc-to-etc-hosts" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Add bastion, node0, etc. to /etc/hosts</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ cat /etc/hosts
### Used for ansible/ vagrant
192.168.50.20 bastion
192.168.50.4 node0
192.168.50.5 node1
192.168.50.6 node2
192.168.50.7 node3
192.168.50.8 node4
192.168.50.9 node5
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We can use ssh-keyscan just like we did before to add these hosts to our <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">known_hosts</code> file.</div>
<h4 id="add-keys-to-known-hosts-to-avoid-prompts" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Add keys to known_hosts to avoid prompts</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ssh-keyscan node0 node1 node2 >> ~/.ssh/known_hosts
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Then just like before we can start up an <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code> and add our keys.</div>
<h4 id="start-ssh-agent-and-add-keys-1" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Start ssh-agent and add keys</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ssh-agent bash
$ ssh-add ~/.ssh/test_rsa
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible.cfg</code> and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">inventory.ini</code> files are a bit different than on our bastion server because we have to add the user name.</div>
<h4 id="notice-the-ansible-cfg-file-and-inventory-ini-file-in-the-project-dir" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Notice the ansible.cfg file and inventory.ini file in the project dir</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ cd ~/github/cassandra-image
$ cat ansible.cfg
[defaults]
hostfile = inventory.ini
cat inventory.ini
[nodes]
node0 ansible_user=vagrant
node1 ansible_user=vagrant
node2 ansible_user=vagrant
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Ansible will use these.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
From the project directory, you should be able to ping <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">node0</code> and all of the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">nodes</code> just like before.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Ping <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">node0</code> with ansible.</div>
<h4 id="ansible-ping-cassandra-database-node" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Ansible Ping Cassandra Database node</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ansible node0 -m ping
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Output</div>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">node0 | SUCCESS => {
"changed": false,
"ping": "pong"
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Ping all of the Cassandra nodes with ansible.</div>
<h4 id="ansible-ping-all-cassandra-database-cluster-nodes-1" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Ansible Ping All Cassandra Database Cluster nodes</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ansible nodes -m ping
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Output</div>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">node0 | SUCCESS => {
"changed": false,
"ping": "pong"
}
node2 | SUCCESS => {
"changed": false,
"ping": "pong"
}
node1 | SUCCESS => {
"changed": false,
"ping": "pong"
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
In the next article, we cover how to <a href="http://cloudurable.com/blog/aws-ansible-packer-ssh-for-devops/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">setup <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~.ssh/config</code></a> so you don’t have to remember to use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code>.</div>
<h3 id="using-ansible-to-run-nodetool-on-cassandra-cluster-nodes" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Using ansible to run nodetool on Cassandra Cluster nodes</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
You may recall from the first article that we would log into the servers (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">vagrant ssh node0</code>) and then check that they could see the other nodes with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">nodetool describecluster</code>. We could run this command with all three servers (from <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bastion</code> or on our dev laptop) with ansible.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s use ansible to run <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">describecluster</code> against all of the nodes.</div>
<h4 id="ansible-running-nodetool-describecluster-against-all-cassandra-cluster-nodes" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Ansible running nodetool describecluster against all Cassandra Cluster nodes</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ansible nodes -a "/opt/cassandra/bin/nodetool describecluster"
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
This command allows us to check the status of every node quickly.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s say that we wanted to update a schema or do a rolling restart of our Cassandra cluster nodes, which could be a very common task. Perhaps before the update, we want to decommission the node and back things up. To do this sort of automation, we could create an Ansible playbook.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<a href="http://docs.ansible.com/ansible/playbooks_intro.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">Ansible Playbooks</a> are more powerful than executing ad-hoc task execution and is especially powerful for managing a cluster of Cassandra servers.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Playbooks allow for configuration management and multi-machine deployment to manage complex tasks like a rolling upgrade or schema updates or perhaps a weekly backup.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Playbooks are declarative configurations. Ansible Playbooks also orchestrate steps into a simpler task. This automation gets rid of a lot of manually ordered process and allows for an immutable infrastructure.</div>
<h3 id="our-describe-cluster-playbook-for-cassandra-database-cluster-nodes" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Our describe-cluster playbook for Cassandra Database Cluster nodes</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Creating a complicated playbook is beyond the scope of this article. But let’s create a simple playbook and execute it. This playbook will run the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">nodetool describecluster</code> on each node.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Here is our playbook that runs Cassandra <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">nodetool describecluster</code> on each Cassandra node in our cluster.</div>
<h4 id="playbooks-descibe-cluster-yml-simple-ansible-playbook-that-runs-cassandra-nodetool-describecluster" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
playbooks/descibe-cluster.yml - simple ansible playbook that runs Cassandra nodetool describecluster</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">---
- hosts: nodes
gather_facts: no
remote_user: vagrant
tasks:
- name: Run NodeTool Describe Cluster command against each Cassandra Cluster node
command: /opt/cassandra/bin/nodetool describecluster
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
To run this, we use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible-playbook</code> as follow.</div>
<h4 id="running-describe-cluster-playbook" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Running describe-cluster playbook</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ansible-playbook playbooks/describe-cluster.yml --verbose
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Between this article and the last, we modified our <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">Vagrantfile</code> quite a bit. It now uses a for loop to create the Cassandra nodes, and it uses <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">ansible</em></span> provisioning.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Here is our new Vagrantfile with updates:</div>
<h4 id="complete-code-listing-of-vagrantfile-that-sets-up-our-devops-dba-cassandra-database-cluster" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Complete code listing of Vagrantfile that sets up our DevOps/DBA Cassandra Database Cluster</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;"># -*- mode: ruby -*-
# vi: set ft=ruby :
numCassandraNodes = 3
Vagrant.configure("2") do |config|
config.vm.box = "centos/7"
# Define Cassandra Nodes
(0..numCassandraNodes-1).each do |i|
port_number = i + 4
ip_address = "192.168.50.#{port_number}"
seed_addresses = "192.168.50.4,192.168.50.5,192.168.50.6"
config.vm.define "node#{i}" do |node|
node.vm.network "private_network", ip: ip_address
node.vm.provider "virtualbox" do |vb|
vb.memory = "2048"
vb.cpus = 4
end
node.vm.provision "shell", inline: <<-SHELL
sudo /vagrant/scripts/000-vagrant-provision.sh
sudo /opt/cloudurable/bin/cassandra-cloud -cluster-name test \
-client-address #{ip_address} \
-cluster-address #{ip_address} \
-cluster-seeds #{seed_addresses}
SHELL
node.vm.provision "ansible" do |ansible|
ansible.playbook = "playbooks/ssh-addkey.yml"
end
end
end
# Define Bastion Node
config.vm.define "bastion" do |node|
node.vm.network "private_network", ip: "192.168.50.20"
node.vm.provider "virtualbox" do |vb|
vb.memory = "256"
vb.cpus = 1
end
node.vm.provision "shell", inline: <<-SHELL
yum install -y epel-release
yum update -y
yum install -y ansible
mkdir /home/vagrant/resources
cp -r /vagrant/resources/* /home/vagrant/resources/
mkdir -p ~/resources
cp -r /vagrant/resources/* ~/resources/
mkdir -p /home/vagrant/.ssh/
cp /vagrant/resources/server/certs/* /home/vagrant/.ssh/
sudo /vagrant/scripts/002-hosts.sh
ssh-keyscan node0 node1 node2 >> /home/vagrant/.ssh/known_hosts
mkdir ~/playbooks
cp -r /vagrant/playbooks/* ~/playbooks/
sudo cp /vagrant/resources/home/inventory.ini /etc/ansible/hosts
chown -R vagrant:vagrant /home/vagrant
SHELL
end
#
# View the documentation for the provider you are using for more
# information on available options.
# Define a Vagrant Push strategy for pushing to Atlas. Other push strategies
# such as FTP and Heroku are also available. See the documentation at
# https://docs.vagrantup.com/v2/push/atlas.html for more information.
config.push.define "atlas" do |push|
push.app = "cloudurable/cassandra"
end
end
</code></pre>
<h3 id="conclusion" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Conclusion</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We set up Ansible for our <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">Cassandra Database</em></span> Cluster to do automate common DevOps/DBA tasks. We created an ssh key and then set up our instances with this key so we could use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh</code>, <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">scp</code>, and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible</code>. We set up a bastion server with Vagrant. We used <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible</code>playbook (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-addkey.yml</code>) from Vagrant to install our <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">test</code> cluster key on each server. We ran <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible ping</code> against a single server. We ran <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible ping</code> against many servers (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">nodes</code>). We set up our local dev machine with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible.cfg</code> and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">inventory.ini</code> so we could use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible</code> commands direct to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">node0</code> and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">nodes</code>. We ran <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">nodetool describecluster</code> against all of the nodes from our dev machine. Lastly, we created a very simple playbook that can run <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">nodetool describecluster</code>. Ansible is a very powerful tool that can help you manage a cluster of Cassandra instances. In later articles, we will use Ansible to create more complex playbooks like backing up Cassandra nodes to S3.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
<h3 id="retrospective-past-articles-in-this-cassandra-cluster-devops-dba-series" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin: 20px 0px; position: relative;">
Cassandra Tutorial: Cassandra Cluster DevOps/DBA series</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The first tutorial in this <i>Cassandra tutorial</i> series focused on setting up a Cassandra Cluster. The first Cassandra tutorial setting up a Cassandra <a href="http://cloudurable.com/blog/cassandra-image-vagrant-cluster-example/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">cluster with Vagrant</a> (also appeared on DZone with some additional content <a href="https://dzone.com/articles/setting-up-a-cassandra-cluster-with-vagrant" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">DZone Setting up a Cassandra Cluster with Vagrant</a>. The second article in this series was about <a href="http://cloudurable.com/blog/cassandra-image-vagrant-cluster-example/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">setting up SSL for a Cassandra cluster using Vagrant</a> (which also appeared with more content as <a href="https://dzone.com/articles/setting-up-a-cassandra-cluster-with-ssl" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">DZone Setting up a Cassandra Cluster with SSL</a>). The <a href="http://cloudurable.com/blog/ansible-cassandra-vagrant-devops/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">third article</a> in this series was about configuring and using Ansible (building on the first two articles). This article (the 4th) will cover applying the tools and techniques from the first three articles to produce an image (EC2 AMI to be precise) that we can deploy to AWS/EC2. To do this explanation, we will use Packer, Ansible, and the Aws Command Line tools. The AWS command line tools are essential for doing DevOps with AWS.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Check out more information about the Cassandra Database</div>
<ul style="background-color: white; color: #666666; font-family: "Trebuchet MS", Trebuchet, Verdana, sans-serif; font-size: 13.2px; line-height: 1.4; margin: 0.5em 0px; padding: 0px 2.5em;">
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/service-architecture-analysis-cassandra-or-kafka-aws-ec2/index.html" style="color: #888888; text-decoration: none;">Cassandra Consulting: Architecture Analysis</a></li>
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/service-quick-start-mentoring-cassandra-or-kafka-aws-ec2/index.html" style="color: #888888; text-decoration: none;">Cassandra Consulting: Quick Start</a></li>
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/cassandra-course/index.html" style="color: #888888; text-decoration: none;">Cassandra Course</a></li>
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/" style="color: #888888; text-decoration: none;">Amazon Cassandra Support</a></li>
</ul>
<div>
<span style="color: #666666; font-family: Trebuchet MS, Trebuchet, Verdana, sans-serif;"><span style="font-size: 13.2px;"><br /></span></span></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com1tag:blogger.com,1999:blog-6128525281101058933.post-28865544259545107402017-03-04T14:44:00.002-08:002017-03-04T14:44:48.353-08:00Setting up Ansible, SSH to configure AWS EC2 instances<h3 id="ansible-and-ec2" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Setting up Ansible, SSH to configure AWS EC2 instances</h3>
<div>
We pick up with our <a href="http://cloudurable.com/blog/ansible-cassandra-vagrant-devops/index.html">ansible tutorial </a> to focus on this <a href="http://cloudurable.com/blog/aws-ansible-packer-ssh-for-devops/index.html">AWS ansible tutorial</a> on how to use ansible with EC2 as well as mastering ansible inventory setup, ssh-agent redux and covering ssh client config so you don't have to have long convoluted scripts and have to remember the id file, username, etc. for ssh, scp and ansible. </div>
<div>
<br /></div>
<div>
<h3 id="overview" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Overview</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
This <b><i>amazon ansible tutorial</i></b> covers the following:</div>
<ul style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px; margin-top: 0px;">
<li style="box-sizing: border-box;">Setting up ssh client config</li>
<li style="box-sizing: border-box;">Setting up ansible to manage our EC2 instance (ansible uses ssh)</li>
<li style="box-sizing: border-box;">Setting up a <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code> and adding ssh identities (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-add</code>)</li>
<li style="box-sizing: border-box;">Setting ssh using <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/.ssh/config</code> so we don’t have to pass credentials around</li>
<li style="box-sizing: border-box;">Using ansible dynamic inventory with EC2</li>
<li style="box-sizing: border-box;">AWS command line tools to manage DNS entries with <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">Route 53</em></span></li>
</ul>
<span style="background-color: white; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px;"><div>
<span style="background-color: white; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px;"><br /></span></div>
This <b><i>AWS ansible tutorial</i></b> covers how to use </span><span style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; font-weight: 700;"><em style="box-sizing: border-box;">ansible dynamic inventory</em></span><span style="background-color: white; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px;"> management with EC2. A related topic, which is crucial for DevOps is how to setup ssh config to simplify logging into remote machines: like creating simple aliases, auto-passing id files, auto-passing user id information.</span><br /><div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Lastly, we use <em style="box-sizing: border-box;">AWS Route 53</em> to setup a DNS name for our new instance which we then use from ansible, and never have to reconfigure our ansible config when we create a new AMI or EC2 instance. If you are using EC2 and you are not using <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh config files </code> and ansible and you are doing <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">DevOps</em></span>, this article is a must.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
This is part 4 of this series of articles on creating a the Cassandra Database image and DevOps/DBA. You don’t truly need those articles to follow this one, but they might provide a lot of contexts. If you are new to ansible the <a href="http://cloudurable.com/blog/ansible-cassandra-vagrant-devops/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">last article on ansbible</a> would be good to at least skim. This one picks up and covers ansible more deeply with regards to AWS/EC2.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
You can find the source for the first, second, third and this Cassandra tutorial (Ansible/Cassandra tutorial) at our <a href="https://github.com/cloudurable/cassandra-image" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">Cloudurable Cassandra Image for Packer, EC2, Docker, AWS and Vagrant</a>. In later articles, we will set up a working cluster in AWS using VPCs, Subnets, Availability Zones and Placement groups. We will continue to use ansible to manage our Cassandra cluster.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The source code for this article is in this <a href="https://github.com/cloudurable/cassandra-image/tree/article4-packer-ec2-ansible" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">branch on github</a>.</div>
</div>
</div>
<h3 id="ansible-and-ec2" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Ansible and EC2</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Although we have base images, since Cassandra is stateful, we will want the ability to update the images in place for our <b><i>Amazon Cassandra</i></b> support.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The options for configuration and orchestration management are endless (Puppet, Chef, Boto, etc.). This article uses Ansible for many of these tasks. Ansible is an agentless architecture and works over <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">ssh</em></span> (secure shell) as we covered in our last article (<a href="http://cloudurable.com/blog/ansible-cassandra-vagrant-devops/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">Setting up Ansible for our Cassandra Cluster to do DevOps/DBA tasks</a>). There are some very helpful <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">Ansible/AWS</em></span> integrations which will try to cover in future articles.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The Ansible framework allows DevOps/DBA staff to run commands against Amazon EC2 instances as soon as they are available. Ansible is very suitable for provisioning hosts in a Cassandra cluster as well as performing routine DevOps/DBA tasks like replacing a failed node, backing up a node, profiling Cassandra, performing a rolling upgrade and more.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Since Ansible relies on <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh</code>, we should make sure that <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh</code> is working for us.</div>
<h3 id="debugging-possible-ssh-problems-before-we-get-started-with-ansible" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Making sure ssh works before we get started with ansible</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
If ssh is not working, you can't use ansible because ansible needs ssh. </div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Before you go about using ansible with AWS/EC2 to manage your Cassandra clusters, you have to make sure that you can, in fact, connect with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The first step in our journey is to get the IP of the EC2 instance that you just launched.</div>
<blockquote style="background-color: white; border-left: 5px solid rgb(0, 191, 255); box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin: 0px 0px 20px; padding: 10px 20px;">
<div style="box-sizing: border-box;">
Another key tip for using ansible is to use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">-vvvv</code> if it can’t connect so you can see why it can’t connect</div>
</blockquote>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s get the IP of the new instance using <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">get-IP-cassandra.sh</code>, which we <a href="http://rick-hightower.blogspot.com/2017/03/using-aws-cli-to-create-our-cassandra.html">covered earlier</a>.</div>
<h4 id="getting-the-ip" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Getting the IP</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ bin/get-IP-cassandra.sh
54.218.113.95
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now we can log in with the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">pem</code> file associated with our AWS key-pair that we used to launch our Cassandra EC2 instance.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s see if we can log into the Cassandra EC2 instance with ssh.</div>
<h4 id="can-i-log-in-with-the-pem-file" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Can I log in with the pem file?</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">ssh -i ~/.ssh/cloudurable-us-west-2.pem centos@54.218.113.95
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
If you can do this, then your security group is setup properly. If you can’t do this, make sure you VPC security group associated with the instance has port 22 open. (Limit logging into instances via SSH port 22 to only your IP address.)</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
In addition to the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">pem</code> file that AWS creates, we have our private rsa key for the test cluster (<a href="http://cloudurable.com/blog/ansible-cassandra-vagrant-devops/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">which we covered in the last article</a>). Recall that the rsa key is used with the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible</code> user (also described in the last article on ansible).</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s see if we can log in with our RSA private key.</div>
<h4 id="can-i-log-in-with-the-key-we-generated-for-ansible" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Can I log in with the key we generated for ansible?</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">ssh -i ~/.ssh/test_rsa ansible@54.218.113.95
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
If you can log in with the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">pem</code> but not the rsa key we created for the test cluster, then you have an issue with a key mismatch (perhaps). You could try to regenerate the keys with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bin/setupkeys-cassandra-security.sh</code> then either copy them with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">scp</code> copy or upload them with the <a href="http://docs.ansible.com/ansible/list_of_files_modules.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">ansible file/copy module or file/synchronize module</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Passing the key on each ansible command is tiresome, let’s use the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code> (discussed in the last <a href="http://cloudurable.com/blog/ansible-cassandra-vagrant-devops/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">article</a>), to add (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-add</code>) our cluster key identity (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/.ssh/test_rsa</code>) to all <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh</code> commands that we use (including ansible).</div>
<h4 id="can-i-install-the-key-and-log-in-using-ssh-agent" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Can I install the key and log in using ssh-agent?</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ssh-agent bash
$ ssh-add ~/.ssh/test_rsa
$ ssh ansible@54.218.113.95
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
If you were able to log in with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh</code> by adding the key to the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code>, then you are ready to use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible</code>. To test that you can connect via ansible add these entries to the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">inventory.ini</code> file in the project root (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/github/cassandra-cloud</code>).</div>
<h3 id="setting-up-ansible-using-the-inventory-ini" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Setting up Ansible using the inventory.ini</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We assume you have set up the cluster key as follow:</div>
<h4 id="setup-cluster-key-for-ansible" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Setup cluster key for ansible</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ssh-agent bash
$ ssh-add ~/.ssh/test_rsa
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Recall that <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bin/setupkeys-cassandra-security.sh</code> creates the RSA key and installs it under ~/.ssh/test_rsa. Then the provisioning scripts install the key correctly on the EC2 image (AMI).</div>
<h4 id="add-this-to-inventory-ini-for-ansible" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Add this to inventory.ini for ansible</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">[aws-nodes]
54.218.113.95 ansible_user=ansible
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The above tells ansible that this server <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">54.218.113.95</code> exists, and it is in the group <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">aws-nodes</code>, and that when we connect to it that we should use the user <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible</code>. (Remember we looked up the IP of the Cassandra Datbase EC2 instance using <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bin/get-IP-cassandra.sh</code>).</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Once that is setup, we can run the <a href="http://docs.ansible.com/ansible/ping_module.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">ansible ping module</a> against our Cassandra EC2 instance as follows:</div>
<h4 id="run-ansible-ping-modules-against-aws-nodes" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Run ansible ping modules against aws-nodes</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ansible aws-nodes -m ping
54.218.113.95 | SUCCESS => {
"changed": false,
"ping": "pong"
}
</code></pre>
<h3 id="dynamic-ansible-inventory" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Dynamic Ansible inventory</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
When doing Amazon Web Services EC2 DevOps, you could be managing several groups of servers. EC2 allows you to use placement groups, autoscale groups, security groups, and tags to organize and manage your instances. AWS EC2 is rich with meta-data about the instances.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
If you are running Dev, QA, production or even green/blue deploys with CI and CD, you will be running many EC2 instances over time. Hosts can come and go in EC2. Because of the ephemeral nature of hosts in EC2, ansbile allows you to use external scripts to manage ansible inventory lists. There is such an <a href="http://docs.ansible.com/ansible/intro_dynamic_inventory.html#example-aws-ec2-external-inventory-script" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">ansible inventory script for AWS EC2</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
As you can imagine if you are doing DevOps, <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">ansible AWS EC2 dynamic inventory</em></span> is a must.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
You can set up AWS via your <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/.aws/config</code> and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/.aws/credentials</code> files, and if you installed the aws command line, then you likely have this setup or the requisite environment variables.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
To use Ansible’s <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">-i</code> command line option and specify the path to the script.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Before we do that, we need to download the <a href="https://aws.amazon.com/blogs/apn/getting-started-with-ansible-and-dynamic-amazon-ec2-inventory-management/" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">ec2 ansible inventory script</a> and mark it executable.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Download the dynamic ansible inventory script as follows.</div>
<h4 id="download-the-ansible-ec2-inventory-script-make-it-executable" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Download the ansible ec2 inventory script, make it executable</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ pwd
~/github/cassandra-image/
$ wget https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.py -O ansible-ec2/ec2.py
$ chmod +x ansible-ec2/ec2.py
$ wget https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.ini -O ansible-ec2/ec2.ini
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
After you download it, you can start using it.</div>
<h3 id="using-a-dynamic-inventory" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Using a dynamic inventory</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now let’s use the dynamic inventory ansible script.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Before we do that, let’s add the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">pem</code> associated with our AWS Key Pair to the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code> as follows (if you have not done so already).</div>
<h4 id="add-centos-pem-file-key-pair" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Add centos pem file (key pair)</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ssh-add ~/.ssh/cloudurable-us-west-2.pem
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Then we can ping the EC2 instance with the ansible ping module as follows:</div>
<h4 id="pass-dynamic-list-to-ansible-use-user-centos" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Pass dynamic list to ansible use user centos</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ansible -i ansible-ec2/ec2.py tag_Name_cassandra_node -u centos -m ping
54.218.113.95 | SUCCESS => {
"changed": false,
"ping": "pong"
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">-i</code> option passes a script that will generate a JSON inventory list. You can even write a custom script that produces an inventory as long as that script uses the same JSON format. Above we are passing the script that we just downloaded. Remember that an <a href="http://docs.ansible.com/ansible/intro_inventory.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">ansible inventory list</a> is just a list of servers that we are managing with ansible.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now we know we can use ansible with our AWS key pair and our AWS PEM file. But can we use it with our RSA key?</div>
<h3 id="using-ansible-with-rsa-key-and-ansible-user-from-last-article" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Using ansible with RSA Key and ansible user from last article</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Please recall from the last article that we set up a user called <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible</code> which used an RSA private key file that we created in the last article as well (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/.ssh/test_rsa</code>).</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We should be able to manage our EC2 instance with the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible</code> user using the RSA key. Let’s try.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Add the ansible users RSA key to the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code> as follows.</div>
<h4 id="add-ansible-users-rsa-private-key-file-test-rsa-file" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Add ansible users RSA private key file - test_rsa file</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ssh-add `~/.ssh/test_rsa`
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now we can access ansible via the ansible user using <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/.ssh/test_rsa</code> as our private key. Use ansible with the ansible users (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">-u ansible</code>) and the RSA key we just installed.</div>
<h4 id="pass-dynamic-list-to-ansible-use-user-ansible" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Pass dynamic list to ansible use user ansible</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ansible -i ansible-ec2/ec2.py tag_Name_cassandra_node -u ansible -m ping
54.218.113.95 | SUCCESS => {
"changed": false,
"ping": "pong"
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Often DevOps tasks require you to manage different machines, Ubuntu, CoreOS, CentOS, RedHat, Debian, and AmazonLinux. The various EC2 instances will have different users to log in. For example, CentOS has the user <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">centos</code> and Ubuntu has the user <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ubuntu</code> (I have run into <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">admin</code>, <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">root</code>, etc.). It is a good idea to create a standard user like <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible</code> (or <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">devops</code> or <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ops</code> or <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">admin</code>) to run <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible</code>commands against different flavors of Unix. Also, AWS PEM files / key pairs do not change once an instance if launched, and Cassandra instances tend to be less ephemeral (due to the statefulness of the Cassandra Database and the potentially large amounts of data on a node) then some other EC2 instances. The ability to regenerate the RSA key periodically is important as you do not want the keys to getting into the wrong hands.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The AWS inventory list command uses security groups, VPC ids, instance id, image type, EC2 tags, AZ, scaling groups, region and more to group EC2 instances to run ansible commands against, which is very flexible for DevOps operations.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s see a list of all of the aliases and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible groups</code> that our one Cassandra Database EC2 instance is exposed.</div>
<h4 id="show-all-ansible-groups-that-our-cassandra-ec2-instance-can-be-accessed-by" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Show all ansible groups that our Cassandra Database EC2 instance can be accessed by</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">./ansible-ec2/ec2.py | jq "keys"
[
"_meta",
"ami_6db4410e", //by AMI
"ec2", //All ec2 instances
"i-754a8a4f693b58d1b", //by instance id
"key_cloudurable_us_west_2",//by key pair
"security_group_allow_ssh", //by security group
"tag_Name_cassandra_node", //by EC2 tag
"type_m4_large", //by EC2 instance type
"us-west-2", //by Region
"us-west-2c", //by AZ Availability Zone
"vpc_id_vpc_c78000a0" //by VPC Virtual Private Cloud
]
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
You can use any of these ansible groups to ping a set of servers. Let’s ping every server (we only have one) in the AWS <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">us-west-2</code>region.</div>
<h4 id="ping-all-servers-in-the-us-west-2-region" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Ping all Cassandra Database nodes in the us-west-2 region</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ansible -i ansible-ec2/ec2.py us-west-2 -u ansible -m ping
54.218.113.95 | SUCCESS => {
"changed": false,
"ping": "pong"
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
I don’t know about you, but I don’t like passing around the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">-i</code> and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">-u</code> option on every command. Let’s see what we can do to remedy this.</div>
<h3 id="installing-dynamic-inventory-as-the-default-inventory" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Installing Dynamic Inventory as the default inventory</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Another option besides using the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">-i</code> is to copy the dynamic inventory script to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">/etc/ansible/ec2.py</code> and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">chmod +x</code> it. You will also need to copy the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ec2.ini</code> file to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">/etc/ansible/ec2.ini</code>. Then we will be able to use <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">ansible EC2 dynamic inventory</em></span> without passing the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">-i</code> (making it a lot easier to use).</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s install the <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">ansible dynamic inventory</em></span> script and config as follows.</div>
<h4 id="installing-ansible-ec2-dynamic-inventory-script-as-the-default" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Installing ansible EC2 dynamic inventory script as the default</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ cd ~/github/cassandra-image
$ wget https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.py -O ansible-ec2/ec2.py
$ chmod +x ansible-ec2/ec2.py
$ sudo cp ansible-ec2/ec2.py /etc/ansible/ec2.py
$ wget https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.ini -O ansible-ec2/ec2.ini
sudo cp ansible-ec2/ec2.ini /etc/ansible/ec2.ini
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
You will also need to add the script (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ANSIBLE_HOSTS</code>) and the ini file (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">EC2_INI_PATH</code>) to environment variable which you can put in your `<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/.bash_profile</code>.</div>
<h4 id="environment-varaibles-needed-to-make-dynamic-inventory-work" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Environment variables needed to make dynamic inventory work</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">export ANSIBLE_HOSTS=/etc/ansible/ec2.py
export EC2_INI_PATH=/etc/ansible/ec2.ini
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now when you use ansible, you will not have to specify <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">-i</code> every time.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s try ansible using the dynamic inventory list without the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">-i</code>.</div>
<h4 id="using-dynamic-inventory-without-i" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Using dynamic inventory without -i</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ansible tag_Name_cassandra_node -u ansible -m ping
54.218.113.95 | SUCCESS => {
"changed": false,
"ping": "pong"
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now that we got rid of <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">-i</code> to specify the ansible dynamic inventory list script, let’s get rid of the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">-u</code> to specify the user. At least let’s try.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Again before we do that, let’s see if we can use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh</code> without passing the user name.</div>
<h3 id="specifying-default-user-via-ssh-config" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Specifying default user via ~/.ssh/config</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
If you’re like most developers doing DevOps, you have a half dozen remote servers (or these days, local virtual machines, EC2 instances, Docker containers) you might need to deal with.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Remembering all of those usernames, passwords, domain names, identity files, and command line options to ssh can be daunting. You want a way to <a href="http://nerderati.com/2011/03/17/simplify-your-life-with-an-ssh-config-file/" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">simplify your life with an ssh config file</a>.</div>
<blockquote style="background-color: white; border-left: 5px solid rgb(0, 191, 255); box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin: 0px 0px 20px; padding: 10px 20px;">
<div style="box-sizing: border-box;">
Using ssh effectively is another one of those essential DevOp skills!</div>
</blockquote>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
You can create an <a href="https://www.cyberciti.biz/faq/create-ssh-config-file-on-linux-unix/" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">ssh config file</a> that configures host names, user names, and private keys to connect to ssh. There are <a href="https://linux.die.net/man/5/ssh_config" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">many custom ssh config options</a> to <a href="https://www.digitalocean.com/community/tutorials/how-to-configure-custom-connection-options-for-your-ssh-client" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">configure ssh and make life easier</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We will show how to configure <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/.ssh/config</code> to make logging into our EC2 instance easier, and eventually get rid of the need to run the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code> or use the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">-i</code> option when using ansible.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We wrote a small bash script that gets the DNS name of our instance using the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">aws</code> command line as follows:</div>
<h4 id="bin-get-dns-name-cassandra-sh-get-the-dns-name-of-our-cassandra-ec2-instance-using-aws-command-line" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
bin/get-DNS-name-cassandra.sh - Get the DNS name of our Cassandra EC2 instance using <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 16.2px; padding: 2px 4px;">aws</code>command line</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/bin/bash
set -e
source bin/ec2-env.sh
aws ec2 describe-instances --filters "Name=tag:Name,Values=${EC2_INSTANCE_NAME}" \
| jq --raw-output .Reservations[].Instances[].PublicDnsName
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We can use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bin/get-DNS-name-cassandra.sh</code> to get the DNS name of our instance as follows:</div>
<h4 id="getting-the-dns-name-of-the-cassandra-ec2-instance" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Getting the DNS name of the Cassandra EC2 instance</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">bin/get-DNS-name-cassandra.sh
ec2-54-218-113-95.us-west-2.compute.amazonaws.com
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now let’s see the IP address associated with this instance.</div>
<h4 id="ec2-cassandra-host" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
EC2 Cassandra host</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ host ec2-54-218-113-95.us-west-2.compute.amazonaws.com
54.218.113.95
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Note that for this discussion that we are using <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">54.218.113.95</code> as the public IP address of our Cassandra node (that we created with packer and launched with the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">aws</code> command line tools).</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now we can configure <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/.ssh/config</code> to use this information.</div>
<h4 id="ssh-config" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/.ssh/config</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;"># Note we can use wild star so any that match this pattern will work.
Host *.us-west-2.compute.amazonaws.com
ForwardAgent yes
IdentityFile ~/.ssh/test_rsa
User ansible
# Note we can use the IP address
# so if we ssh into it, we don't have to pass username and the id file
Host 54.218.113.95
ForwardAgent yes
IdentityFile ~/.ssh/test_rsa
User ansible
# We even create an alias for ssh that has username and the id file.
Host cnode0
Hostname 54.218.113.95
ForwardAgent yes
IdentityFile ~/.ssh/test_rsa
User ansible
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Read the comments in the file above.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now when we log into <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">cnode0</code> using <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh</code> as follows:</div>
<h4 id="ssh-cnode0" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
ssh cnode0</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ssh cnode0
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Note that <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">cnode0</code> is an alias that we set up in <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/.ssh/config</code> and that we don’t have to use the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">-i</code> option to pass the identity file or use the username.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Would you rather need to remember <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh cnode0</code> or <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh -i ~/.ssh/my-long-pem-name-with-region-info.pem someuserForEachLinuxFlavor@ec2-ip-address-that-changes-every-time-i-build-a-new-instance.us-west-2.compute.amazonaws.com</code>?</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Keep in mind; you do not have to use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code> or <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-add</code> anymore to use ansible since we configured the identity file and username in <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/.ssh/config</code>. Forgetting to set up the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code> and adding the right key file with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-add</code> was error prone at best, and often left me personally confused. Now that issue of confusion is gone, but since we set up <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ForwardAgent yes</code>, once we log into a remote instance the keys we set up with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code> and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-add</code> get passed to the remote host. This way those keys do not have to live on the remote host. You can, for example, log into a bastion server and then ssh into a private subnet with the keys you set up with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code>, and none of those private keys have to live on the remote instances (to avoid getting used by someone else). Mastering <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh</code>, <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code>, and ssh key management is essential to being good at DevOps.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Given the above config, you can also log into the Cassandra dev instance with its public domain name as follows:</div>
<h4 id="ssh-into-box-using-public-address" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
ssh into box using public address</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ssh ec2-54-218-113-95.us-west-2.compute.amazonaws.com
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The above uses the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">Host *.us-west-2.compute.amazonaws.com</code> which is a domain pattern that would work for all ec2 instances in the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">us-west-2</code> AWS region. Since different regions will use different AWS key-pairs, you can set up a pattern/key-pair <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">pem</code> file for each region easily.</div>
<h3 id="attempt-get-ansible-to-use-public-dns-names-instead-of-ip-addresses-does-not-work" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Attempt Get ansible to use public DNS names instead of IP addresses (does not work)</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Given the above <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">./ssh/config</code> to get rid of <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">-u</code> option one might imagine you could tell <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ec2.py</code>, the script that generates the ansible inventory, you could configure it to use public domain names instead of public IP addresses, and you can.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
If you make this change, to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ec2.ini</code>.</div>
<h4 id="etc-ansible-ec2-ini" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
/etc/ansible/ec2.ini</h4>
<h4 id="change-vpc-destination-variable-ip-address-to-vpc-destination-variable-public-dns-name" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Change vpc_destination_variable = ip_address to vpc_destination_variable = public_dns_name</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">vpc_destination_variable = public_dns_name
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Then <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ec2.py</code> will use the public domain names from EC2 instead of public IP addresses, for example, <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ec2-54-218-113-95.us-west-2.compute.amazonaws.com</code> instead of <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">54.218.113.95</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
But then all of the ansible commands stop working. It seems, as best that I can tell, that ansible does not like domain names with dashes. We searched for a workaround for this and could not find it. If you know the answer, please write us.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We even tried to add this directly to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">inventory.ini</code>.</div>
<h4 id="adding-ec2-host-direct" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
adding ec2 host direct</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">[aws-nodes]
# 54.186.15.163 ansible_user=ansible
ec2-54-218-113-95.us-west-2.compute.amazonaws.com ansible_user=ansible
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Then tried running the ansible commands against the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">aws-nodes</code> and we got the same result until we tried the fix for <a href="https://github.com/ansible/ansible/issues/11536" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">EC2 domain name being too long for Ansible</a>, but we never got the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ec2.py</code> to work with the longer DNS names (we were able to get past parts of it).</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
This problem is either ansible not handling dashes or <a href="https://github.com/ansible/ansible/issues/15508" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">long dns name problem</a>. The fix seems to be in the comments of this <a href="https://github.com/ansible/ansible/issues/11536" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">fix for EC2 domain name being too long for Ansible</a>, but again worked but only in the non-dynamic config. For the most part, we tried the fix and it did not work (still getting <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ERROR! Specified hosts and/or --limit does not match any hosts</code>).</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
It is okay, though. The only real limitation here is that when you use ansible with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ec2.py</code> that you will need to pass the user and continue to use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code> and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-add</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
This workaround of having to give the username with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">-u</code> is not too serious. We still wish there was a way to use ansible without passing a username and identity file just like we have with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh</code>. And there is, but it involves <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">AWS Route 53</em></span> and configuring `<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/ssh/config</code>.</div>
<h3 id="using-ansible-without-passing-the-id-file-or-username" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Using ansible without passing the id file or username</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Another way to use ansible with our Cassandra cluster is to create DNS names for the Cassandra nodes that we want to manage. The problem with using the public IP address or the AWS generated DNS name is that they change each time we terminate and recreate the instance. We plan on terminating and recreating the instance a lot.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The solution is where DNS comes in and <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">AWS route 53</em></span>. After we create the instance, we can use an <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">internal hosted zone</em></span> of <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">Route 53</em></span> (for VPN) or a public hosted zone and associate the IP address with our new instance. We could do this for all of the Cassandra seed nodes and all of the cluster nodes for that matter.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Before we get started let’s add two more variables to our <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bin/ec2-env.sh</code>, namely, <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">HOSTED_ZONE_ID</code> and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">NODE0_DNS</code> as follows:</div>
<h4 id="bin-ec2-env-sh" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
bin/ec2-env.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/bin/bash
set -e
export AMI_CASSANDRA=ami-abc1234
export VPC_SECURITY_GROUP=sg-abc1234
export SUBNET_CLUSTER=subnet-abc1234
export KEY_NAME_CASSANDRA=cloudurable-us-west-2
export PEM_FILE="${HOME}/.ssh/${KEY_NAME_CASSANDRA}.pem"
export IAM_PROFILE_CASSANDRA=IAM_PROFILE_CASSANDRA
export EC2_INSTANCE_NAME=cassandra-node
export HOSTED_ZONE_ID="Z1-abc1234"
export NODE0_DNS="node0.cas.dev.cloudurable.com."
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now let’s define a new script that will use the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">aws</code> command line. We will use the <a href="http://docs.aws.amazon.com/cli/latest/reference/route53/change-resource-record-sets.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">aws route53 change-resource-record-sets</a> to associate a DNS name with the IP address as follows:</div>
<h4 id="bin-associate-dns-with-ip-sh" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
bin/associate-DNS-with-IP.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/bin/bash
set -e
source bin/ec2-env.sh
IP_ADDRESS=`bin/get-IP-CASSANDRA.sh`
REQUEST_BATCH="
{
\"Changes\":[
{
\"Action\": \"UPSERT\",
\"ResourceRecordSet\": {
\"Type\": \"A\",
\"Name\": \"$NODE0_DNS\",
\"TTL\": 300,
\"ResourceRecords\": [{
\"Value\": \"$IP_ADDRESS\"
}]
}
}
]
}
"
echo "$REQUEST_BATCH"
changeId=$(aws route53 change-resource-record-sets --hosted-zone-id "$HOSTED_ZONE_ID" --change-batch "$REQUEST_BATCH" \
| jq --raw-output .ChangeInfo.Id)
aws route53 wait resource-record-sets-changed --id "$changeId"
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that we are running this change against our Route 53 Hosted ZONE with <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">aws route53 change-resource-record-sets</code> as follows:</div>
<h4 id="change-batch-for-route-53-hosted-zone" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Change batch for Route 53 hosted zone</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-javascript" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">{
"Changes":[
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Type": "A",
"Name": "node0.cas.dev.cloudurable.com.",
"TTL": 300,
"ResourceRecords": [{
"Value": "54.218.113.95"
}]
}
}
]
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice we are using <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">UPSERT</code> which will update or add the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">A</code> record to Route 53’s DNS resources to associate the name <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">node0.cas.dev.cloudurable.com.</code> with the IP address <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">54.218.113.95</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now that we have a domain name, and it is scripted/automated (we added a call to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bin/associate-DNS-with-IP.sh</code> into <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bin/create-ec2-cassandra.sh</code>), we can configure <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/.ssh/config</code> to use this domain name which will not change like the public IP or our Cassandra instance public DNS name changes.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s update the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/.ssh/config</code> to refer to our new DNS name as follows:</div>
<h4 id="ssh-config-use-new-dns-naming" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 16.2px; padding: 2px 4px;">~/.ssh/config</code> - Use new DNS naming</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">Host *.us-west-2.compute.amazonaws.com
ForwardAgent yes
IdentityFile ~/.ssh/test_rsa
User ansible
Host *.cas.dev.cloudurable.com
ForwardAgent yes
IdentityFile ~/.ssh/test_rsa
User ansible
Host cnode0
Hostname node0.cas.dev.cloudurable.com
ForwardAgent yes
IdentityFile ~/.ssh/test_rsa
User ansible
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice we added the pattern <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">*.cas.dev.cloudurable.com</code> (where <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">cas</code> stands for Cassandra and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">dev</code> means this is our development environment). We also added an alias for our the Cassandra Database instance called <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">cnode0</code> that refers to<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">node0.cas.dev.cloudurable.com</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We can ssh into <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">cnode0</code> or <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">node0.cas.dev.cloudurable.com</code> without passing the username or identity file (private key) each time. This config is like before but using a DNS name that does not change when we rebuild our servers. This concept is important; you would not want to modify <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/.ssh/config</code> every time you rebuild a server.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now let’s change our inventory.ini file in the project directory (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/github/cassandra-image</code>) to use this as follows:</div>
<h4 id="github-cassandra-image-inventory-ini" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
~/github/cassandra-image/inventory.ini</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">[aws-nodes]
cnode0
node0.cas.dev.cloudurable.com
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that we use the short name and the long name.</div>
<blockquote style="background-color: white; border-left: 5px solid rgb(0, 191, 255); box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin: 0px 0px 20px; padding: 10px 20px;">
<div style="box-sizing: border-box;">
Note you truly just need one but we have two just for this article. Never put the same box twice in the same ansible group, all commands and playbooks will run twice.</div>
</blockquote>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Now we can run ansible ping against these servers and not pass the username or identity file.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Use ansible ping module against <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">cnode</code> and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">node0.cas.dev.cloudurable.com</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Run against all (see note above).</div>
<h4 id="running-ansible-ping-against-all-of-the-instances" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Running ansible ping against all of the “instances”</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ansible aws-nodes -u ansible -m ping
cnode0 | SUCCESS => {
"changed": false,
"ping": "pong"
}
node0.cas.dev.cloudurable.com | SUCCESS => {
"changed": false,
"ping": "pong"
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We can also just run it against one of the instances by using just that instances name.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Run against <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">cnode0</code>.</div>
<h4 id="ansible-cnode0-u-ansible-m-ping" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
ansible cnode0 -u ansible -m ping</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ansible cnode0 -u ansible -m ping
cnode0 | SUCCESS => {
"changed": false,
"ping": "pong"
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We can do this for any server.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Run against <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">node0.cas.dev.cloudurable.com</code>.</div>
<h4 id="ansible-node0-cas-dev-cloudurable-com-u-ansible-m-ping" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
ansible node0.cas.dev.cloudurable.com -u ansible -m ping</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ansible node0.cas.dev.cloudurable.com -u ansible -m ping
node0.cas.dev.cloudurable.com | SUCCESS => {
"changed": false,
"ping": "pong"
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Keep in mind; you do not have to use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-agent</code> or <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ssh-add</code> anymore to use ansible since we configured the identity file and username in <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/.ssh/config</code>. We can rebuild our server at will. Each time we do, our creation script will update the IP address in DNS to look at the new server. Then all of our ansible scripts and playbooks will continue to work.</div>
<h3 id="using-ansible-to-manage-our-cassandra-database-cluster" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Using Ansible to manage our Cassandra Database cluster</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We won’t do much actual cluster management in this article per se. And, unlike our last article that used Vagrant and ansible, we don’t have a cluster per se (or rather we have a cluster of one).</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We can now use Ansible for our Cassandra Database Cluster to do automate common DevOps/DBA tasks.</div>
<h4 id="ansible-running-nodetool-against-all-nodes" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Ansible running nodetool against all nodes</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">ansible aws-nodes -a "/opt/cassandra/bin/nodetool describecluster"
cnode0 | SUCCESS | rc=0 >>
Cluster Information:
Name: Test Cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
86afa796-d883-3932-aa73-6b017cef0d19: [127.0.0.1]
node0.cas.dev.cloudurable.com | SUCCESS | rc=0 >>
Cluster Information:
Name: Test Cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
86afa796-d883-3932-aa73-6b017cef0d19: [127.0.0.1]
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s say that we wanted to update a schema or do a rolling restart of our Cassandra nodes, which could be a very common task. Perhaps before the update, we want to decommission the node and back things up. To do this sort of automation, we could create an Ansible playbook.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s run an Ansible playbook from the last article.</div>
<h4 id="running-describe-cluster-playbook" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Running describe-cluster playbook</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ ansible-playbook playbooks/describe-cluster.yml --verbose
Using /Users/jean/github/cassandra-image/ansible.cfg as config file
PLAY [aws-nodes] ***************************************************************
TASK [Run NodeTool Describe Cluster command] ***********************************
changed: [node0.cas.dev.cloudurable.com] => {"changed": true, "cmd": ["/opt/cassandra/bin/nodetool", "describecluster"],
"delta": "0:00:02.192589", "end": "2017-03-03 08:02:58.537371", "rc": 0, "start": "2017-03-03 08:02:56.344782",
"stderr": "", "stdout": "Cluster Information:\n\tName: Test Cluster\n\tSnitch:
org.apache.cassandra.locator.DynamicEndpointSnitch\n\tPartitioner: org.apache.cassandra.dht.Murmur3Partitioner
\n\tSchema versions:\n\t\t86afa796-d883-3932-aa73-6b017cef0d19: [127.0.0.1]", "stdout_lines": ["Cluster Information:",
"\tName: Test Cluster", "\tSnitch: org.apache.cassandra.locator.DynamicEndpointSnitch",
...
PLAY RECAP *********************************************************************
cnode0 : ok=1 changed=1 unreachable=0 failed=0
node0.cas.dev.cloudurable.com : ok=1 changed=1 unreachable=0 failed=0
</code></pre>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">
</code></pre>
<h3 id="retrospective-past-articles-in-this-cassandra-cluster-devops-dba-series" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin: 20px 0px; position: relative;">
Cassandra Tutorial: Cassandra Cluster DevOps/DBA series</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The first tutorial in this <i>Cassandra tutorial</i> series focused on setting up a Cassandra Cluster. The first Cassandra tutorial setting up a Cassandra <a href="http://cloudurable.com/blog/cassandra-image-vagrant-cluster-example/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">cluster with Vagrant</a> (also appeared on DZone with some additional content <a href="https://dzone.com/articles/setting-up-a-cassandra-cluster-with-vagrant" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">DZone Setting up a Cassandra Cluster with Vagrant</a>. The second article in this series was about <a href="http://cloudurable.com/blog/cassandra-image-vagrant-cluster-example/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">setting up SSL for a Cassandra cluster using Vagrant</a> (which also appeared with more content as <a href="https://dzone.com/articles/setting-up-a-cassandra-cluster-with-ssl" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">DZone Setting up a Cassandra Cluster with SSL</a>). The <a href="http://cloudurable.com/blog/ansible-cassandra-vagrant-devops/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">third article</a> in this series was about configuring and using Ansible (building on the first two articles). This article (the 4th) will cover applying the tools and techniques from the first three articles to produce an image (EC2 AMI to be precise) that we can deploy to AWS/EC2. To do this explanation, we will use Packer, Ansible, and the Aws Command Line tools. The AWS command line tools are essential for doing DevOps with AWS.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Check out more information about the Cassandra Database</div>
<ul style="background-color: white; color: #666666; font-family: "Trebuchet MS", Trebuchet, Verdana, sans-serif; font-size: 13.2px; line-height: 1.4; margin: 0.5em 0px; padding: 0px 2.5em;">
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/service-architecture-analysis-cassandra-or-kafka-aws-ec2/index.html" style="color: #888888; text-decoration: none;">Cassandra Consulting: Architecture Analysis</a></li>
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/service-quick-start-mentoring-cassandra-or-kafka-aws-ec2/index.html" style="color: #888888; text-decoration: none;">Cassandra Consulting: Quick Start</a></li>
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/cassandra-course/index.html" style="color: #888888; text-decoration: none;">Cassandra Course</a></li>
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/" style="color: #888888; text-decoration: none;">Amazon Cassandra Support</a></li>
</ul>
<div>
<br /></div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com13tag:blogger.com,1999:blog-6128525281101058933.post-6308404716333193972017-03-04T14:27:00.001-08:002017-03-04T14:27:18.062-08:00Using AWS CLI to create our Cassandra EC2 instance from custom AMI<h3 id="using-aws-cli-to-create-our-cassandra-ec2-instance" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Using AWS CLI to create our Cassandra EC2 instance from our custom AMI (Amazon image)</h3>
<h3 id="using-packer-to-build-our-ec2-ami" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
<br /></h3>
<div>
Earlier we used <a href="http://rick-hightower.blogspot.com/2017/03/packer-ec2-support-and-ansible-for-our.html">packer to create an Amazon Cassandra image - Amazon image (AMI)</a>. </div>
<div>
Packer <i>installs Cassandra</i> on the AMI. Then we use the AMI to produce <i>Amazon Cassandra</i> EC2 instances. Now we can use that <b><i>Amazon Cassandra</i></b> AMI to create an <b><i>Amazon Cassandra</i></b> instance. </div>
<h3 id="using-packer-to-build-our-ec2-ami" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Packer building Amazon Cassandra AMI</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We built the <i>Amazon Cassandra</i> image using <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">packer build</code> as follows.</div>
<h4 id="building-the-aws-ami" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Building the AWS AMI</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ packer build packer-ec2.json
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
After the packer build completes, it will print out the name of the AMI image it created, e.g., <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ami-6db33abc</code>. Now it is time to use the Amazon CLI (aws cli) to create the ec2 instance.</div>
<h3 id="using-aws-cli-to-create-our-cassandra-ec2-instance" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Using AWS CLI to create our Cassandra EC2 instance</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <a href="http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">AWS Command Line Interface</a> is the ultimate utility to DevOp manage your AWS services.</div>
<blockquote style="background-color: white; border-left: 5px solid rgb(0, 191, 255); box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin: 0px 0px 20px; padding: 10px 20px;">
<div style="box-sizing: border-box;">
“With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts.” –AWS CLI Docs</div>
</blockquote>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The AWS command line tool does it all. You can create VPCs. You can run CloudFormations. You can even use it to back up the Amazon Cassandra Database snapshot files to S3. If you are working with AWS and doing DevOps, you must master the AWS CLI.</div>
<h3 id="automating-ec2-image-creation-with-aws-cli" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Automating Amazon Cassandra EC2 instance creation</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Starting up an EC2 instance with the right, AMI id, IAM instance role, into the correct subnet, using the appropriate security groups, with the right AWS key-pair name can be tedious. We must automate as using the AWS console (GUI) is error prone (requires too much human intervention).</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Instead of using the AWS console, we use the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">aws</code> command line. We create four scripts to automate creating and connecting to EC2 instances:</div>
<ul style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px; margin-top: 0px;">
<li style="box-sizing: border-box;"><span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">bin/ec2-env.sh</em></span> - setups common AWS references to subnets, security groups, key pairs</li>
<li style="box-sizing: border-box;"><span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">bin/create-ec2-instance.sh</em></span> - uses <a href="http://docs.aws.amazon.com/cli/latest/userguide/installing.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">aws command line</a> to create an ec2 instance</li>
<li style="box-sizing: border-box;"><span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">bin/login-ec2-cassandra.sh</em></span> Uses ssh to log into Cassandra node we are testing</li>
<li style="box-sizing: border-box;"><span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">bin/get-IP-cassandra.sh</em></span> Uses <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">aws command line</em></span> to get the public IP address of the cassandra instance</li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Note to parse the JSON coming back from the *<span style="box-sizing: border-box; font-weight: 700;">aws command line</span> we use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">jq</code>. Note that <a href="https://stedolan.github.io/jq/" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">jq</a> is a lightweight command-line JSON processor. To download and install <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">jq</code> see the <a href="https://stedolan.github.io/jq/download/" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">jq download documents</a>.</div>
<h4 id="bin-create-ec2-instance-sh-create-an-ec2-instance-based-on-our-new-ami-from-packer" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
bin/create-ec2-instance.sh Create an EC2 instance based on our new AMI from Packer</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/bin/bash
set -e
source bin/ec2-env.sh
instance_id=$(aws ec2 run-instances --image-id "$AMI_CASSANDRA" --subnet-id "$SUBNET_CLUSTER" \
--instance-type m4.large --iam-instance-profile "Name=$IAM_PROFILE_CASSANDRA" \
--associate-public-ip-address --security-group-ids "$VPC_SECURITY_GROUP" \
--key-name "$KEY_NAME_CASSANDRA" | jq --raw-output .Instances[].InstanceId)
echo "${instance_id} is being created"
aws ec2 wait instance-exists --instance-ids "$instance_id"
aws ec2 create-tags --resources "${instance_id}" --tags Key=Name,Value="${EC2_INSTANCE_NAME}"
echo "${instance_id} was tagged waiting to login"
aws ec2 wait instance-status-ok --instance-ids "$instance_id"
bin/login-ec2-cassandra.sh
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice we use the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">aws ec2 wait</code> to ensure the instance is ready before we tag it and before we log into it.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
All of the ids for the servers AWS resources we need to refer to are in <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">scripts/ec2-ens.sh</code>. Notice that all of our AWS/EC2 shell scripts load this env file <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">source bin/ec2-env.sh</code> as follows:</div>
<h4 id="bin-ec2-env-sh-common-aws-resources-exposed-as-env-vars" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
bin/ec2-env.sh common AWS resources exposed as ENV Vars</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/bin/bash
set -e
export AMI_CASSANDRA=ami-6db33abc
export VPC_SECURITY_GROUP=sg-a8653123
export SUBNET_CLUSTER=subnet-dc0f2123
export KEY_NAME_CASSANDRA=cloudurable-us-west-2
export PEM_FILE="${HOME}/.ssh/${KEY_NAME_CASSANDRA}.pem"
export IAM_PROFILE_CASSANDRA=IAM_PROFILE_CASSANDRA
export EC2_INSTANCE_NAME=cassandra-node
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Earlier we created an AWS <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">key pair</a> called <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">cloudurable-us-west-2</code>. You will need to create a <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">VPC security group with ssh access</a>. You should lock it down to only accept ssh connections from your IP. At this stage, you can use a default VPC, and for now use a public subnet. Replace the ids above with your subnet (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">SUBNET_CLUSTER</code>), your key pair (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">KEY_NAME_CASSANDRA</code>), your AMI (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">AMI_CASSANDRA</code>), and your IAM instance role (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">IAM_PROFILE_CASSANDRA</code>). The IAM instance role should have access to create logs and metrics for <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">AWS CloudWatch</em></span>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The login script (<code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">login-ec2-cassandra.sh</code>) uses ssh to log into the instance, but to know what IP to use, it uses <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">get-IP-cassandra.sh</code></div>
<h4 id="bin-login-ec2-cassandra-sh-log-into-new-ec2-cassandra-database-instance-using-ssh" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
bin/login-ec2-cassandra.sh Log into new EC2 Cassandra Database instance using ssh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/bin/bash
set -e
source bin/ec2-env.sh
if [ ! -f "$PEM_FILE" ]; then
echo "Put your key file $PEM_FILE in your .ssh directory."
exit 1
fi
ssh -i "$PEM_FILE" centos@`bin/get-IP-cassandra.sh`
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Ensure you create a key pair in AWS. Copy it to <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">~/.ssh</code> and then <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">run <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">chmod 400</code> on the pem file</a>. Note the above script uses <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bin/get-IP-cassandra.sh</code> to get the IP address of the server as follows:</div>
<h4 id="bin-get-ip-cassandra-sh-get-public-ip-address-of-new-ec2-instance-using-aws-cmdline" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
bin/get-IP-cassandra.sh Get public IP address of new EC2 instance using aws cmdline</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/bin/bash
set -e
source bin/ec2-env.sh
aws ec2 describe-instances --filters "Name=tag:Name,Values=${EC2_INSTANCE_NAME}" \
| jq --raw-output .Reservations[].Instances[].PublicIpAddress
</code></pre>
<h3 id="running-bin-create-ec2-instance-sh" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Running bin/create-ec2-instance.sh</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
To run <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">bin/create-ec2-instance.sh</code></div>
<h4 id="running-bin-create-ec2-instance-sh-1" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Running bin/create-ec2-instance.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ bin/create-ec2-instance.sh
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Let’s show how to check to see if everything is up and running.</div>
<h4 id="interactive-session-showing-everything-running" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Interactive session showing everything running</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ pwd
~/github/cassandra-image
$ bin/create-ec2-instance.sh
i-013daca3d11137a8c is being created
i-013daca3d11137a8c was tagged waiting to login
The authenticity of host '54.202.110.114 (54.202.110.114)' can't be established.
ECDSA key fingerprint is SHA256:asdfasdfasdfasdfasdf.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '54.202.110.114' (ECDSA) to the list of known hosts.
[centos@ip-172-31-5-57 ~]$ systemctl status cassandra
● cassandra.service - Cassandra Service
Loaded: loaded (/etc/systemd/system/cassandra.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2017-03-01 02:15:10 UTC; 14min ago
Process: 456 ExecStart=/opt/cassandra/bin/cassandra -p /opt/cassandra/PID (code=exited, status=0/SUCCESS)
Main PID: 5240 (java)
CGroup: /system.slice/cassandra.service
└─5240 java -Xloggc:/opt/cassandra/bin/../logs/gc.log -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=1000003 -XX:+AlwaysPreTouch -XX:-UseBiasedLocking -XX:+U...
Mar 01 02:14:13 ip-172-31-22-103.us-west-2.compute.internal systemd[1]: Starting Cassandra Service...
Mar 01 02:15:10 ip-172-31-5-57 systemd[1]: Started Cassandra Service.
[centos@ip-172-31-5-57 ~]$ systemctl status metricds
Unit metricds.service could not be found.
[centos@ip-172-31-5-57 ~]$ systemctl status metricsd
● metricsd.service - MetricsD OS Metrics
Loaded: loaded (/etc/systemd/system/metricsd.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2017-03-01 02:15:10 UTC; 14min ago
Main PID: 5243 (metricsd)
CGroup: /system.slice/metricsd.service
└─5243 /opt/cloudurable/bin/metricsd
Mar 01 02:25:15 ip-172-31-5-57 metricsd[5243]: INFO : [worker] - 2017/03/01 02:25:15 config.go:30: Loading config /etc/metricsd.conf
Mar 01 02:25:15 ip-172-31-5-57 metricsd[5243]: INFO : [worker] - 2017/03/01 02:25:15 config.go:46: Loading log...
[centos@ip-172-31-5-57 ~]$ systemctl status systemd-cloud-watch
● systemd-cloud-watch.service - SystemD Cloud Watch Sends Journald logs to CloudWatch
Loaded: loaded (/etc/systemd/system/systemd-cloud-watch.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2017-03-01 02:15:10 UTC; 15min ago
Main PID: 5241 (systemd-cloud-w)
CGroup: /system.slice/systemd-cloud-watch.service
└─5241 /opt/cloudurable/bin/systemd-cloud-watch /etc/systemd-cloud-watch.conf
Mar 01 02:30:44 ip-172-31-5-57 systemd-cloud-watch[5241]: main INFO: 2017/03/01 02:30:44 workers.go:138: Read record &{i-013daca3d11137a8c 1488335194775 5241 0 0 systemd-cloud-w /opt/cloudurable/bin/systemd-cloud-watch /opt/cloudurable/bin...
...
Mar 01 02:30:44 ip-172-31-5-57 systemd-cloud-watch[5241]: main INFO: 2017/03/01 02:30:44 workers.go:138: Read record &{i-013daca3d11137a8c 1488335194776 5241 0 0 systemd-cloud-w /opt/cloudurable/bin/systemd-cloud-watch /opt...7f10a2c35de4098
Mar 01 02:30:44 ip-172-31-5-57 systemd-cloud-watch[5241]: repeater INFO: 2017/03/01 02:30:44 cloudwatch_journal_repeater.go:209: SENT SUCCESSFULLY
Mar 01 02:30:44 ip-172-31-5-57 systemd-cloud-watch[5241]: repeater
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We used <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">systemctl status systemd-cloud-watch</code>, <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">systemctl status cassandra</code>, and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">systemctl status metricsd</code>to ensure it is all working.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
<h3 id="retrospective-past-articles-in-this-cassandra-cluster-devops-dba-series" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin: 20px 0px; position: relative;">
Cassandra Tutorial: Cassandra Cluster DevOps/DBA series</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The first tutorial in this <i>Cassandra tutorial</i> series focused on setting up a Cassandra Cluster. The first Cassandra tutorial setting up a Cassandra <a href="http://cloudurable.com/blog/cassandra-image-vagrant-cluster-example/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">cluster with Vagrant</a> (also appeared on DZone with some additional content <a href="https://dzone.com/articles/setting-up-a-cassandra-cluster-with-vagrant" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">DZone Setting up a Cassandra Cluster with Vagrant</a>. The second article in this series was about <a href="http://cloudurable.com/blog/cassandra-image-vagrant-cluster-example/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">setting up SSL for a Cassandra cluster using Vagrant</a> (which also appeared with more content as <a href="https://dzone.com/articles/setting-up-a-cassandra-cluster-with-ssl" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">DZone Setting up a Cassandra Cluster with SSL</a>). The <a href="http://cloudurable.com/blog/ansible-cassandra-vagrant-devops/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">third article</a> in this series was about configuring and using Ansible (building on the first two articles). This article (the 4th) will cover applying the tools and techniques from the first three articles to produce an image (EC2 AMI to be precise) that we can deploy to AWS/EC2. To do this explanation, we will use Packer, Ansible, and the Aws Command Line tools. The AWS command line tools are essential for doing DevOps with AWS.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Check out more information about the Cassandra Database</div>
<ul style="background-color: white; color: #666666; font-family: "Trebuchet MS", Trebuchet, Verdana, sans-serif; font-size: 13.2px; line-height: 1.4; margin: 0.5em 0px; padding: 0px 2.5em;">
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/service-architecture-analysis-cassandra-or-kafka-aws-ec2/index.html" style="color: #888888; text-decoration: none;">Cassandra Consulting: Architecture Analysis</a></li>
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/service-quick-start-mentoring-cassandra-or-kafka-aws-ec2/index.html" style="color: #888888; text-decoration: none;">Cassandra Consulting: Quick Start</a></li>
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/cassandra-course/index.html" style="color: #888888; text-decoration: none;">Cassandra Course</a></li>
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/" style="color: #888888; text-decoration: none;">Amazon Cassandra Support</a></li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com8tag:blogger.com,1999:blog-6128525281101058933.post-62547526576489124672017-03-04T14:05:00.003-08:002017-03-04T14:05:50.583-08:00Running the Cassandra Database as a systemd service<h3 id="running-the-cassandra-database-as-a-systemd-service" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Running the Cassandra Database as a systemd service</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
If the <b><i>Cassandra Database</i></b> stops for whatever reason, systemd can attempt to restart it. The <b>systemd</b> unit file can ensure that our Cassandra service stays running. The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">systemd-cloud-watch</code> utility will be sure to log all restarts to <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">AWS CloudWatch</em></span>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Here is the systemd unit file for Cassandra.</div>
<h4 id="etc-systemd-system-cassandra-service" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
/etc/systemd/system/cassandra.service</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">[Unit]
Description=Cassandra Service
[Service]
Type=forking
PIDFile=/opt/cassandra/PID
ExecStartPre=- /sbin/swapoff -a
ExecStartPre=- /bin/chown -R cassandra /opt/cassandra
ExecStart=/opt/cassandra/bin/cassandra -p /opt/cassandra/PID
WorkingDirectory=/opt/cassandra
Restart=always
RestartSec=60
TimeoutStopSec=60
TimeoutStartSec=60
User=cassandra
[Install]
WantedBy=multi-user.target
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The above will tells systemd to restart the <b><i>Cassandra Database</i></b> in one minute if it goes down. Since we are using OS log aggregation to AWS Cloudwatch every time Cassandra goes down or is restarted by systemd, we will get log messages that we can create alerts and trigger in CloudWatch to then run AWS Lambdas that work with the rest of the AWS ecosystem. Critical bugs in queries or UDF or UFA could cause Cassandra to go down. These could be hard to track down and sporadic. Logging aggregation helps.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
<h3 id="retrospective-past-articles-in-this-cassandra-cluster-devops-dba-series" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin: 20px 0px; position: relative;">
Redux - Past Articles in this Cassandra Tutorial: Cassandra Cluster DevOps/DBA series</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The first article in this <i>Cassandra tutorial</i> series was about setting up a Cassandra <a href="http://cloudurable.com/blog/cassandra-image-vagrant-cluster-example/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">cluster with Vagrant</a> (also appeared on DZone with some additional content <a href="https://dzone.com/articles/setting-up-a-cassandra-cluster-with-vagrant" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">DZone Setting up a Cassandra Cluster with Vagrant</a>. The second article in this series was about <a href="http://cloudurable.com/blog/cassandra-image-vagrant-cluster-example/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">setting up SSL for a Cassandra cluster using Vagrant</a> (which also appeared with more content as <a href="https://dzone.com/articles/setting-up-a-cassandra-cluster-with-ssl" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">DZone Setting up a Cassandra Cluster with SSL</a>). The <a href="http://cloudurable.com/blog/ansible-cassandra-vagrant-devops/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">third article</a> in this series was about configuring and using Ansible (building on the first two articles). This article (the 4th) will cover applying the tools and techniques from the first three articles to produce an image (EC2 AMI to be precise) that we can deploy to AWS/EC2. To do this explanation, we will use Packer, Ansible, and the Aws Command Line tools. The AWS command line tools are essential for doing DevOps with AWS.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Check out more information about the Cassandra Database</div>
<ul style="background-color: white; color: #666666; font-family: "Trebuchet MS", Trebuchet, Verdana, sans-serif; font-size: 13.2px; line-height: 1.4; margin: 0.5em 0px; padding: 0px 2.5em;">
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/service-architecture-analysis-cassandra-or-kafka-aws-ec2/index.html" style="color: #888888; text-decoration: none;">Cassandra Consulting: Architecture Analysis</a></li>
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/service-quick-start-mentoring-cassandra-or-kafka-aws-ec2/index.html" style="color: #888888; text-decoration: none;">Cassandra Consulting: Quick Start</a></li>
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/cassandra-course/index.html" style="color: #888888; text-decoration: none;">Cassandra Course</a></li>
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/" style="color: #888888; text-decoration: none;">Amazon Cassandra Support</a></li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com1tag:blogger.com,1999:blog-6128525281101058933.post-33790782703215497612017-03-04T14:02:00.001-08:002017-03-04T14:02:32.620-08:00systemd-cloud-watch to send Linux logs (systemd journald) to AWS CloudWatch<h3 id="systemd-cloud-watch-to-send-os-logs-to-aws-log-aggregation" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
systemd-cloud-watch to send OS logs to AWS log aggregation</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We use <a href="https://github.com/cloudurable/systemd-cloud-watch" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">systemd-cloud-watch to read OS logs from systemd/journald and send data to AWS CloudWatch Log</a>. The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">systemd-cloud-watch</code> daemon journald logs and aggregates them to <a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">AWS CloudWatch Logging</a>. Just like <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">metricsd</code> we install <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">systemd-cloud-watch</code> as a systemd process which depends on cassandra. Remember that we also install Cassandra as a <a href="http://cloudurable.com/blog/systemd-dependencies/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">systemd process</a>, which we will cover in a moment.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">systemd-cloud-watch </code> daemon gets installed as a <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">systemd service</em></span> by our provisioning scripts.</div>
<h4 id="installing-systemd-cloud-watch-systemd-service-from-our-provisioning-scripts" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Installing systemd-cloud-watch systemd service from our provisioning scripts</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">cp ~/resources/etc/systemd/system/systemd-cloud-watch.service /etc/systemd/system/systemd-cloud-watch.service
cp ~/resources/etc/systemd-cloud-watch.conf /etc/systemd-cloud-watch.conf
systemctl enable systemd-cloud-watch
systemctl start systemd-cloud-watch
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">systemctl enable</code> to install <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">systemd-cloud-watch</code> to start up when the system starts. We then use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">systemctl start</code> to start <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">systemd-cloud-watch</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">systemd-cloud-watch</code> system unit depends on the Cassandra service. The unit file is as follows:</div>
<h4 id="etc-systemd-system-systemd-cloud-watch-service" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
/etc/systemd/system/systemd-cloud-watch.service</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">[Unit]
Description=SystemD Cloud Watch Sends Journald logs to CloudWatch
Requires=cassandra.service
After=cassandra.service
[Service]
ExecStart=/opt/cloudurable/bin/systemd-cloud-watch /etc/systemd-cloud-watch.conf
WorkingDirectory=/opt/cloudurable
Restart=always
RestartSec=60
TimeoutStopSec=60
TimeoutStartSec=60
[Install]
WantedBy=multi-user.target
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Note to use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">metricsd</code> and <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">systemd-cloud-watch</code> we have to set up the right <a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/auth-and-access-control-cw.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">AWS IAM roles</a>, and then associate that IAM instance role with our instances when we start them up.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">systemd-cloud-watch.conf</code> is set up to use the AWS log group <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">cassandra</code> as follows:</div>
<h4 id="systemd-cloud-watch-conf" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
systemd-cloud-watch.conf</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">log_priority=7
debug=true
log_group="cassandra"
batchSize=5
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
For this to work, we will have to create a log group called <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">cassandra</code>.</div>
<h4 id="creating-a-aws-cloudwatch-log-group" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Creating a AWS CloudWatch log group</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ aws logs create-log-group --log-group-name cassandra
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
To learn more about <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">systemd-cloud-watch</code>, please see the <a href="https://github.com/cloudurable/systemd-cloud-watch" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;"><code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">systemd-cloud-watch</code> GitHub project</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
<h3 id="retrospective-past-articles-in-this-cassandra-cluster-devops-dba-series" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin: 20px 0px; position: relative;">
Retrospective - Past Articles in this Cassandra Tutorial: Cassandra Cluster DevOps/DBA series</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The first article in this <i>Cassandra tutorial</i> series was about setting up a Cassandra <a href="http://cloudurable.com/blog/cassandra-image-vagrant-cluster-example/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">cluster with Vagrant</a> (also appeared on DZone with some additional content <a href="https://dzone.com/articles/setting-up-a-cassandra-cluster-with-vagrant" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">DZone Setting up a Cassandra Cluster with Vagrant</a>. The second article in this series was about <a href="http://cloudurable.com/blog/cassandra-image-vagrant-cluster-example/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">setting up SSL for a Cassandra cluster using Vagrant</a> (which also appeared with more content as <a href="https://dzone.com/articles/setting-up-a-cassandra-cluster-with-ssl" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">DZone Setting up a Cassandra Cluster with SSL</a>). The <a href="http://cloudurable.com/blog/ansible-cassandra-vagrant-devops/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">third article</a> in this series was about configuring and using Ansible (building on the first two articles). This article (the 4th) will cover applying the tools and techniques from the first three articles to produce an image (EC2 AMI to be precise) that we can deploy to AWS/EC2. To do this explanation, we will use Packer, Ansible, and the Aws Command Line tools. The AWS command line tools are essential for doing DevOps with AWS.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Check out more information about the Cassandra Database</div>
<ul>
<li><a href="http://cloudurable.com/service-architecture-analysis-cassandra-or-kafka-aws-ec2/index.html">Cassandra Consulting: Architecture Analysis</a></li>
<li><a href="http://cloudurable.com/service-quick-start-mentoring-cassandra-or-kafka-aws-ec2/index.html">Cassandra Consulting: Quick Start</a></li>
<li><a href="http://cloudurable.com/cassandra-course/index.html">Cassandra Course</a></li>
<li><a href="http://cloudurable.com/">Amazon Cassandra Support</a></li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Find more information on the systemd-cloud-watch daemon at its github project page:</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
<div style="background-color: white; box-sizing: border-box; margin-bottom: 10px;">
<span style="color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif;"><span style="font-size: 14px;"><a href="https://github.com/cloudurable/systemd-cloud-watch">https://github.com/cloudurable/systemd-cloud-watch</a></span></span></div>
<div style="background-color: white; box-sizing: border-box; margin-bottom: 10px;">
<span style="color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif;"><br /></span></div>
<h1 style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; line-height: 1.25; margin-bottom: 16px; margin-left: 0px; margin-right: 0px; margin-top: 0px !important; padding-bottom: 0.3em;">
Yet another fork of Systemd Journal CloudWatch Writer</h1>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
Why this fork.</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
The project by Rick Hightower and Geoff Chandler forked a project that was a bit broken that repeated journald logs to cloudwatch. They fixed it up and made is useable and performant.</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
If you are using systemd based operating system (all modern Linux server distributions use systemd and journald), then you want critical messages (errors, warnings, etc.) from the OS and core daemons to be sent to AWS cloudwatch.</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
Once they are in AWS CloudWatch logs, you can write triggers and filters to trigger alerts which trigger lambda function, etc. They become actionable.</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
However, as great as journald is, you still have legacy apps that use syslog and then you have Java application that support MDC. Thus instead of just getting logs from journald and batch sending them to cloudwatch, we want to repeat logs from syslog (over UDP) and repeat logs from Java applications (like Cassandra) over <a href="https://github.com/logstash/logstash-logback-encoder" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">JSON logstash format using logback</a>.</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
This is mainly for our <a href="https://github.com/cloudurable/cassandra-image" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">Cassandra AMI</a> from <a href="http://cloudurable.com/" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">Cloudurable</a>, but it can be used other places.</div>
<ul style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">Use QBit to pass / batch log messages efficiently</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Listen on UDP syslog</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Listen to UDP JSON messages (logstash-logback-encoder).</li>
</ul>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
The rest of the readme is largely adapted from <a href="https://github.com/advantageous/systemd-cloud-watch" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">advantageous</a>.</div>
<h1 style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; line-height: 1.25; margin: 24px 0px 16px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#systemd-journal-cloudwatch-writer" id="user-content-systemd-journal-cloudwatch-writer" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Systemd Journal CloudWatch Writer</h1>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
This utility reads from the <a href="https://www.freedesktop.org/software/systemd/man/systemd-journald.service.html" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">systemd journal</a>, and sends the data in batches to <a href="https://aws.amazon.com/cloudwatch/" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">Cloudwatch</a>.</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
This is an alternative process to the AWS-provided logs agent. The AWS logs agent copies data from on-disk text log files into <a href="https://aws.amazon.com/cloudwatch/" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">Cloudwatch</a>. This utility <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">systemd-cloud-watch</code> reads the <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">systemd journal</code> and writes that data in batches to CloudWatch.</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
There are other ways to do this using various techniques. But depending on the size of log messages and size of the core parts these other methods are fragile as AWS CloudWatch limits the size of the messages. This utility allows you cap the log field size, include only the fields that you want, or exclude the fields you don't want. We find that this is not only useful but essential.</div>
<h2 style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#log-format" id="user-content-log-format" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Log format</h2>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
The journal event data is written to <span style="box-sizing: border-box; font-weight: 600;"><em style="box-sizing: border-box;">CloudWatch</em></span> Logs in JSON format, making it amenable to filtering using the JSON filter syntax. Log records are translated to <span style="box-sizing: border-box; font-weight: 600;"><em style="box-sizing: border-box;">CloudWatch</em></span> JSON events using a structure like the following:</div>
<h4 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#sample-log" id="user-content-sample-log" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Sample log</h4>
<div class="highlight highlight-source-json" style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">{
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>instanceId<span class="pl-pds" style="box-sizing: border-box;">"</span></span> : <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>i-xxxxxxxx<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>pid<span class="pl-pds" style="box-sizing: border-box;">"</span></span> : <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">12354</span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>uid<span class="pl-pds" style="box-sizing: border-box;">"</span></span> : <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>gid<span class="pl-pds" style="box-sizing: border-box;">"</span></span> : <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>cmdName<span class="pl-pds" style="box-sizing: border-box;">"</span></span> : <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>cron<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>exe<span class="pl-pds" style="box-sizing: border-box;">"</span></span> : <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/usr/sbin/cron<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>cmdLine<span class="pl-pds" style="box-sizing: border-box;">"</span></span> : <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/usr/sbin/CRON -f<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>systemdUnit<span class="pl-pds" style="box-sizing: border-box;">"</span></span> : <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>cron.service<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>bootId<span class="pl-pds" style="box-sizing: border-box;">"</span></span> : <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>fa58079c7a6d12345678b6ebf1234567<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>hostname<span class="pl-pds" style="box-sizing: border-box;">"</span></span> : <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>ip-10-1-0-15<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>transport<span class="pl-pds" style="box-sizing: border-box;">"</span></span> : <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>syslog<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>priority<span class="pl-pds" style="box-sizing: border-box;">"</span></span> : <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>INFO<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>message<span class="pl-pds" style="box-sizing: border-box;">"</span></span> : <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>pam_unix(cron:session): session opened for user root by (uid=0)<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>syslogFacility<span class="pl-pds" style="box-sizing: border-box;">"</span></span> : <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">10</span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>syslogIdent<span class="pl-pds" style="box-sizing: border-box;">"</span></span> : <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>CRON<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
}</pre>
</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
The JSON-formatted log events could also be exported into an AWS ElasticSearch instance using the <span style="box-sizing: border-box; font-weight: 600;"><em style="box-sizing: border-box;">CloudWatch</em></span>sync mechanism. Once in ElasticSearch, you can use an ELK stack to obtain more elaborate filtering and query capabilities.</div>
<h2 style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#installation" id="user-content-installation" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Installation</h2>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
If you have a binary distribution, you just need to drop the executable file somewhere.</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
This tool assumes that it is running on an EC2 instance.</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
This tool uses <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">libsystemd</code> to access the journal. systemd-based distributions generally ship with this already installed, but if yours doesn't you must manually install the library somehow before this tool will work.</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
There are instructions on how to install the Linux requirements for development below see - <a href="https://github.com/cloudurable/systemd-cloud-watch#setting-up-a-linux-env-for-testingdeveloping-centos7" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">Setting up a Linux env for testing/developing (CentOS7)</a>.</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
We also have two excellent examples of setting up a dev environment using <a href="https://www.packer.io/" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">bin.packer</a> for both <a href="https://github.com/cloudurable/systemd-cloud-watch#building-the-ec2-image-with-packer-to-build-the-linux-instance-to-build-this-project" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">AWS EC2</a> and<a href="https://github.com/cloudurable/systemd-cloud-watch#building-the-docker-image-to-build-the-linux-instance-to-build-this-project" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">Docker</a>. We setup CentoOS 7. The EC2 instance bin.packer build uses the <span style="box-sizing: border-box; font-weight: 600;"><em style="box-sizing: border-box;">aws command line</em></span> to create and connect to a running image. These should be instructive for how to setup this utility in your environment to run with <span style="box-sizing: border-box; font-weight: 600;"><em style="box-sizing: border-box;">systemd</em></span>as we provide all of the systemd scripts in the bin.packer provision scripts for EC2. An example is good. A running example is better.</div>
<h2 style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#configuration" id="user-content-configuration" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Configuration</h2>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
This tool uses a small configuration file to set some values that are required for its operation. Most of the configuration values are optional and have default settings, but a couple are required.</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
The configuration file uses a syntax like this:</div>
<div class="highlight highlight-source-js" style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">log_group <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>my-awesome-app<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
</pre>
</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
The following configuration settings are supported:</div>
<ul style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">aws_region</code>: (Optional) The AWS region whose CloudWatch Logs API will be written to. If not provided, this defaults to the region where the host EC2 instance is running.</div>
</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ec2_instance_id</code>: (Optional) The id of the EC2 instance on which the tool is running. There is very little reason to set this, since it will be automatically set to the id of the host EC2 instance.</div>
</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">journal_dir</code>: (Optional) Override the directory where the systemd journal can be found. This is useful in conjunction with remote log aggregation, to work with journals synced from other systems. The default is to use the local system's journal.</div>
</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">log_group</code>: (Required) The name of the cloudwatch log group to write logs into. This log group must be created before running the program.</div>
</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">log_priority</code>: (Optional) The highest priority of the log messages to read (on a 0-7 scale). This defaults to DEBUG (all messages). This has a behaviour similar to <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">journalctl -p <priority></code>. At the moment, only a single value can be specified, not a range. Possible values are: <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">0,1,2,3,4,5,6,7</code> or one of the corresponding <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">"emerg", "alert", "crit", "err", "warning", "notice", "info", "debug"</code>. When a single log level is specified, all messages with this log level or a lower (hence more important) log level are read and pushed to CloudWatch. For more information about priority levels, look at<a href="https://www.freedesktop.org/software/systemd/man/journalctl.html" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">https://www.freedesktop.org/software/systemd/man/journalctl.html</a></div>
</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">log_stream</code>: (Optional) The name of the cloudwatch log stream to write logs into. This defaults to the EC2 instance id. Each running instance of this application (along with any other applications writing logs into the same log group) must have a unique <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">log_stream</code> value. If the given log stream doesn't exist then it will be created before writing the first set of journal events.</div>
</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">buffer_size</code>: (Optional) The size of the event buffer to send to CloudWatch Logs API. The default is 50. This means that cloud watch will send 50 logs at a time.</div>
</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">fields</code>: (Optional) Specifies which fields should be included in the JSON map that is sent to CloudWatch.</div>
</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">omit_fields</code>: (Optional) Specifies which fields should NOT be included in the JSON map that is sent to CloudWatch.</div>
</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">field_length</code>: (Optional) Specifies how long string fileds can be in the JSON map that is sent to CloudWatch. The default is 255 characters.</div>
</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">queue_batch_size</code> : (Optional) Internal. Default to 10,000 entries, how large the queue buffer is. This is chunks of log entries that can be sent to the cloud watch repeater.</div>
</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">queue_channel_size</code>: (Optional) Internal. Default to 3 entries, how large the queue buffer is. This is how many <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">queue_batch_size</code> can be around to send before the journald reader waits for the cloudwatch repeater.</div>
</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">queue_poll_duration_ms</code> : (Optional) Internal. Default to 10 ms, how long the queue manager will wait if there are no log entries to send to check again to see if there are log entries to send.</div>
</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">queue_flush_log_ms</code> : (Optional) If <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">queue_batch_size</code> has not been met because there are no more journald entries to read, how long to flush the buffer to cloud watch receiver. Defaults to 100 ms.</div>
</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">debug</code>: (Optional) Turns on debug logging.</div>
</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">local</code>: (Optional) Used for unit testing. Will not try to create an AWS meta-data client to read region and AWS credentials.</div>
</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">tail</code>: (Optional) Start from the tail of log. Only send new log entries. This is good for reboot so you don't send all of the logs in the system, which is the default behavior.</div>
</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">rewind</code>: (Optional) Used to rewind X number of entries from the tail of the log. Must be used in conjunction with the <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">tail</code> setting.</div>
</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;">
<code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">mock-cloud-watch</code> : (Optional) Used to send logs to a Journal Repeater that just spits out message and priority to the console. This is used for development only.</div>
</li>
</ul>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
If your average log message was 500 bytes, and your used the default setting then assuming the server was generating journald messages rapidly you could use a heap of up to <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">queue_channel_size</code> (3) * <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">queue_batch_size</code>(10,000) * 500 bytes (15,000,000). If you had a very resource constrained env, reduce the <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">queue_batch_size</code> and/or the <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">queue_channel_size</code>.</div>
<h3 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#aws-api-access" id="user-content-aws-api-access" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>AWS API access</h3>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
This program requires access to call some of the Cloudwatch API functions. The recommended way to achieve this is to create an <a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">IAM Instance Profile</a> that grants your EC2 instance a role that has Cloudwatch API access. The program will automatically discover and make use of instance profile credentials.</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
The following IAM policy grants the required access across all log groups in all regions:</div>
<h4 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#iam-file" id="user-content-iam-file" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>IAM file</h4>
<div class="highlight highlight-source-json" style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">{
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Version<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>2012-10-17<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Statement<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: [
{
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Effect<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Allow<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Action<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: [
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>logs:CreateLogStream<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>logs:PutLogEvents<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>logs:DescribeLogStreams<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
],
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Resource<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: [
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>arn:aws:logs:*:*:log-group:*<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>arn:aws:logs:*:*:log-group:*:log-stream:*<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
]
}
]
}</pre>
</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
In more complex environments you may want to restrict further which regions, groups and streams the instance can write to. You can do this by adjusting the two ARN strings in the <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">"Resource"</code> section:</div>
<ul style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">The first <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">*</code> in each string can be replaced with an AWS region name like <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">us-east-1</code> to grant access only within the given region.</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">The <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">*</code> after <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">log-group</code> in each string can be replaced with a Cloudwatch Logs log group name to grant access only to the named group.</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">The <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">*</code> after <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">log-stream</code> in the second string can be replaced with a Cloudwatch Logs log stream name to grant access only to the named stream.</li>
</ul>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
Other combinations are possible too. For more information, see <a href="http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#arn-syntax-cloudwatch-logs" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">the reference on ARNs and namespaces</a>.</div>
<h3 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#coexisting-with-the-official-cloudwatch-logs-agent" id="user-content-coexisting-with-the-official-cloudwatch-logs-agent" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Coexisting with the official Cloudwatch Logs agent</h3>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
This application can run on the same host as the official Cloudwatch Logs agent but care must be taken to ensure that they each use a different log stream name. Only one process may write into each log stream.</div>
<h2 style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#running-on-system-boot" id="user-content-running-on-system-boot" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Running on System Boot</h2>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
This program is best used as a persistent service that starts on boot and keeps running until the system is shut down. If you're using <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">journald</code> then you're presumably using systemd; you can create a systemd unit for this service. For example:</div>
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; word-wrap: normal;"><code style="background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow: visible; padding: 0px; word-break: normal; word-wrap: normal;">[Unit]
Description=journald-cloudwatch-logs
Wants=basic.target
After=basic.target network.target
[Service]
User=nobody
Group=nobody
ExecStart=/usr/local/bin/journald-cloudwatch-logs /usr/local/etc/journald-cloudwatch-logs.conf
KillMode=process
Restart=on-failure
RestartSec=42s
</code></pre>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
This program is designed under the assumption that it will run constantly from some point during system boot until the system shuts down.</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
If the service is stopped while the system is running and then later started again, it will "lose" any journal entries that were written while it wasn't running. However, on the initial run after each boot it will clear the backlog of logs created during the boot process, so it is not necessary to run the program particularly early in the boot process unless you wish to <em style="box-sizing: border-box;">promptly</em> capture startup messages.</div>
<h2 style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#building" id="user-content-building" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Building</h2>
<h4 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#test-cloud-watch-package" id="user-content-test-cloud-watch-package" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Test cloud-watch package</h4>
<div class="highlight highlight-source-shell" style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">go <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">test</span> -v github.com/advantageous/systemd-cloud-watch/cloud-watch</pre>
</div>
<h4 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#build-and-test-on-linux-centos7" id="user-content-build-and-test-on-linux-centos7" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Build and Test on Linux (Centos7)</h4>
<div class="highlight highlight-source-shell" style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> ./run_build_linux.sh</pre>
</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
The above starts up a docker container, runs <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">go get</code>, <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">go build</code>, <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">go test</code> and then copies the binary to <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">systemd-cloud-watch_linux</code>.</div>
<h4 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#debug-process-running-linux" id="user-content-debug-process-running-linux" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Debug process running Linux</h4>
<div class="highlight highlight-source-shell" style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> ./run_test_container.sh</pre>
</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
The above starts up a docker container that you can develop with that has all the prerequisites needed to compile and test this project.</div>
<h4 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#sample-debug-session" id="user-content-sample-debug-session" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Sample debug session</h4>
<div class="highlight highlight-source-shell" style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">$ ./run_test_container.sh
latest: Pulling from advantageous/golang-cloud-watch
Digest: sha256:eaf5c0a387aee8cc2d690e1c5e18763e12beb7940ca0960ce1b9742229413e71
Status: Image is up to date <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">for</span> advantageous/golang-cloud-watch:latest
[root@6e0d1f984c03 /]<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> cd gopath/src/github.com/advantageous/systemd-cloud-watch/</span>
.git/ README.md cloud-watch/ bin.packer/ sample.conf
.gitignore build_linux.sh main.go run_build_linux.sh systemd-cloud-watch.iml
.idea/ cgroup/ output.json run_test_container.sh systemd-cloud-watch_linux
[root@6e0d1f984c03 /]<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> cd gopath/src/github.com/advantageous/systemd-cloud-watch/</span>
[root@6e0d1f984c03 systemd-cloud-watch]<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> ls</span>
README.md build_linux.sh cgroup cloud-watch main.go output.json bin.packer run_build_linux.sh
run_test_container.sh sample.conf systemd-cloud-watch.iml systemd-cloud-watch_linux
[root@6e0d1f984c03 systemd-cloud-watch]<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> source ~/.bash_profile</span>
[root@6e0d1f984c03 systemd-cloud-watch]<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> export GOPATH=/gopath</span>
[root@6e0d1f984c03 systemd-cloud-watch]<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> /usr/lib/systemd/systemd-journald &</span>
[1] 24
[root@6e0d1f984c03 systemd-cloud-watch]<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> systemd-cat echo "RUNNING JAVA BATCH JOB - ADF BATCH from `pwd`"</span>
[root@6e0d1f984c03 systemd-cloud-watch]<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> echo "Running go clean"</span>
Running go clean
[root@6e0d1f984c03 systemd-cloud-watch]<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> go clean</span>
[root@6e0d1f984c03 systemd-cloud-watch]<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> echo "Running go get"</span>
Running go get
[root@6e0d1f984c03 systemd-cloud-watch]<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> go get</span>
[root@6e0d1f984c03 systemd-cloud-watch]<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> echo "Running go build"</span>
Running go build
[root@6e0d1f984c03 systemd-cloud-watch]<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> go build</span>
[root@6e0d1f984c03 systemd-cloud-watch]<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> echo "Running go test"</span>
Running go <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">test</span>
[root@6e0d1f984c03 systemd-cloud-watch]<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> go test -v github.com/advantageous/systemd-cloud-watch/cloud-watch</span>
=== RUN TestRepeater
config DEBUG: 2016/11/30 08:53:34 config.go:66: Loading log...
aws INFO: 2016/11/30 08:53:34 aws.go:42: Config <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">set</span> to <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">local</span>
aws INFO: 2016/11/30 08:53:34 aws.go:72: Client missing credentials not looked up
aws INFO: 2016/11/30 08:53:34 aws.go:50: Client missing using config to <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">set</span> region
aws INFO: 2016/11/30 08:53:34 aws.go:52: AWSRegion missing using default region us-west-2
repeater ERROR: 2016/11/30 08:53:44 cloudwatch_journal_repeater.go:141: Error from putEvents NoCredentialProviders: no valid providers <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">in</span> chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
--- SKIP: TestRepeater (10.01s)
cloudwatch_journal_repeater_test.go:43: Skipping WriteBatch, you need to setup AWS credentials <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">for</span> this to work
=== RUN TestConfig
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">test</span> DEBUG: 2016/11/30 08:53:44 config.go:66: Loading log...
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">test</span> INFO: 2016/11/30 08:53:44 config_test.go:33: [Foo Bar]
--- PASS: TestConfig (0.00s)
=== RUN TestLogOmitField
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">test</span> DEBUG: 2016/11/30 08:53:44 config.go:66: Loading log...
--- PASS: TestLogOmitField (0.00s)
=== RUN TestNewJournal
--- PASS: TestNewJournal (0.00s)
=== RUN TestSdJournal_Operations
--- PASS: TestSdJournal_Operations (0.00s)
journal_linux_test.go:41: Read value=Runtime journal is using 8.0M (max allowed 4.0G, trying to leave 4.0G free of 55.1G available → current limit 4.0G).
=== RUN TestNewRecord
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">test</span> DEBUG: 2016/11/30 08:53:44 config.go:66: Loading log...
--- PASS: TestNewRecord (0.00s)
=== RUN TestLimitFields
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">test</span> DEBUG: 2016/11/30 08:53:44 config.go:66: Loading log...
--- PASS: TestLimitFields (0.00s)
=== RUN TestOmitFields
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">test</span> DEBUG: 2016/11/30 08:53:44 config.go:66: Loading log...
--- PASS: TestOmitFields (0.00s)
PASS
ok github.com/advantageous/systemd-cloud-watch/cloud-watch 10.017s</pre>
</div>
<h4 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#building-the-docker-image-to-build-the-linux-instance-to-build-this-project" id="user-content-building-the-docker-image-to-build-the-linux-instance-to-build-this-project" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Building the docker image to build the linux instance to build this project</h4>
<div class="highlight highlight-source-shell" style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> from project root</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">cd</span> bin.packer
bin.packer build packer_docker.json</pre>
</div>
<h4 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#to-run-docker-dev-image" id="user-content-to-run-docker-dev-image" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>To run docker dev image</h4>
<div class="highlight highlight-source-shell" style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> from project root</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">cd</span> bin.packer
./run.sh
</pre>
</div>
<h4 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#building-the-ec2-image-with-binpacker-to-build-the-linux-instance-to-build-this-project" id="user-content-building-the-ec2-image-with-binpacker-to-build-the-linux-instance-to-build-this-project" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Building the ec2 image with bin.packer to build the linux instance to build this project</h4>
<div class="highlight highlight-source-shell" style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> from project root</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">cd</span> bin.packer
bin.packer build packer_ec2.json</pre>
</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
We use the <a href="https://www.packer.io/docs/builders/docker.html" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">docker</a> support for <a href="https://www.packer.io/" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">bin.packer</a>. ("Packer is a tool for creating machine and container images for multiple platforms from a single source configuration.")</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
Use <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ec2_env.sh_example</code> to create a <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ec2_env.sh</code> with the instance id that was just created.</div>
<h4 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#ec2_envsh_example" id="user-content-ec2_envsh_example" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>ec2_env.sh_example</h4>
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; word-wrap: normal;"><code style="background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow: visible; padding: 0px; word-break: normal; word-wrap: normal;">#!/usr/bin/env bash
export ami=ami-YOURAMI
export subnet=subnet-YOURSUBNET
export security_group=sg-YOURSG
export iam_profile=YOUR_IAM_ROLE
export key_name=MY_PEM_FILE_KEY_NAME
</code></pre>
<h5 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 0.875em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#using-ec2-image-assumes-you-have-ssh-config-setup" id="user-content-using-ec2-image-assumes-you-have-ssh-config-setup" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Using EC2 image (assumes you have ~/.ssh config setup)</h5>
<div class="highlight highlight-source-shell" style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> from project root</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">cd</span> bin.packer
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> Run and log into dev env running in EC2</span>
./runEc2Dev.sh
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> Log into running server</span>
./loginIntoEc2Dev.sh
</pre>
</div>
<h2 style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#setting-up-a-linux-env-for-testingdeveloping-centos7" id="user-content-setting-up-a-linux-env-for-testingdeveloping-centos7" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Setting up a Linux env for testing/developing (CentOS7).</h2>
<div class="highlight highlight-source-shell" style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">yum -y install wget
yum install -y git
yum install -y gcc
yum install -y systemd-devel
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">echo</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>installing go<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">cd</span> /tmp
wget https://storage.googleapis.com/golang/go1.7.3.linux-amd64.tar.gz
tar -C /usr/local/ -xzf go1.7.3.linux-amd64.tar.gz
rm go1.7.3.linux-amd64.tar.gz
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">echo</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>export PATH=$PATH:/usr/local/go/bin<span class="pl-pds" style="box-sizing: border-box;">'</span></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">>></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">~</span>/.bash_profile</pre>
</div>
<h2 style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#setting-up-java-to-write-to-systemd-journal" id="user-content-setting-up-java-to-write-to-systemd-journal" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Setting up Java to write to systemd journal</h2>
<h4 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#gradle-build" id="user-content-gradle-build" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>gradle build</h4>
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; word-wrap: normal;"><code style="background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow: visible; padding: 0px; word-break: normal; word-wrap: normal;">compile 'org.gnieh:logback-journal:0.2.0'
</code></pre>
<h4 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#logbackxml" id="user-content-logbackxml" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>logback.xml</h4>
<div class="highlight highlight-text-xml" style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><?<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">xml</span><span class="pl-e" style="box-sizing: border-box; color: #795da3;"> version</span>=<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>1.0<span class="pl-pds" style="box-sizing: border-box;">"</span></span><span class="pl-e" style="box-sizing: border-box; color: #795da3;"> encoding</span>=<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>UTF-8<span class="pl-pds" style="box-sizing: border-box;">"</span></span>?>
<<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">configuration</span>>
<<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">appender</span> <span class="pl-e" style="box-sizing: border-box; color: #795da3;">name</span>=<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>journal<span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-e" style="box-sizing: border-box; color: #795da3;">class</span>=<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>org.gnieh.logback.SystemdJournalAppender<span class="pl-pds" style="box-sizing: border-box;">"</span></span> />
<<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">root</span> <span class="pl-e" style="box-sizing: border-box; color: #795da3;">level</span>=<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>INFO<span class="pl-pds" style="box-sizing: border-box;">"</span></span>>
<<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">appender-ref</span> <span class="pl-e" style="box-sizing: border-box; color: #795da3;">ref</span>=<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>journal<span class="pl-pds" style="box-sizing: border-box;">"</span></span> />
<<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">customFields</span>>{"serviceName":"adfCalcBatch","serviceHost":"${HOST}"}</<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">customFields</span>>
</<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">root</span>>
<<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">logger</span> <span class="pl-e" style="box-sizing: border-box; color: #795da3;">name</span>=<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>com.mycompany<span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-e" style="box-sizing: border-box; color: #795da3;">level</span>=<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>INFO<span class="pl-pds" style="box-sizing: border-box;">"</span></span>/>
</<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">configuration</span>></pre>
</div>
<h2 style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#commands-for-controlling-systemd-service-ec2-dev-env" id="user-content-commands-for-controlling-systemd-service-ec2-dev-env" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Commands for controlling systemd service EC2 dev env</h2>
<div class="highlight highlight-source-shell" style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> Get status</span>
sudo systemctl status journald-cloudwatch
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> Stop Service</span>
sudo systemctl stop journald-cloudwatch
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> Find the service</span>
ps -ef <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">|</span> grep cloud
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">#</span> Run service manually</span>
/usr/bin/systemd-cloud-watch_linux /etc/journald-cloudwatch.conf
</pre>
</div>
<h2 style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#derived" id="user-content-derived" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Derived</h2>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
This is based on <a href="https://github.com/advantageous/journald-cloudwatch-logs" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">advantageous journald-cloudwatch-logs</a> which was forked from <a href="https://github.com/saymedia/journald-cloudwatch-logs" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">saymedia journald-cloudwatch-logs</a>.</div>
<h2 style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#status" id="user-content-status" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Status</h2>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
Done and released.</div>
<h3 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#using-as-a-lib" id="user-content-using-as-a-lib" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Using as a lib.</h3>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
You can use this project as a lib and you can pass in your own <em style="box-sizing: border-box;">JournalRepeater</em> and your own <em style="box-sizing: border-box;">Journal</em>.</div>
<h4 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#interface-for-journalrepeater" id="user-content-interface-for-journalrepeater" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Interface for JournalRepeater</h4>
<div class="highlight highlight-source-go" style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> cloud_watch
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">type</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">Record</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">struct</span> {...} <span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span>see source code</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">type</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">JournalRepeater</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">interface</span> {
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> Close closes a journal opened with NewJournal.</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">Close</span>() <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">error</span>;
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">WriteBatch</span>(records []<span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">Record</span>) <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">error</span>;
}</pre>
</div>
<h4 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#interface-for-journal" id="user-content-interface-for-journal" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Interface for Journal</h4>
<div class="highlight highlight-source-go" style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">type</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">Journal</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">interface</span> {
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> Close closes a journal opened with NewJournal.</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">Close</span>() <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">error</span>;
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> Next advances the read pointer into the journal by one entry.</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">Next</span>() (<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">uint64</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">error</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> NextSkip advances the read pointer by multiple entries at once,</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> as specified by the skip parameter.</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">NextSkip</span>(skip <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">uint64</span>) (<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">uint64</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">error</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> Previous sets the read pointer into the journal back by one entry.</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">Previous</span>() (<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">uint64</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">error</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> PreviousSkip sets back the read pointer by multiple entries at once,</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> as specified by the skip parameter.</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">PreviousSkip</span>(skip <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">uint64</span>) (<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">uint64</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">error</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> GetDataValue gets the data object associated with a specific field from the</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> current journal entry, returning only the value of the object.</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">GetDataValue</span>(field <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">string</span>) (<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">string</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">error</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> GetRealtimeUsec gets the realtime (wallclock) timestamp of the current</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> journal entry.</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">GetRealtimeUsec</span>() (<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">uint64</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">error</span>);
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">AddLogFilters</span>(config *Config)
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> GetMonotonicUsec gets the monotonic timestamp of the current journal entry.</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">GetMonotonicUsec</span>() (<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">uint64</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">error</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> GetCursor gets the cursor of the current journal entry.</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">GetCursor</span>() (<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">string</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">error</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> SeekHead seeks to the beginning of the journal, i.e. the oldest available</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> entry.</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">SeekHead</span>() <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">error</span>;
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> SeekTail may be used to seek to the end of the journal, i.e. the most recent</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> available entry.</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">SeekTail</span>() <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">error</span>;
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> SeekCursor seeks to a concrete journal cursor.</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">SeekCursor</span>(cursor <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">string</span>) <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">error</span>;
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> Wait will synchronously wait until the journal gets changed. The maximum time</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> this call sleeps may be controlled with the timeout parameter. If</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> sdjournal.IndefiniteWait is passed as the timeout parameter, Wait will</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span> wait indefinitely for a journal change.</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">Wait</span>(timeout time.<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Duration</span>) <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">int</span>;
}
</pre>
</div>
<h4 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/cloudurable/systemd-cloud-watch#using-as-a-lib-1" id="user-content-using-as-a-lib-1" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Using as a lib</h4>
<div class="highlight highlight-source-go" style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> main
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> (
jcw <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>github.com/advantageous/systemd-cloud-watch/cloud-watch<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>flag<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>os<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">var</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">help</span> = flag.<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">Bool</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>help<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">false</span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>set to true to show this help<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">func</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">main</span>() {
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">logger</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:=</span> jcw.<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">NewSimpleLogger</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>main<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">nil</span>)
flag.<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">Parse</span>()
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">if</span> *help {
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">usage</span>(logger)
os.<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">Exit</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>)
}
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">configFilename</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:=</span> flag.<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">Arg</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">if</span> configFilename == <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span><span class="pl-pds" style="box-sizing: border-box;">"</span></span> {
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">usage</span>(logger)
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">panic</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>config file name must be set!<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
}
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">config</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:=</span> jcw.<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">CreateConfig</span>(configFilename, logger)
logger = jcw.<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">NewSimpleLogger</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>main<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, config)
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">journal</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:=</span> jcw.<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">CreateJournal</span>(config, logger) <span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span>Instead of this, load your own journal</span>
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">repeater</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:=</span> jcw.<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">CreateRepeater</span>(config, logger) <span class="pl-c" style="box-sizing: border-box; color: #969896;"><span class="pl-c" style="box-sizing: border-box;">//</span>Instead of this, load your own repeater</span>
jcw.<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">RunWorkers</span>(journal, repeater, logger, config )
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">func</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">usage</span>(<span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">logger</span> *<span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">jcw</span>.<span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">Logger</span>) {
logger.<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Error</span>.<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">Println</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Usage: systemd-cloud-watch <config-file><span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
flag.<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">PrintDefaults</span>()
}
</pre>
</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
You could for example create a <em style="box-sizing: border-box;">JournalRepeater</em> that writes to <em style="box-sizing: border-box;">InfluxDB</em> instead of <em style="box-sizing: border-box;">CloudWatch</em>.</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
Improvements:</div>
<ul style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">Added unit tests (there were none).</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Heavily reduced locking by using <a href="https://github.com/advantageous/go-qbit" style="box-sizing: border-box; color: #0366d6; text-decoration: none;">qbit</a> instead of original implementation.</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Added cross compile so I can develop/test on my laptop (MacOS).</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Made logging stateless. No more need for a state file.</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">No more getting out of sync with CloudWatch.</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Detects being out of sync and recovers.</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Fixed error with log messages being too big.</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Added ability to include or omit logging fields.</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Created docker image and scripts to test on Linux (CentOS7).</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Created EC2 image and scripts to test on Linux running in AWS EC2 (CentOS7).</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Code organization (we use a package).</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Added comprehensive logging which includes debug logging by config.</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Uses actual timestamp from journal log record instead of just current time</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Auto-creates CloudWatch log group if it does not exist</li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Allow this to be used as a library by providing interface for Journal and JournalWriter.</li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<br /></div>
<div>
<br /></div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com0tag:blogger.com,1999:blog-6128525281101058933.post-11770312806064305092017-03-04T13:54:00.002-08:002017-03-04T13:54:53.385-08:00MetricsD to send Linux OS metrics to Amazon CloudWatch<h3 id="metricsd-to-send-os-metrics-to-aws" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
metricsd to send Linux OS metrics to AWS</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We are using <a href="https://github.com/cloudurable/metricsd" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">metricsd to read OS metrics and send data to AWS CloudWatch Metrics</a>. Metricsd gathers OS KPIs for <a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CW_Support_For_AWS.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">AWS CloudWatch Metrics</a>. We install this as a systemd process which depends on cassandra. We also install the Cassandra Database as a <a href="http://cloudurable.com/blog/systemd-dependencies/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">systemd process</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We use systemd unit quite a bit. We use systemd to start up Cassandra config scripts. We use systemd to start up Cassandra/Kafka, and to shut Cassandra/Kakfa (this article does not cover Kafka at all) down nicely. Since systemd is pervasive in all new mainstream Linux distributions, you can see that systemd is an important concept for DevOps.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Metricsd gets installed as a systemd service by our provisioning scripts.</div>
<h4 id="installing-metricsd-systemd-from-our-provisioning-scripts" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Installing metricsd systemd from our provisioning scripts</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">cp ~/resources/etc/systemd/system/metricsd.service /etc/systemd/system/metricsd.service
cp ~/resources/etc/metricsd.conf /etc/metricsd.conf
systemctl enable metricsd
systemctl start metricsd
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">systemctl enable</code> to install <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">metricsd</code> to start up on system start. We then use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">systemctl start</code> to start <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">metricsd</code>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We could write a whole article on <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">metricsd</code> and <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">AWS CloudWatch metrics</em></span>, and perhaps we will. For more informatino about metricsd please see the <a href="https://github.com/cloudurable/metricsd" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">metricsd github project</a>.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
The <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">metricsd</code> system unit depends on the <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">Cassandra</code> service. The unit file is as follows.</div>
<h4 id="etc-systemd-system-metricsd-service" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
/etc/systemd/system/metricsd.service</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">[Unit]
Description=MetricsD OS Metrics
Requires=cassandra.service
After=cassandra.service
[Service]
ExecStart=/opt/cloudurable/bin/metricsd
WorkingDirectory=/opt/cloudurable
Restart=always
RestartSec=60
TimeoutStopSec=60
TimeoutStartSec=60
[Install]
WantedBy=multi-user.target
</code></pre>
<div>
<code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; padding: 0px;"><h3 id="retrospective-past-articles-in-this-cassandra-cluster-devops-dba-series" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px; white-space: normal;">
Retrospective - Past Articles in this Cassandra Cluster DevOps/DBA series</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px; white-space: normal;">
The first article in this series was about setting up a Cassandra <a href="http://cloudurable.com/blog/cassandra-image-vagrant-cluster-example/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">cluster with Vagrant</a> (also appeared on DZone with some additional content <a href="https://dzone.com/articles/setting-up-a-cassandra-cluster-with-vagrant" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">DZone Setting up a Cassandra Cluster with Vagrant</a>. The second article in this series was about <a href="http://cloudurable.com/blog/cassandra-image-vagrant-cluster-example/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">setting up SSL for a Cassandra cluster using Vagrant</a> (which also appeared with more content as <a href="https://dzone.com/articles/setting-up-a-cassandra-cluster-with-ssl" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">DZone Setting up a Cassandra Cluster with SSL</a>). The <a href="http://cloudurable.com/blog/ansible-cassandra-vagrant-devops/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">third article</a> in this series was about configuring and using Ansible (building on the first two articles). This article (the 4th) will cover applying the tools and techniques from the first three articles to produce an image (EC2 AMI to be precise) that we can deploy to AWS/EC2. To do this explanation, we will use Packer, Ansible, and the Aws Command Line tools. The AWS command line tools are essential for doing DevOps with AWS.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px; white-space: normal;">
<br /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px; white-space: normal;">
Check out more information about the Cassandra Database</div>
<ul style="background-color: white; color: #666666; font-family: "Trebuchet MS", Trebuchet, Verdana, sans-serif; font-size: 12px; line-height: 1.2; margin: 0px; padding: 0px 0px 0px 1.25em;">
<li style="margin: 0px; padding: 0.25em 0px;"><a href="http://cloudurable.com/service-architecture-analysis-cassandra-or-kafka-aws-ec2/index.html" style="color: #888888; text-decoration: none;">Cassandra Consulting: Architecture Analysis</a></li>
<li style="margin: 0px; padding: 0.25em 0px;"><a href="http://cloudurable.com/service-quick-start-mentoring-cassandra-or-kafka-aws-ec2/index.html" style="color: #888888; text-decoration: none;">Cassandra Consulting: Quick Start</a></li>
<li style="margin: 0px; padding: 0.25em 0px;"><a href="http://cloudurable.com/cassandra-course/index.html" style="color: #888888; text-decoration: none;">Cassandra Course</a></li>
<li style="margin: 0px; padding: 0.25em 0px;"><a href="http://cloudurable.com/" style="color: #888888; text-decoration: none;">Amazon Cassandra Support</a></li>
</ul>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px; white-space: normal;">
<br /></div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px; white-space: normal;">
Check out the metricsd github project page. </div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px; white-space: normal;">
<br /></div>
<div style="background-color: white; box-sizing: border-box; margin-bottom: 10px;">
<span style="color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif;"><span style="font-size: 14px;"><a href="https://github.com/advantageous/metricsd">https://github.com/advantageous/metricsd</a></span></span></div>
<div style="background-color: white; box-sizing: border-box; margin-bottom: 10px;">
<span style="color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif;"><span style="font-size: 14px;"><br /></span></span></div>
<h2 style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; line-height: 1.25; margin-bottom: 16px; margin-top: 0px !important; padding-bottom: 0.3em;">
Metricsd</h2>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
Reads OS metrics and sends data to AWS CloudWatch Metrics.</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
Metricsd gathers OS metrics for AWS CloudWatch. You can install it as a systemd process.</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
Configuration</div>
<h4 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/metricsd#etcmetricsdconf" id="user-content-etcmetricsdconf" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>/etc/metricsd.conf</h4>
<pre lang="conf" style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; word-wrap: normal;"><code style="background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow: visible; padding: 0px; word-break: normal; word-wrap: normal;">
# AWS Region string `hcl:"aws_region"`
# If not set, uses aws current region for this instance.
# Used for testing only.
# aws_region = "us-west-1"
# EC2InstanceId string `hcl:"ec2_instance_id"`
# If not set, uses aws instance id for this instance
# Used for testing only.
# ec2_instance_id = "i-my-fake-instanceid"
# Debug bool `hcl:"debug"`
# Used for testing and debugging
debug = false
# Local bool `hcl:"local"`
# Used to ingore local ec2 meta-data, used for development only.
# local = true
# TimePeriodSeconds time.Duration `hcl:"interval_seconds"`
# Defaults to 30 seconds, how often metrics are collected.
interval_seconds = 10
# Used to specify the environment: prod, dev, qa, staging, etc.
# This gets used as a dimension that is sent to cloudwatch.
env="dev"
# Used to specify the top level namespace in cloudwatch.
namespace="Cassandra Cluster"
# Used to specify the role of the AMI instance.
# Gets used as a dimension.
# e.g., dcos-master, consul-master, dcos-agent, cassandra-node, etc.
server_role="dcos-master"
</code></pre>
<h2 style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/metricsd#installing-as-a-service" id="user-content-installing-as-a-service" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Installing as a service</h2>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
If you are using systemd you should install this as a service.</div>
<h4 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/metricsd#etcsystemdsystemmetricsdservice" id="user-content-etcsystemdsystemmetricsdservice" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>/etc/systemd/system/metricsd.service</h4>
<pre lang="conf" style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; word-wrap: normal;"><code style="background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow: visible; padding: 0px; word-break: normal; word-wrap: normal;">
[Unit]
Description=metricsd
Wants=basic.target
After=basic.target network.target
[Service]
User=centos
Group=centos
ExecStart=/usr/bin/metricsd
KillMode=process
Restart=on-failure
RestartSec=42s
[Install]
WantedBy=multi-user.target
</code></pre>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
Copy the binary to <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">/usr/bin/metricsd</code>. Copy the config to <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">/etc/metricsd.conf</code>. You can specify a different conf location by using <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">/usr/bin/metricsd -conf /foo/bar/myconf.conf</code>.</div>
<h4 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/metricsd#installing" id="user-content-installing" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Installing</h4>
<div class="highlight highlight-source-shell" style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">$ sudo cp metricsd_linux /usr/bin/metricsd
$ sudo systemctl stop metricsd.service
$ sudo systemctl <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">enable</span> metricsd.service
$ sudo systemctl start metricsd.service
$ sudo systemctl status metricsd.service
● metricsd.service - metricsd
Loaded: loaded (/etc/systemd/system/metricsd.service<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">;</span> enabled<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">;</span> vendor preset: disabled)
Active: active (running) since Wed 2016-12-21 20:19:59 UTC<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">;</span> 8s ago
Main PID: 718 (metricsd)
CGroup: /system.slice/metricsd.service
└─718 /usr/bin/metricsd
Dec 21 20:19:59 ip-172-31-29-173 systemd[1]: Started metricsd.
Dec 21 20:19:59 ip-172-31-29-173 systemd[1]: Starting metricsd...
Dec 21 20:19:59 ip-172-31-29-173 metricsd[718]: INFO <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">:</span> [main] - 2016/12/21 20:19:59 config.go:29: Loading config /et....conf
Dec 21 20:19:59 ip-172-31-29-173 metricsd[718]: INFO <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">:</span> [main] - 2016/12/21 20:19:59 config.go:45: Loading log...</pre>
</div>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
There are full example packer install scripts under bin/packer/packer_ec2.json. The best doc is a working example.</div>
<h2 style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/metricsd#metrics" id="user-content-metrics" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Metrics</h2>
<h3 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/metricsd#cpu-metrics" id="user-content-cpu-metrics" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>CPU metrics</h3>
<ul style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">softIrqCnt</code> - count of soft interrupts for the last period</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">intrCnt</code> - count of interrupts for the last period</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ctxtCnt</code> - count of context switches for the last period</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">processesStrtCnt</code> - count of processes started for the last period</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">GuestJif</code> - jiffies spent in guest mode for last time period</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">UsrJif</code> - jiffies spent in usr mode for last time period</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">IdleJif</code> - jiffies spent in usr mode for last time period</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">IowaitJif</code> - jiffies spent handling IO for last time period</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">IrqJif</code> - jiffies spent handling interrupts for last time period</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">GuestniceJif</code> - guest nice mode</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">StealJif</code> - time stolen by noisy neighbors for last time period</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">SysJif</code> - jiffies spent doing OS stuff like system calls in last time period</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">SoftIrqJif</code> - jiffies spent handling soft IRQs in the last time period</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">procsRunning</code> - count of processes currently running</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">procsBlocked</code> - count of processes currently blocked (could be for IO or just waiting to get CPU time)</li>
</ul>
<h3 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/metricsd#disk-metrics" id="user-content-disk-metrics" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Disk metrics</h3>
<ul style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">dUVol<VOLUME_NAME>AvailPer</code> - percentage of disk space left (per volume)</li>
</ul>
<h3 style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/metricsd#mem-metrics" id="user-content-mem-metrics" style="box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Mem metrics</h3>
<ul style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">mFreeLvl</code> - free memory in kilobytes</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">mUsedLvl</code> - used memory in kilobytes</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">mSharedLvl</code> - shared memory in kilobytes</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">mBufLvl</code> - memory used by IO buffers in kilobytes</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">mAvailableLvl</code> - memory available in kilobytes</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">mFreePer</code> - percentage of memory free</li>
<li style="box-sizing: border-box; margin-top: 0.25em;"><code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">mUsedPer</code> - percentage of memory used</li>
</ul>
<div style="box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px;">
If swapping is enabled (which is unlikely), then you will get the above with <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">mSwpX</code> instead of <code style="background-color: rgba(27, 31, 35, 0.0470588); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">mX</code>.</div>
<div style="color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; white-space: pre-wrap;">
<br /></div>
</code></div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com0tag:blogger.com,1999:blog-6128525281101058933.post-19249302205002571292017-03-04T13:49:00.001-08:002017-03-04T14:11:21.525-08:00Packer EC2 support and Ansible for our Cassandra Database Clusters<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<a href="https://www.packer.io/" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">Packer</a> is used to generate machine and container images for multiple platforms from a single source configuration. We use Packer to create AWS EC2 AMIs (images) and Docker images. (We use Vagrant to setup dev images on Virtual Box.) </div>
<h3 id="packer-script-to-create-the-cassandra-database-ec2-image" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Packer for AWS Cassandra Database EC2/AMI</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
This code listing is our Packer provisioning script to produce an EC2 AMI which we can later use to produce and EC2 instance with Cassandra installed. This script will <i>install Cassandra</i> on the EC2 image. </div>
<h4 id="packer-ec2-json-packer-creation-script-for-ec2-cassandra-database-instance" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
packer-ec2.json - Packer creation script for EC2 Cassandra Database instance</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-javascript" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">{
"variables": {
"aws_access_key": "",
"aws_secret_key": "",
"aws_region": "us-west-2",
"aws_ami_image": "ami-d2c924b2",
"aws_instance_type": "m4.large",
"image_version" : "0.2.2"
},
"builders": [
{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "{{user `aws_region`}}",
"source_ami": "{{user `aws_ami_image`}}",
"instance_type": "{{user `aws_instance_type`}}",
"ssh_username": "centos",
"ami_name": "cloudurable-cassandra-{{user `image_version`}}",
"tags": {
"Name": "cloudurable-cassandra-{{user `image_version`}}",
"OS_Version": "LinuxCentOs7",
"Release": "7",
"Description": "CentOS 7 image for Cloudurable Cassandra image"
},
"user_data_file": "config/user-data.sh"
}
],
"provisioners": [
{
"type": "file",
"source": "scripts",
"destination": "/home/centos/"
},
{
"type": "file",
"source": "resources",
"destination": "/home/centos/"
},
{
"type": "shell",
"scripts": [
"scripts/000-ec2-provision.sh"
]
},
{
"type": "ansible",
"playbook_file": "playbooks/ssh-addkey.yml"
}
]
}
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Notice that we are using a <a href="https://www.packer.io/docs/builders/amazon-ebs.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">packer <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">amazon-ebs</code> builder</a> to build an AMI image based on our local dev boxes EC2 setup.</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Also, see that we use a series of Packer <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">provisioners</code>. The <a href="https://www.packer.io/docs/provisioners/file.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">packer <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">file</code> provisioner</a> can copy files or directories to a machine image. The <a href="https://www.packer.io/docs/provisioners/shell.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">packer <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">shell</code> provisioner</a> can run shell scripts. Lastly the <a href="https://www.packer.io/docs/provisioners/ansible-local.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">packer <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ansible</code> provisioner</a> can run ansible playbooks. We covered what <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">playbooks/ssh-addkey.yml</code> does in the previous article, but in short it sets up the keys so we use ansible with our the Cassandra Database cluster nodes.</div>
<h3 id="bash-provisioning" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Bash provisioning</h3>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
Before we started applying ansible to do provisioning, we used <span style="box-sizing: border-box; font-weight: 700;"><em style="box-sizing: border-box;">bash scripts</em></span> that get reused for packer/docker, packer/aws, and vagrant/virtual-box. The script <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">000-ec2-provision.sh</code> invokes these provisioning scripts which the first three articles covered at varying degrees (skim those articles if you are curious or the source code, but you don’t need it per se to follow). This way we can use the same provisioning scripts with AMIs, VirtualBox, and AWS EC2.</div>
<h4 id="scripts-000-ec2-provision-sh" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
scripts/000-ec2-provision.sh</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#!/bin/bash
set -e
sudo cp -r /home/centos/resources/ /root/
sudo mv /home/centos/scripts/ /root/
echo RUNNING PROVISION
sudo /root/scripts/000-provision.sh
echo Building host file
sudo /root/scripts/002-hosts.sh
echo RUNNING TUNE OS
sudo /root/scripts/010-tune-os.sh
echo RUNNING INSTALL the Cassandra Database
sudo /root/scripts/020-cassandra.sh
echo RUNNING INSTALL CASSANDRA CLOUD
sudo /root/scripts/030-cassandra-cloud.sh
echo RUNNING INSTALL CERTS
sudo /root/scripts/040-install-certs.sh
echo RUNNING SYTSTEMD SETUP
sudo /root/scripts/050-systemd-setup.sh
sudo chown -R cassandra /opt/cassandra/
sudo chown -R cassandra /etc/cassandra/
</code></pre>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
We covered what each of those provisioning scripts does in the first three articles, but for those just joining us, they install packages, programs and configure stuff.<br />
<br />
<h3 id="using-packer-to-build-our-ec2-ami" style="box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
Using Packer to build our ec2 AMI</h3>
<div style="box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; margin-bottom: 10px;">
To build the AWS AMI, we use <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">packer build</code> as follows.</div>
<h4 id="building-the-aws-ami" style="box-sizing: border-box; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.1; margin-bottom: 10px; margin-top: 10px;">
Building the AWS AMI</h4>
<pre style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; overflow: auto; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><code class="language-sh" style="background-color: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">$ packer build packer-ec2.json
</code></pre>
<div style="box-sizing: border-box; font-family: Roboto, Helvetica, Arial, sans-serif; margin-bottom: 10px;">
After the packer build completes, it will print out the name of the AMI image it created, e.g., <code style="background-color: #f9f2f4; border-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-size: 12.6px; padding: 2px 4px;">ami-6db33abc</code>.</div>
<div>
<br /></div>
</div>
<div style="background-color: white; box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 10px;">
<span style="color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px;">Cassandra Tutorial: Cassandra Cluster DevOps/DBA series</span></div>
<h3 id="retrospective-past-articles-in-this-cassandra-cluster-devops-dba-series" style="background-color: white; color: #333333; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 24px; line-height: 1.1; margin-bottom: 20px; margin-top: 20px;">
<div style="box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; font-weight: normal; margin-bottom: 10px;">
The first tutorial in this <i>Cassandra tutorial</i> series was about setting up a Cassandra <a href="http://cloudurable.com/blog/cassandra-image-vagrant-cluster-example/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">cluster with Vagrant</a> (also appeared on DZone with some additional content <a href="https://dzone.com/articles/setting-up-a-cassandra-cluster-with-vagrant" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">DZone Setting up a Cassandra Cluster with Vagrant</a>. The second article in this series was about <a href="http://cloudurable.com/blog/cassandra-image-vagrant-cluster-example/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">setting up SSL for a Cassandra cluster using Vagrant</a> (which also appeared with more content as <a href="https://dzone.com/articles/setting-up-a-cassandra-cluster-with-ssl" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">DZone Setting up a Cassandra Cluster with SSL</a>). The <a href="http://cloudurable.com/blog/ansible-cassandra-vagrant-devops/index.html" style="background-color: transparent; box-sizing: border-box; color: deepskyblue; text-decoration: none; transition: all 0.2s ease-out;">third article</a> in this series was about configuring and using Ansible (building on the first two articles). This article (the 4th) will cover applying the tools and techniques from the first three articles to produce an image (EC2 AMI to be precise) that we can deploy to AWS/EC2. To do this explanation, we will use Packer, Ansible, and the Aws Command Line tools. The AWS command line tools are essential for doing DevOps with AWS.</div>
<div style="box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; font-weight: normal; margin-bottom: 10px;">
<br /></div>
<div style="box-sizing: border-box; color: #555555; font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 14px; font-weight: normal; margin-bottom: 10px;">
Check out more information about the Cassandra Database</div>
<ul style="color: #666666; font-family: "Trebuchet MS", Trebuchet, Verdana, sans-serif; font-size: 13.2px; font-weight: normal; line-height: 1.4; margin: 0.5em 0px; padding: 0px 2.5em;">
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/service-architecture-analysis-cassandra-or-kafka-aws-ec2/index.html" style="color: #888888; text-decoration: none;">Cassandra Consulting: Architecture Analysis</a></li>
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/service-quick-start-mentoring-cassandra-or-kafka-aws-ec2/index.html" style="color: #888888; text-decoration: none;">Cassandra Consulting: Quick Start</a></li>
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/cassandra-course/index.html" style="color: #888888; text-decoration: none;">Cassandra Course</a></li>
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://cloudurable.com/" style="color: #888888; text-decoration: none;">Amazon Cassandra Support</a></li>
</ul>
</h3>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com0tag:blogger.com,1999:blog-6128525281101058933.post-33917171185527329532016-09-14T09:42:00.002-07:002016-09-14T09:42:18.686-07:00Using ansible to install Oracle Java on an ec2 box running in the cloud<div style="box-sizing: border-box; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 24px; margin-bottom: 16px;">
I am writing this down so I don't forget. I started this task but had to stop a few times, and then remember where I left off and at some point, others will need to know how to get started.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://gist.github.com/RichardHightower/c37036dc5a55b302d5d99e2ffc1bcefd#install-ansible" id="user-content-install-ansible" style="box-sizing: border-box; color: #4078c0; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Install ansible</h4>
<div class="highlight highlight-source-shell" style="box-sizing: border-box; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 24px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">brew install ansible</pre>
</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://gist.github.com/RichardHightower/c37036dc5a55b302d5d99e2ffc1bcefd#install-amazon-ec2-ansible-integration-tool" id="user-content-install-amazon-ec2-ansible-integration-tool" style="box-sizing: border-box; color: #4078c0; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Install Amazon EC2 Ansible integration tool</h4>
<div style="box-sizing: border-box; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 24px; margin-bottom: 16px;">
Go here and follow instructions for <a href="https://aws.amazon.com/blogs/apn/getting-started-with-ansible-and-dynamic-amazon-ec2-inventory-management/" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">Amazon EC2 Ansible integration tool</a>. You will run a Python script and setup a few environment variable. It is painless. This will create an ansible inventory file based on your EC2 environment.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://gist.github.com/RichardHightower/c37036dc5a55b302d5d99e2ffc1bcefd#start-ssh-agent-with-your-key" id="user-content-start-ssh-agent-with-your-key" style="box-sizing: border-box; color: #4078c0; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Start ssh agent with your key</h4>
<div class="highlight highlight-source-shell" style="box-sizing: border-box; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 24px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">$ ssh-agent bash
$ ssh-add <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">~</span>/.ssh/YOUR_KEY.pem </pre>
</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://gist.github.com/RichardHightower/c37036dc5a55b302d5d99e2ffc1bcefd#install-ansible-oracle-java-install-plugin" id="user-content-install-ansible-oracle-java-install-plugin" style="box-sizing: border-box; color: #4078c0; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Install ansible Oracle Java install plugin</h4>
<div class="highlight highlight-source-shell" style="box-sizing: border-box; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 24px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">$ ansible-galaxy install ansiblebit.oracle-java</pre>
</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://gist.github.com/RichardHightower/c37036dc5a55b302d5d99e2ffc1bcefd#start-up-an-ec2-box-tag-it-as-elkelk" id="user-content-start-up-an-ec2-box-tag-it-as-elkelk" style="box-sizing: border-box; color: #4078c0; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Start up an EC2 box tag it as elk=elk</h4>
<div style="box-sizing: border-box; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 24px; margin-bottom: 16px;">
However you like, start up an EC2 instance and tag it with the tag elk=elk. This is just the type of box. In this case, I am in the process of writing an ansible setup script for an ELK stack.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://gist.github.com/RichardHightower/c37036dc5a55b302d5d99e2ffc1bcefd#test-your-ansible-connection-with-the-ping-module" id="user-content-test-your-ansible-connection-with-the-ping-module" style="box-sizing: border-box; color: #4078c0; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Test your ansible connection with the ping module</h4>
<div class="highlight highlight-source-shell" style="box-sizing: border-box; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 24px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">$ ansible tag_elk_elk -m ping -u ec2-user
54.68.31.178 <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">|</span> SUCCESS =<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>changed<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">false</span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>ping<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>pong<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
}
</pre>
</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://gist.github.com/RichardHightower/c37036dc5a55b302d5d99e2ffc1bcefd#create-an-ansible-playbook" id="user-content-create-an-ansible-playbook" style="box-sizing: border-box; color: #4078c0; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Create an ansible playbook</h4>
<div class="highlight highlight-source-yaml" style="box-sizing: border-box; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 24px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">---
- <span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">hosts:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;">tag_elk_elk</span>
<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">user:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;">ec2-user</span>
<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">sudo:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;">yes</span>
<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">roles:</span>
- <span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">{ role:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;">ansiblebit.oracle-java,</span>
<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">oracle_java_set_as_default:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;">yes,</span>
<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">oracle_java_version:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;">8,</span>
<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">oracle_java_version_update:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;">102,</span>
<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">oracle_java_version_build:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;">14}</span>
<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">tasks:</span>
- <span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">name:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;">ensure apache is at the latest version</span>
<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">yum:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;">name=httpd state=latest</span></pre>
</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a aria-hidden="true" class="anchor" href="https://gist.github.com/RichardHightower/c37036dc5a55b302d5d99e2ffc1bcefd#run-the-ansible-playbook" id="user-content-run-the-ansible-playbook" style="box-sizing: border-box; color: #4078c0; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Run the ansible playbook</h4>
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; word-wrap: normal;"><code style="background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow: visible; padding: 0px; word-break: normal; word-wrap: normal;">$ ansible-playbook elk_install_playbook.yml
</code></pre>
<div style="box-sizing: border-box; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 24px;">
Ok. Now back to creating my ansible ELK install script. There are also tasks for creating an Amazon box with Ansible.</div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com0tag:blogger.com,1999:blog-6128525281101058933.post-67892759702657619882016-06-21T23:23:00.002-07:002016-06-21T23:23:38.060-07:00KPI Microservices Monitoring with QBit<div class="markdown-body" style="background-color: white; box-sizing: border-box; color: #333333; font-family: "Helvetica Neue", Helvetica, "Segoe UI", Arial, freesans, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 1.6; margin: 0px -30px; padding: 0px 30px; word-break: break-word; word-wrap: break-word;">
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
KPI Microservices Monitoring</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Home</a> <a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservice-Lib" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;"><<Part 1</a>, <a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Lib-Batteries-Included" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;"><Part 2</a>, Part 3 -- <a href="https://github.com/MammatusTech/qbit-microservices-examples/tree/master/todo-statsd" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">-example code-</a> <a href="https://github.com/advantageous/qbit/wiki" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">-qbit docs-</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<img alt="Mammatus Tech" data-canonical-src="http://www.mammatustech.com/_/rsrc/1242100869097/config/app/images/customLogo/customLogo.gif?revision=6" src="https://camo.githubusercontent.com/101992fdbcf4e5c9a212587dd2f4a2c7d9220ede/687474703a2f2f7777772e6d616d6d61747573746563682e636f6d2f5f2f727372632f313234323130303836393039372f636f6e6669672f6170702f696d616765732f637573746f6d4c6f676f2f637573746f6d4c6f676f2e6769663f7265766973696f6e3d36" style="border-style: none; box-sizing: content-box; max-width: 100%;" /></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
We recently revised this. Here is the <a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring-OLD" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">old version</a>. You can see how much QBit and Reakt have progressed.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
There has been a lot written on the subject of <a href="http://www.mammatustech.com/Home/reactive-microservices-monitoring" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Microservices Monitoring</a>. Monitoring is a bit of an overloaded term. There is service health monitoring, which can be done with tools like Mesosphere/Marathon, Nomad, Consul, etc. There is also KPI monitoring, which is done with tools like Grafana, Graphite, InfluxDB, StatsD, etc. Then there is log monitoring and search with tools like the ELK stack (elastic-search, LogStash and Kibana) and Splunk, where you can easily trace logs down to the requests or client ID in a request header. And, then there is system monitoring (JVM, slow query logs, network traffic, etc.), with tools like SystemD, and more. You will want all of this when you are doing Microservices Development.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The more insight you have into your system, the easier it will be to support and debug. Microservices imply async distributed development. Doing async distributed development without monitoring is like running with scissors.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
To summarize Microservices Monitoring is:</div>
<ul style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">KPI Monitoring (e.g., StatsD, Grafana, Graphite, InfluxDB, etc.)</li>
<li style="box-sizing: border-box;">Health Monitoring (e.g., Consul, Nomad, Mesosphere/Marathon, Heroku, etc.)</li>
<li style="box-sizing: border-box;">Log monitoring (e.g., ELK stack, Splunk, etc.)</li>
</ul>
<div style="box-sizing: border-box; margin-bottom: 16px;">
QBit has support for ELK/Splunk by providing support for <a href="http://rick-hightower.blogspot.com/2015/11/qbit-microservices-lib-mdc-logging.html" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">MDC</a>. QBit has support for systems that can monitor health like Mesosphere/Marathon, Heroku, Consul, Nomad, etc. by having an internal health system that QBit <span style="box-sizing: border-box; font-weight: bolder;"><em style="box-sizing: border-box;">service actors</em></span> all check-in with that then gets rolled up to other systems like Mesosphere/Marathon, Heroku, Consul, Nomad, etc.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
In this tutorial we are going to just cover KPI monitoring for microservices which is sometimes called Metrics Monitoring or Stats Monitoring. KPI stands for Key Performance Indicators. These are the things you really care about to see if your system is up and running, and how hard it is getting hit, and how it is performing.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
At the heart of the QBit KPI system is the Metrics collector. QBit uses the <a href="https://github.com/advantageous/metrik" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Metrik interface</a> for tracking Microservice KPIs.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#metrik-interface-for-tracking-kpis" id="user-content-metrik-interface-for-tracking-kpis" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Metrik Interface for tracking KPIs</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">interface</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">MetricsCollector</span> {
default <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">increment</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">name</span>) {
recordCount(name, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1</span>);
}
default <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">recordCount</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">name</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">long</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">count</span>) {
}
default <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">recordLevel</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">name</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">long</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">level</span>) {
}
default <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">recordTiming</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">name</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">long</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">duration</span>) {
}
}</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
We are recording counts per time period, current level or gauge at this instance in time and timings which is how long did something take.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#demonstrating-using-qbit-metrics" id="user-content-demonstrating-using-qbit-metrics" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Demonstrating using QBit metrics</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
This guide assumes you have read through the <a href="http://advantageous.github.io/qbit/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">main overview of QBit</a> and have gone through the first tutorials, but you should be able to follow along if you have not, you just will be able to follow along better if you read the docs (at least skimmed) and went through the first set of tutorials.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Let's show it. First we need to build. Use Gradle as follows:</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#gradlebuild" id="user-content-gradlebuild" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>gradle.build</h4>
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; word-wrap: normal;"><code style="background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow: visible; padding: 0px; word-break: normal; word-wrap: normal;">group 'qbit-ex'
version '1.0-SNAPSHOT'
apply plugin: 'java'
apply plugin: 'application'
mainClassName = "com.mammatustech.todo.TodoServiceMain"
compileJava {
sourceCompatibility = 1.8
}
repositories {
mavenCentral()
mavenLocal()
}
dependencies {
testCompile group: 'junit', name: 'junit', version: '4.11'
compile 'io.advantageous.reakt:reakt:2.8.17'
compile 'io.advantageous.qbit:qbit-admin:1.10.0.RELEASE'
compile 'io.advantageous.qbit:qbit-vertx:1.10.0.RELEASE'
}
dependencies {
testCompile group: 'junit', name: 'junit', version: '4.11'
}
</code></pre>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Read the comments in all of the code listings.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
In this example we will use StatsD but QBit is not limited to StatsD for Microservice KPI monitoring. In fact QBit can do amazing things with its StatsService like clustered rate limiting based on OAuth header info, but that is beyond this tutorial.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="https://github.com/advantageous/qbit/wiki/%5BZ-Blog%5D-StatsD-and-QBit" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">StatsD is a protocol</a> that you can send KPI messages to over UDP. Digital Ocean has a really nice<a href="https://www.digitalocean.com/community/tutorials/an-introduction-to-tracking-statistics-with-graphite-statsd-and-collectd" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">description of StatsD</a>.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The easiest way to setup StatsD is to use Docker. Docker is a great tool for development, and using Docker with Nomad, CoreOS, Mesosphere/Marathon, etc. is a great way to deploy Docker containers, but at a minimum you should be using the Docker tools for development.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Set up the StatsD server stack by using this <a href="https://hub.docker.com/r/samuelebistoletti/docker-statsd-influxdb-grafana/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">public docker container</a>.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
QBit ships with StatsD support in the <span style="box-sizing: border-box; font-weight: bolder;"><em style="box-sizing: border-box;">qbit-admin lib</em></span> (jar). It has done this for a long time.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
We will connect to StatsD with this URI.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#uri-to-connect-to-with-statsd" id="user-content-uri-to-connect-to-with-statsd" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>URI to connect to with StatsD</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">URI</span> statsdURI <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">URI</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>create(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>udp://192.168.99.100:8125<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Depending on how you have Docker setup, your URI might look a bit different. If you are running Docker tools with a Mac, then that should be your URI. (On Linux the above IO is likely to be<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">localhost</code> not <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">192.168.99.100</code>, go through the docker tool tutorials if you are lost at this point. It will be worth your time. I promise. I promise.invoke promise.)</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
If you have not already followed the instructions at <a href="https://hub.docker.com/r/samuelebistoletti/docker-statsd-influxdb-grafana/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">statsD, grafana, influxdb docker container docs</a>, do so now.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#running-docker" id="user-content-running-docker" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Running Docker</h4>
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; word-wrap: normal;"><code style="background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow: visible; padding: 0px; word-break: normal; word-wrap: normal;">docker run -d \
--name docker-statsd-influxdb-grafana \
-p 3003:9000 \
-p 3004:8083 \
-p 8086:8086 \
-p 22022:22 \
-p 8125:8125/udp \
samuelebistoletti/docker-statsd-influxdb-grafana
</code></pre>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The above yields</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#servers" id="user-content-servers" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Servers</h4>
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; word-wrap: normal;"><code style="background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow: visible; padding: 0px; word-break: normal; word-wrap: normal;">Host Port Service
3003 9000 grafana to see the results
8086 8086 influxdb to store the results
3004 8083 influxdb-admin to query the results
8125 8125 statsd server that listens to statsD UPD messages
22022 22 sshd
</code></pre>
<div style="box-sizing: border-box; margin-bottom: 16px;">
If you want to see the metrics and see if this is working, go through the <a href="https://docs.influxdata.com/influxdb/v0.8/introduction/getting_started/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">influxDB tutorial</a> and look around at the measurements with the influx-admin. Influx is a time series database. Grafana allows you to see pretty graphs and charts of the microservice KPIs that we are collecting. You will want to <a href="http://docs.grafana.org/guides/gettingstarted/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">learn grafana</a> as well.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
We use the host and port of the URI to connect to the StatsD daemon that is running on the docker container.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#setting-up-statsd-by-using-qbit-managedservicebuilder" id="user-content-setting-up-statsd-by-using-qbit-managedservicebuilder" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Setting up StatsD by using QBit managedServiceBuilder</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
managedServiceBuilder<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>enableStatsD(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">URI</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>create(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>udp://192.168.99.100:8125<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
managedServiceBuilder<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getContextMetaBuilder()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>setTitle(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>TodoMicroService<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
We covered using and setting up the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">managedServiceBuilder</code> in the first tutorials, and the complete code listing is below. You could use <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">managedServiceBuilder</code> to create a<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">statsCollector</code> as follows:</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#you-could-do-this-managedservicebuilder-to-create-the-statscollectormetricscollector" id="user-content-you-could-do-this-managedservicebuilder-to-create-the-statscollectormetricscollector" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>You could do this... managedServiceBuilder to create the StatsCollector/MetricsCollector</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-smi" style="box-sizing: border-box;">StatsCollector</span> statsCollector <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> managedServiceBuilder<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>createStatsCollector();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Start the service. */</span>
managedServiceBuilder<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>addEndpointService(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">TodoService</span>(reactor, statsCollector))</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Since services typically deal with the health system, the reactor (callback management, tasks management, repeating tasks) and the stats collector we created a ServiceManagementBundle that is a facade over the health system, stats, and the reactor.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#better-way-to-work-with-stats-health-and-the-reactor" id="user-content-better-way-to-work-with-stats-health-and-the-reactor" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Better way to work with stats, health and the reactor</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Create the management bundle for this service. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">ServiceManagementBundle</span> serviceManagementBundle <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span>
serviceManagementBundleBuilder()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>setServiceName(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>TodoServiceImpl<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
.setManagedServiceBuilder(managedServiceBuilder)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>build();</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The QBit <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">StatsCollector</code> interface extends the Metrik <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">MetricsCollector</code> interface (from QBit 1.5 onwards). <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceManagementBundle</code> has a stats method that returns a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">StatsCollector</code> as well as common facade methods on the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceManagementBundle</code></div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#using-the-statscollector" id="user-content-using-the-statscollector" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Using the StatsCollector.</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Then we just need to use it.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#using-the-statscollector-to-collect-kpis-about-our-service" id="user-content-using-the-statscollector-to-collect-kpis-about-our-service" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Using the StatsCollector to collect KPIs about our service</h4>
<div style="box-sizing: border-box; margin-bottom: 16px;">
For kicks, we track the KPI <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">todoservice.i.am.alive</code> every three seconds.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#tracking-kpi-iamamalive" id="user-content-tracking-kpi-iamamalive" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Tracking KPI i.am.am.alive</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">@RequestMapping(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/todo-service<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">class</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoServiceImpl</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">implements</span> <span class="pl-e" style="box-sizing: border-box; color: #795da3;">TodoService</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Map<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span>, <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span> todoMap <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">TreeMap<></span>();
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">ServiceManagementBundle</span> mgmt;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoServiceImpl</span>(<span class="pl-smi" style="box-sizing: border-box;">ServiceManagementBundle</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">mgmt</span>) {
<span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">this</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>mgmt <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> mgmt;
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Send stat count i.am.alive every three seconds. */</span>
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>reactor()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>addRepeatingTask(<span class="pl-smi" style="box-sizing: border-box;">Duration</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>ofSeconds(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">3</span>),
() <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>increment(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>i.am.alive<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
}</pre>
</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#tracking-calls-to-add-method" id="user-content-tracking-calls-to-add-method" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Tracking calls to add method</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> @<span class="pl-smi" style="box-sizing: border-box;">Override</span>
@<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">POST</span>(value <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/todo<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> addTodo(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span> todo) {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> invokablePromise(promise <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Send KPI addTodo called every time the addTodo method gets called. */</span>
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>increment(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>addTodo.called<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
todoMap<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>put(todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getId(), todo);
promise<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>accept(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">true</span>);
});
}
</pre>
</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#tracking-calls-to-remove-method" id="user-content-tracking-calls-to-remove-method" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Tracking calls to remove method</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> @<span class="pl-smi" style="box-sizing: border-box;">Override</span>
@<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">DELETE</span>(value <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/todo<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> removeTodo(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> @RequestParam(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>id<span class="pl-pds" style="box-sizing: border-box;">"</span></span>) <span class="pl-smi" style="box-sizing: border-box;">String</span> id) {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> invokablePromise(promise <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Send KPI addTodo.removed every time the removeTodo method gets called. */</span>
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>increment(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>removeTodo.called<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
todoMap<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>remove(id);
promise<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>accept(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">true</span>);
});
}
</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
You can register repeating tasks with @QueueCallback as follows:</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#managing-callbacks-and-repeating-tasks" id="user-content-managing-callbacks-and-repeating-tasks" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Managing callbacks and repeating tasks</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> @QueueCallback({<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">EMPTY</span>, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">IDLE</span>, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">LIMIT</span>})
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> process() {
reactor<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>process();
}
</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
But you do not need to if you use the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceManagementBundle</code>. Just specify it when you add the service to the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">managedServiceBuilder</code>.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#adding-service-to-managedservicebuilder-with-a-servicemanagementbundle" id="user-content-adding-service-to-managedservicebuilder-with-a-servicemanagementbundle" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Adding service to managedServiceBuilder with a serviceManagementBundle</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Start the service. */</span>
managedServiceBuilder
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Register TodoServiceImpl</span>
.addEndpointServiceWithServiceManagmentBundle(todoService, serviceManagementBundle)
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Build and start the server.</span>
.startApplication();</pre>
</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#complete-example" id="user-content-complete-example" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Complete example</h2>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#todojava" id="user-content-todojava" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Todo.java</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> <span class="pl-smi" style="box-sizing: border-box;">com.mammatustech.todo</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">class</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">Todo</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-smi" style="box-sizing: border-box;">String</span> id;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">String</span> name;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">String</span> description;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">long</span> createTime;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">Todo</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">name</span>, <span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">description</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">long</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">createTime</span>) {
<span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">this</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>name <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> name;
<span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">this</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>description <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> description;
<span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">this</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>createTime <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> createTime;
<span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">this</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>id <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> name <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">+</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>::<span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">+</span> createTime;
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getId</span>() {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">if</span> (id <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">==</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">null</span>) {
<span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">this</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>id <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> name <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">+</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>::<span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">+</span> createTime;
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> id;
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getName</span>() {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> name;
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getDescription</span>() {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> description;
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">long</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getCreateTime</span>() {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> createTime;
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@Override</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">boolean</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">equals</span>(<span class="pl-smi" style="box-sizing: border-box;">Object</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">o</span>) {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">if</span> (<span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">this</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">==</span> o) <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">true</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">if</span> (o <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">==</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">null</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">||</span> getClass() <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">!=</span> o<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getClass()) <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">false</span>;
<span class="pl-smi" style="box-sizing: border-box;">Todo</span> todo <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> (<span class="pl-smi" style="box-sizing: border-box;">Todo</span>) o;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">if</span> (createTime <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">!=</span> todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>createTime) <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">false</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">!</span>(name <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">!=</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">null</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">?</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">!</span>name<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>equals(todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>name) <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>name <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">!=</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">null</span>);
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@Override</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">int</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">hashCode</span>() {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">int</span> result <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> name <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">!=</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">null</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">?</span> name<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>hashCode() <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>;
result <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">31</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">*</span> result <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">+</span> (<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">int</span>) (createTime <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">^</span> (createTime <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">>>></span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">32</span>));
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> result;
}
}
</pre>
</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#todoserviceimpljava-to-show-tracking-kpis" id="user-content-todoserviceimpljava-to-show-tracking-kpis" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>TodoServiceImpl.java to show tracking KPIs.</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> <span class="pl-smi" style="box-sizing: border-box;">com.mammatustech.todo</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.admin.ServiceManagementBundle</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.annotation.RequestMapping</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.annotation.RequestMethod</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.annotation.RequestParam</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.annotation.http.DELETE</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.annotation.http.GET</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.annotation.http.POST</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.reakt.promise.Promise</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">java.time.Duration</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">java.util.ArrayList</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">java.util.Map</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">java.util.TreeMap</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import static</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.reakt.promise.Promises.invokablePromise</span>;
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Default port for admin is 7777.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Default port for main endpoint is 8888.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * <p></span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * <pre></span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * <code></span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Access the service:</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * $ curl http://localhost:8888/v1/...</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * To see swagger file for this service:</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * $ curl http://localhost:7777/__admin/meta/</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * To see health for this service:</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * $ curl http://localhost:8888/__health -v</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Returns "ok" if all registered health systems are healthy.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * OR if same port endpoint health is disabled then:</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * $ curl http://localhost:7777/__admin/ok -v</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Returns "true" if all registered health systems are healthy.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * A node is a service, service bundle, queue, or server endpoint that is being monitored.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * List all service nodes or endpoints</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * $ curl http://localhost:7777/__admin/all-nodes/</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * List healthy nodes by name:</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * $ curl http://localhost:7777/__admin/healthy-nodes/</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * List complete node information:</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * $ curl http://localhost:7777/__admin/load-nodes/</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Show service stats and metrics</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * $ curl http://localhost:8888/__stats/instance</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * </code></span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * </pre></span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
@RequestMapping(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/todo-service<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">class</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoServiceImpl</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">implements</span> <span class="pl-e" style="box-sizing: border-box; color: #795da3;">TodoService</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Map<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span>, <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span> todoMap <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">TreeMap<></span>();
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">ServiceManagementBundle</span> mgmt;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoServiceImpl</span>(<span class="pl-smi" style="box-sizing: border-box;">ServiceManagementBundle</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">mgmt</span>) {
<span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">this</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>mgmt <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> mgmt;
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Send stat count i.am.alive every three seconds. */</span>
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>reactor()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>addRepeatingTask(<span class="pl-smi" style="box-sizing: border-box;">Duration</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>ofSeconds(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">3</span>),
() <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>increment(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>i.am.alive<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@Override</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@POST</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">value</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/todo<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">addTodo</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">todo</span>) {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> invokablePromise(promise <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Send KPI addTodo called every time the addTodo method gets called. */</span>
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>increment(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>addTodo.called<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
todoMap<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>put(todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getId(), todo);
promise<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>accept(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">true</span>);
});
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@Override</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@DELETE</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">value</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/todo<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">removeTodo</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@RequestParam</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>id<span class="pl-pds" style="box-sizing: border-box;">"</span></span>) <span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">id</span>) {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> invokablePromise(promise <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Send KPI addTodo.removed every time the removeTodo method gets called. */</span>
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>increment(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>removeTodo.called<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
todoMap<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>remove(id);
promise<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>accept(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">true</span>);
});
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@Override</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@GET</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">value</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/todo<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">method</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">RequestMethod</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>GET</span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-k" style="box-sizing: border-box;">ArrayList<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">listTodos</span>() {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> invokablePromise(promise <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Send KPI addTodo.listTodos every time the listTodos method gets called. */</span>
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>increment(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>listTodos.called<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
promise<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>accept(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">ArrayList<></span>(todoMap<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>values()));
});
}
}
</pre>
</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#todoservicemainjava-showing-how-to-configure-statsd-qbit-for-microservice-kpi-tracking" id="user-content-todoservicemainjava-showing-how-to-configure-statsd-qbit-for-microservice-kpi-tracking" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>TodoServiceMain.java showing how to configure StatsD QBit for MicroService KPI tracking</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> <span class="pl-smi" style="box-sizing: border-box;">com.mammatustech.todo</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.admin.ManagedServiceBuilder</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.admin.ServiceManagementBundle</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">java.net.URI</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import static</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.admin.ManagedServiceBuilder.managedServiceBuilder</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import static</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.admin.ServiceManagementBundleBuilder.serviceManagementBundleBuilder</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">class</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoServiceMain</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">static</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">main</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">String</span>... <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">args</span>) <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">throws</span> <span class="pl-smi" style="box-sizing: border-box;">Exception</span> {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Create the ManagedServiceBuilder which manages a clean shutdown, health, stats, etc. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">ManagedServiceBuilder</span> managedServiceBuilder <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> managedServiceBuilder()
.setRootURI(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/v1<span class="pl-pds" style="box-sizing: border-box;">"</span></span>) <span class="pl-c" style="box-sizing: border-box; color: #969896;">//Defaults to services</span>
.setPort(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">8888</span>); <span class="pl-c" style="box-sizing: border-box; color: #969896;">//Defaults to 8080 or environment variable PORT</span>
managedServiceBuilder<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>enableStatsD(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">URI</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>create(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>udp://192.168.99.100:8125<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
managedServiceBuilder<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getContextMetaBuilder()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>setTitle(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>TodoMicroService<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Create the management bundle for this service. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">ServiceManagementBundle</span> serviceManagementBundle <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span>
serviceManagementBundleBuilder()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>setServiceName(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>TodoServiceImpl<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
.setManagedServiceBuilder(managedServiceBuilder)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>build();
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">TodoService</span> todoService <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">TodoServiceImpl</span>(serviceManagementBundle);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Start the service. */</span>
managedServiceBuilder
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Register TodoServiceImpl</span>
.addEndpointServiceWithServiceManagmentBundle(todoService, serviceManagementBundle)
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Build and start the server.</span>
.startApplication();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Start the admin builder which exposes health end-points and swagger meta data. */</span>
managedServiceBuilder<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getAdminBuilder()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>build()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>startServer();
<span class="pl-smi" style="box-sizing: border-box;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>out<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>println(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Todo Server and Admin Server started<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
}
}
</pre>
</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#todoservice-interface" id="user-content-todoservice-interface" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>TodoService interface</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> <span class="pl-smi" style="box-sizing: border-box;">com.mammatustech.todo</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.reakt.promise.Promise</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">java.util.ArrayList</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">interface</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoService</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">addTodo</span>(<span class="pl-smi" style="box-sizing: border-box;">Todo</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">todo</span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">removeTodo</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">id</span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-k" style="box-sizing: border-box;">ArrayList<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">listTodos</span>();
}</pre>
</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring#todoserviceimpltest-that-shows-how-to-unit-test-with-reakt" id="user-content-todoserviceimpltest-that-shows-how-to-unit-test-with-reakt" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>TodoServiceImplTest that shows how to unit test with Reakt</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> <span class="pl-smi" style="box-sizing: border-box;">com.mammatustech.todo</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.admin.ManagedServiceBuilder</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.admin.ServiceManagementBundle</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.queue.QueueCallBackHandler</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.service.ServiceBuilder</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">org.junit.Test</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">java.util.concurrent.TimeUnit</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import static</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.admin.ManagedServiceBuilder.managedServiceBuilder</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import static</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.admin.ServiceManagementBundleBuilder.serviceManagementBundleBuilder</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import static</span> <span class="pl-smi" style="box-sizing: border-box;">junit.framework.TestCase.assertFalse</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import static</span> <span class="pl-smi" style="box-sizing: border-box;">org.junit.Assert.assertTrue</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">class</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoServiceImplTest</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@Test</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">test</span>() <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">throws</span> <span class="pl-smi" style="box-sizing: border-box;">Exception</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">TodoService</span> todoService <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> createTodoService();
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span> rick <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>foo<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>rick<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1L</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Add Rick</span>
assertTrue(todoService
.addTodo(rick)
.invokeAsBlockingPromise()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get());
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Add Diana</span>
assertTrue(todoService
.addTodo(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>bar<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>diana<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1L</span>))
.invokeAsBlockingPromise()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get());
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Remove Rick</span>
assertTrue(todoService<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>removeTodo(rick<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getId())
.invokeAsBlockingPromise()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get());
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Make sure Diana is in the listTodos</span>
assertTrue(todoService<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>listTodos()
.invokeAsBlockingPromise()
.get()
.stream()
.filter(
todo <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getDescription()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>equals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>diana<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
)
.findFirst()
.isPresent()
);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Make sure Rick is not in the listTodos</span>
assertFalse(todoService<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>listTodos()
.invokeAsBlockingPromise()
.get()
.stream()
.filter(
todo <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getDescription()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>equals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>rick<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
)
.findFirst()
.isPresent()
);
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-smi" style="box-sizing: border-box;">TodoService</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">createTodoService</span>() {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Create the ManagedServiceBuilder which manages a clean shutdown, health, stats, etc. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">ManagedServiceBuilder</span> managedServiceBuilder <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> managedServiceBuilder(); <span class="pl-c" style="box-sizing: border-box; color: #969896;">//Defaults to 8080 or environment variable PORT</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Create the management bundle for this service. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">ServiceManagementBundle</span> serviceManagementBundle <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span>
serviceManagementBundleBuilder()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>setServiceName(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>TodoService<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
.setManagedServiceBuilder(managedServiceBuilder)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>build();
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">TodoService</span> todoServiceImpl <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">TodoServiceImpl</span>(serviceManagementBundle);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> <span class="pl-smi" style="box-sizing: border-box;">ServiceBuilder</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>serviceBuilder()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>setServiceObject(todoServiceImpl)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>addQueueCallbackHandler(
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">QueueCallBackHandler</span>() {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@Override</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">queueProcess</span>() {
serviceManagementBundle<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>process();
}
})
.buildAndStartAll()
.createProxyWithAutoFlush(<span class="pl-smi" style="box-sizing: border-box;">TodoService</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">50</span>, <span class="pl-smi" style="box-sizing: border-box;">TimeUnit</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>MILLISECONDS</span>);
}
}</pre>
</div>
</div>
<div class="wiki-footer gollum-markdown-content boxed-group" id="wiki-footer" style="background-color: white; border-radius: 3px; box-sizing: border-box; clear: none; color: #333333; font-family: Helvetica, arial, nimbussansl, liberationsans, freesans, clean, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 13px; line-height: 18.2px; margin: 30px 30px 0px; position: relative;">
<div class="boxed-group-inner wiki-auxiliary-content wiki-writable markdown-body" style="background: rgb(246, 246, 246); border-radius: 3px; border: 1px solid rgb(216, 216, 216); box-shadow: rgba(0, 0, 0, 0.0588235) 0px 1px 2px; box-sizing: border-box; color: #666666; font-family: "Helvetica Neue", Helvetica, "Segoe UI", Arial, freesans, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; line-height: 1.6; margin: 0px -30px; padding: 10px 15px; word-break: break-word; word-wrap: break-word;">
<a class="wiki-edit-link" href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/_Footer/_edit" style="background-color: transparent; box-sizing: border-box; color: #767676; float: right; margin-top: 0px !important; opacity: 0.2; position: relative; right: -5px; text-decoration: none; transition: opacity 0.2s ease-in-out; z-index: 2;" title="Edit footer"><svg aria-hidden="true" class="octicon octicon-pencil" height="16" version="1.1" viewbox="0 0 14 16" width="14"><path d="M0 12v3h3l8-8-3-3-8 8zm3 2H1v-2h1v1h1v1zm10.3-9.3L12 6 9 3l1.3-1.3a.996.996 0 0 1 1.41 0l1.59 1.59c.39.39.39 1.02 0 1.41z"></path></svg></a><div style="box-sizing: border-box; margin-bottom: 16px;">
Read more:</div>
<ol style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservice-Lib" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit Microservice Hello World tutorial</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Lib-Batteries-Included" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit Microservice Hello World Part 2</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Health-Checks" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit Microservice Hello World Health Checks</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Stats-KPI-Metrics-Monitoring" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit Microservice Hello World Stats, Metrics, and Monitoring</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Reactor-tutorial--%7C-reactively-handling-async-calls-with-QBit-Reactive-Microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit Microservice Reactive programming tutorial</a></li>
</ol>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<img alt="Mammatus Tech" data-canonical-src="http://www.mammatustech.com/_/rsrc/1242100869097/config/app/images/customLogo/customLogo.gif?revision=6" src="https://camo.githubusercontent.com/101992fdbcf4e5c9a212587dd2f4a2c7d9220ede/687474703a2f2f7777772e6d616d6d61747573746563682e636f6d2f5f2f727372632f313234323130303836393039372f636f6e6669672f6170702f696d616765732f637573746f6d4c6f676f2f637573746f6d4c6f676f2e6769663f7265766973696f6e3d36" style="background: none; border-style: none; box-sizing: content-box; max-width: 100%;" /></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
QBit is the Java microservice lib. QBit is a reactive programming lib for building microservices and focuses on JSON, HTTP, WebSocket, and REST. QBit uses reactive programming to build elastic REST, and WebSockets based cloud friendly, web services. SOA evolved for mobile and cloud. ServiceDiscovery, Health, reactive StatService, events, Java idiomatic reactive programming for Microservices.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://rick-hightower.blogspot.com/2015/03/reactive-programming-service-discovery.html" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactive Programming</a>, <a href="http://rick-hightower.blogspot.com/2015/03/java-microservices-architecture.html" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Java Microservices</a>, <a href="http://www.linkedin.com/in/rickhigh" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Rick Hightower</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">High-speed microservices consulting firm and authors of QBit with lots of experience with Vertx - Mammatus Technology</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.aboutobjects.com/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Highly recommended consulting and training firm who specializes in microservices architecture and mobile development that are already very familiar with QBit and Vertx as well as iOS and Android - About Objects</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/java-microservices-architecture" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Java Microservices Architecture</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/Microservice-Service-Discovery-with-Consul" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Microservice Service Discovery with Consul</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/consul-service-discovery-and-health-for-microservices-architecture-tutorial" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Microservices Service Discovery Tutorial with Consul</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/reactive-microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactive Microservices</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/high-speed-microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">High Speed Microservices</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/java-microservices-consulting" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Java Microservices Consulting</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/java-reactive-microservice-training" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Microservices Training</a></div>
<div style="box-sizing: border-box;">
<a href="http://www.mammatustech.com/introduction-to-apache-spark" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Introduction to Apache Spark</a></div>
</div>
</div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com0tag:blogger.com,1999:blog-6128525281101058933.post-87769741407283087342016-05-22T14:20:00.002-07:002016-05-22T14:22:30.361-07:00Understanding the QBit microservices lib's serviceEndpointServer<div class="markdown-body" style="background-color: white; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 1.6; margin: 0px -30px; padding: 0px 30px; word-break: break-word; word-wrap: break-word;">
<div style="box-sizing: border-box; margin-bottom: 16px;">
The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceEndpointServer</code> essentially exposes a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceBundle</code> to <em style="box-sizing: border-box;">WebSocket</em> and <em style="box-sizing: border-box;">REST </em>remote calls. This document is using the Todo example from the discussion of <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; color: #4078c0; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px; text-decoration: none;"><a href="https://github.com/advantageous/qbit/wiki/ServiceQueue" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">ServiceQueue</a> </code>and the <a href="https://github.com/advantageous/qbit/wiki/ServiceBundle" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceBundle</code></a>.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
In fact, you can use <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceEndpointServer</code> very similar to the way we used <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceBundle</code>.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceEndpointServer#creating-a-serviceendpointserver" id="user-content-creating-a-serviceendpointserver" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Creating a serviceEndpointServer</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.server.EndpointServerBuilder</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.server.ServiceEndpointServer</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import static</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.server.EndpointServerBuilder.endpointServerBuilder</span>;
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Create the serviceBundleBuilder. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">EndpointServerBuilder</span> endpointServerBuilder <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span>
endpointServerBuilder();
endpointServerBuilder<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>addService(auditorAddress,
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">AuditorImpl</span>());
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Create the service endpoint server. */</span>
serviceEndpointServer <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> endpointServerBuilder<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>build();</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
We use a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">EndpointServerBuilder</code> to build a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceEndpointServer</code>. You can add services to the builder or you can add them directly to the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceEndpointServer</code>.</div>
<blockquote style="border-left-color: rgb(221, 221, 221); border-left-style: solid; border-left-width: 4px; box-sizing: border-box; color: #777777; margin: 0px 0px 16px; padding: 0px 15px;">
<div style="box-sizing: border-box;">
Note you can use <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">EndpointServerBuilder</code> but most examples will use the<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ManagedServiceBuilder</code> which has the benefit of wiring the services it creates into the microservice health check system and the microservice statistics/monitoring/distributed MDC logging systems that QBit provides.</div>
</blockquote>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceEndpointServer</code> has a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundle</code>.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceEndpointServer#using-serviceendpointservers-servicebundle" id="user-content-using-serviceendpointservers-servicebundle" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using serviceEndpointServer's serviceBundle</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Create a service client proxy for the auditor. */</span>
auditor <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> serviceEndpointServer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>serviceBundle()
.createLocalProxy(<span class="pl-smi" style="box-sizing: border-box;">Auditor</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class, auditorAddress);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Create a todo manager and pass the </span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> client proxy of the auditor to it. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">TodoManagerImpl</span> todoManager <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">TodoManagerImpl</span>(auditor);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Add the todoManager to the serviceBundle.</span>
serviceEndpointServer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>serviceBundle()
.addServiceObject(todoAddress, todoManager);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Create a client proxy to communicate </span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> with the service actor. */</span>
client <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> serviceEndpointServer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>serviceBundle()
.createLocalProxy(<span class="pl-smi" style="box-sizing: border-box;">TodoManagerClient</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class,
todoAddress);</pre>
</div>
<blockquote style="border-left-color: rgb(221, 221, 221); border-left-style: solid; border-left-width: 4px; box-sizing: border-box; color: #777777; margin: 0px 0px 16px; padding: 0px 15px;">
<div style="box-sizing: border-box;">
Note if we wanted to hide access to the auditor, we could put the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">auditor</code> in another<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> or <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundle</code> that was not accessible to WebSocket or REST.</div>
</blockquote>
<div style="box-sizing: border-box; margin-bottom: 16px;">
We can use the proxy client just like we did before. We can create a <em style="box-sizing: border-box;">local microservice actor proxy client</em>. The only real difference is that auto flush is built into <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceEndpointServer</code> and not<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundle</code>.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceEndpointServer#example-of-making-local-calls-to-the-todoservice" id="user-content-example-of-making-local-calls-to-the-todoservice" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Example of making local calls to the TodoService</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-c" style="box-sizing: border-box; color: #969896;">/* A list of promises for things we want to do all at once. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-k" style="box-sizing: border-box;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span>></span> promises <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">ArrayList<></span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">3</span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">CountDownLatch</span> latch <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">CountDownLatch</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1</span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">AtomicBoolean</span> success <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">AtomicBoolean</span>();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Add a todoItem to the client add method */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span> todo <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>write<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Write tutorial<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, timer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>time());
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> promise
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(todo);
promises<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(promise);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Add two more. */</span>
promises<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>callMom<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Call Mom<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, timer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>time())));
promises<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>callSis<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Call Sister<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, timer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>time())));
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Now async wait for them all to come back. */</span>
<span class="pl-smi" style="box-sizing: border-box;">Promises</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>all(promises)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>then(done <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
success<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>set(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">true</span>);
latch<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>countDown();
})<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>catchError(e <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
success<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>set(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">false</span>);
latch<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>countDown();
});
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Invoke the promises. */</span>
promises<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>forEach(<span class="pl-smi" style="box-sizing: border-box;">Promise</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">::</span>invoke);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** They are all going to come back async. */</span>
latch<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>await();
assertTrue(success<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get());</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Ok. Up until this point, nothing is really different than before. The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">TodoManagerImpl</code> is now accessible via REST and WebSocket.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceEndpointServer#using-todomanager-service-over-websocket" id="user-content-using-todomanager-service-over-websocket" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using TodoManager service over WebSocket</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.client.Client</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.client.ClientBuilder</span>;
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//REMOVE THIS Create a client proxy to communicate with the service actor.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//REMOVE client = serviceEndpointServer.serviceBundle()</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//REMOVE .createLocalProxy(TodoManagerClient.class, todoAddress);</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Start the service endpoint server </span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> and wait until it starts. */</span>
serviceEndpointServer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>startServerAndWait();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Create the WebSocket Client Builder. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">ClientBuilder</span> clientBuilder <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">ClientBuilder</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>clientBuilder();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Build the webSocketClient. */</span>
webSocketClient <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> clientBuilder<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>setHost(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>localhost<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
.setPort(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">8080</span>)
.build();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Create a REMOTE client proxy to communicate with the service actor. */</span>
client <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> webSocketClient<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>createProxy(<span class="pl-smi" style="box-sizing: border-box;">TodoManagerClient</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class, todoAddress);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Start the remote client. */</span>
webSocketClient<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>start();
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
@<span class="pl-smi" style="box-sizing: border-box;">After</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> tearDown() throws <span class="pl-smi" style="box-sizing: border-box;">Exception</span>{
<span class="pl-smi" style="box-sizing: border-box;">Thread</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>sleep(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">100</span>);
serviceEndpointServer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>stop(); <span class="pl-c" style="box-sizing: border-box; color: #969896;">//stop the server</span>
webSocketClient<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>stop(); <span class="pl-c" style="box-sizing: border-box; color: #969896;">//stop the client</span>
}
</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The client like the service endpoint server also auto-flushes. You can use the remote client (<em style="box-sizing: border-box;">remote microservice client proxy</em>) just like before (when we showed the local microservice client proxy).</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceEndpointServer#remote-client-gets-used-just-like-the-local-client" id="user-content-remote-client-gets-used-just-like-the-local-client" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Remote client gets used just like the local client.</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-c" style="box-sizing: border-box; color: #969896;">/* A list of promises for things we want to do all at once. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-k" style="box-sizing: border-box;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span>></span> promises <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">ArrayList<></span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">3</span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">CountDownLatch</span> latch <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">CountDownLatch</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1</span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">AtomicBoolean</span> success <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">AtomicBoolean</span>();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Add a todoItem to the client add method */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span> todo <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>write<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Write tutorial<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, timer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>time());
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> promise
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(todo);
promises<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(promise);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Add two more. */</span>
promises<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>callMom<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Call Mom<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, timer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>time())));
promises<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>callSis<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Call Sister<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, timer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>time())));
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Now async wait for them all to come back. */</span>
<span class="pl-smi" style="box-sizing: border-box;">Promises</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>all(promises)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>then(done <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
success<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>set(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">true</span>);
latch<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>countDown();
})<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>catchError(e <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
success<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>set(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">false</span>);
latch<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>countDown();
});
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Invoke the promises. */</span>
promises<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>forEach(<span class="pl-smi" style="box-sizing: border-box;">Promise</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">::</span>invoke);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** They are all going to come back async. */</span>
latch<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>await();
assertTrue(success<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get());</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
To expose the TodoManagerImpl to REST, we will define a main method to start the server. Then we will add <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">@RequestMapping</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">@POST</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">@PUT</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">@DELETE/@RequestParam</code>, and <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">@GET</code>.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceEndpointServer#adding--requestmapping-post-put-deleterequestparam-and-get" id="user-content-adding--requestmapping-post-put-deleterequestparam-and-get" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Adding <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: inherit; margin: 0px; padding: 0.2em 0px;">@RequestMapping</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: inherit; margin: 0px; padding: 0.2em 0px;">@POST</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: inherit; margin: 0px; padding: 0.2em 0px;">@PUT</code>,<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: inherit; margin: 0px; padding: 0.2em 0px;">@DELETE/@RequestParam</code>, and <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: inherit; margin: 0px; padding: 0.2em 0px;">@GET</code></h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> <span class="pl-smi" style="box-sizing: border-box;">com.mammatustech.todo</span>;
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.annotation.*</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.annotation.http.DELETE</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.annotation.http.GET</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.annotation.http.PUT</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.reactive.Callback</span>;
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
@RequestMapping(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/todo-service<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">class</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoManagerImpl</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Map<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span>, <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span> todoMap <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">TreeMap<></span>();
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">Auditor</span> auditor;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoManagerImpl</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">Auditor</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">auditor</span>) {
<span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">this</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>auditor <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> auditor;
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@GET</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/todo/count<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">int</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">size</span>() {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> todoMap<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>size();
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@PUT</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/todo/<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">add</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Callback<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">callback</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">todo</span>) {
todoMap<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>put(todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getId(), todo);
auditor<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>audit(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>add<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>added new todo<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>resolve(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">true</span>);
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@DELETE</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/todo/<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">remove</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Callback<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">callback</span>,
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@RequestParam</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>id<span class="pl-pds" style="box-sizing: border-box;">"</span></span>) <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">id</span>) {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span> removed <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> todoMap<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>remove(id);
auditor<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>audit(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>add<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>removed new todo<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>resolve(removed <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">!=</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">null</span>);
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@GET</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/todo/<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">list</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Callback<<span class="pl-k" style="box-sizing: border-box;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span>></span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">callback</span>) {
auditor<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>audit(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>list<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>auditor added<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>accept(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">ArrayList<></span>(todoMap<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>values()));
}
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
}
</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The main method just creates the microservices and starts the server.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceEndpointServer#main-method-to-start-the-service" id="user-content-main-method-to-start-the-service" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Main method to start the service</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> <span class="pl-smi" style="box-sizing: border-box;">com.mammatustech.todo</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.server.EndpointServerBuilder</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.server.ServiceEndpointServer</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import static</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.server.EndpointServerBuilder.endpointServerBuilder</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">class</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoServiceMain</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">static</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">main</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">String</span>... <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">args</span>) {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Object address to the auditorService service actor. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">String</span> auditorAddress <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>auditorService<span class="pl-pds" style="box-sizing: border-box;">"</span></span>;
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Create the serviceBundleBuilder. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">EndpointServerBuilder</span> endpointServerBuilder <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span>
endpointServerBuilder();
endpointServerBuilder<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>setPort(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">8080</span>)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>setUri(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
endpointServerBuilder<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>addService(auditorAddress,
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">AuditorImpl</span>());
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Create the service server. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">ServiceEndpointServer</span> serviceEndpointServer <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span>
endpointServerBuilder<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>build();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Create a service client proxy for the auditor. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">Auditor</span> auditor <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> serviceEndpointServer
.serviceBundle()
.createLocalProxy(<span class="pl-smi" style="box-sizing: border-box;">Auditor</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class, auditorAddress);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Create a todo manager and pass </span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> the client proxy of the auditor to it. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">TodoManagerImpl</span> todoManager <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">TodoManagerImpl</span>(auditor);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Add the todoManager to the serviceBundle.</span>
serviceEndpointServer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>addService(todoManager);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Start the service endpoint server </span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> and wait until it starts. */</span>
serviceEndpointServer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>startServerAndWait();
<span class="pl-smi" style="box-sizing: border-box;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>out<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>println(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Started<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
}
}</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
No RESTful microservice is proven to be RESTful without some curl script.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceEndpointServer#curl-accessing-service" id="user-content-curl-accessing-service" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>curl accessing service</h4>
<div class="highlight highlight-source-shell" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">echo</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Todo item list before <span class="pl-pds" style="box-sizing: border-box;">"</span></span>
curl http://localhost:8080/todo-service/todo/
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">echo</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">echo</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Count of Todo items <span class="pl-pds" style="box-sizing: border-box;">"</span></span>
curl http://localhost:8080/todo-service/todo/count
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">echo</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">echo</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>PUT a TODO item<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
curl -X PUT http://localhost:8080/todo-service/todo/ \
-H <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>Content-Type: application/json<span class="pl-pds" style="box-sizing: border-box;">'</span></span> \
-d <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>{"name":"wash-car", "description":"Take the car to the car wash", "createTime":1463950095000}<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">echo</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">echo</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Todo item list after add <span class="pl-pds" style="box-sizing: border-box;">"</span></span>
curl http://localhost:8080/todo-service/todo/
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">echo</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">echo</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Count of Todo items after add <span class="pl-pds" style="box-sizing: border-box;">"</span></span>
curl http://localhost:8080/todo-service/todo/count
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">echo</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">echo</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Remove a TODO item<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
curl -X DELETE http://localhost:8080/todo-service/todo/<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">?</span>id=wash-car::1463950095000
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">echo</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">echo</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Todo item list after add <span class="pl-pds" style="box-sizing: border-box;">"</span></span>
curl http://localhost:8080/todo-service/todo/
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">echo</span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">echo</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Count of Todo items after add <span class="pl-pds" style="box-sizing: border-box;">"</span></span>
curl http://localhost:8080/todo-service/todo/count
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">echo</span>
$ ./curl-test.sh
Todo item list before
[]
Count of Todo items
0
PUT a TODO item
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">true</span>
Todo item list after add
[{<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>name<span class="pl-pds" style="box-sizing: border-box;">"</span></span>:<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>wash-car<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>description<span class="pl-pds" style="box-sizing: border-box;">"</span></span>:<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Take the car to the car wash<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>createTime<span class="pl-pds" style="box-sizing: border-box;">"</span></span>:1463950095000,<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>id<span class="pl-pds" style="box-sizing: border-box;">"</span></span>:<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>wash-car::1463950095000<span class="pl-pds" style="box-sizing: border-box;">"</span></span>}]
Count of Todo items after add
1
Remove a TODO item
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">true</span>
Todo item list after add
[]
Count of Todo items after add
0</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
For completeness, here is the build file.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceEndpointServer#build-file" id="user-content-build-file" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Build file</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">group <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>qbit-ex<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
version <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>1.0-SNAPSHOT<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
apply plugin<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>java<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
apply plugin<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>application<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
mainClassName <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>com.mammatustech.todo.TodoServiceMain<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
compileJava {
sourceCompatibility <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1.8</span>
}
repositories {
mavenCentral()
mavenLocal()
}
dependencies {
testCompile group<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>junit<span class="pl-pds" style="box-sizing: border-box;">'</span></span>, name<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>junit<span class="pl-pds" style="box-sizing: border-box;">'</span></span>, version<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>4.11<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
compile <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>io.advantageous.qbit:qbit-vertx:1.9.1<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
compile <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>io.advantageous.qbit:qbit-admin:1.9.1<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
compile <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>io.advantageous.reakt:reakt:2.8.15<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
}
</pre>
</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceEndpointServer#conclusion" id="user-content-conclusion" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Conclusion</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
A <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceEndpointServer</code> exposes a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceBundle</code> as a remote accessible microservice whose methods can be invoked over <em style="box-sizing: border-box;">WebSocket</em> and <em style="box-sizing: border-box;">HTTP/REST</em>. Remote proxies can be created with QBit <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Client/ClientBuilder</code>. The ServiceEndpointServer and the Client are both auto flushing (interval duration of flush is configurable from their respective builders).</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
To learn more about QBit and REST see <a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/QBit-Microservices-Java-Lib-RESTful" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Restful QBit tutorial</a> and <a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/QBit-Resourceful-RESTful-Microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Resourceful RESTful Microservices tutorial</a>.</div>
<div style="box-sizing: border-box;">
To learn about the <b><i>ManagedServiceBuilder</i></b> please read <a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Getting-started-with-QBit-Microservices-Lib-Batteries-Included" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit Batteries included</a> which covers health, stats and microservice monitoring. The QBit batteries included also covers using QBit with <b><i>Swagger</i></b>. QBit can generate swagger JSON from all of its services which you can then use to generate clients for other platforms.</div>
</div>
<div class="wiki-footer gollum-markdown-content boxed-group" id="wiki-footer" style="background-color: white; border-radius: 3px; box-sizing: border-box; clear: none; color: #333333; font-family: Helvetica, arial, nimbussansl, liberationsans, freesans, clean, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 13px; line-height: 18.2px; margin: 30px 30px 0px; position: relative;">
<div class="boxed-group-inner wiki-auxiliary-content wiki-writable markdown-body" style="background: rgb(246, 246, 246); border-radius: 3px; border: 1px solid rgb(216, 216, 216); box-shadow: rgba(0, 0, 0, 0.0588235) 0px 1px 2px; box-sizing: border-box; color: #666666; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; line-height: 1.6; margin: 0px -30px; padding: 10px 15px; word-break: break-word; word-wrap: break-word;">
<a class="wiki-edit-link" href="https://github.com/advantageous/qbit/wiki/_Footer/_edit" style="background-color: transparent; box-sizing: border-box; color: #767676; float: right; margin-top: 0px !important; opacity: 0.2; position: relative; right: -5px; text-decoration: none; transition: opacity 0.2s ease-in-out; z-index: 2;" title="Edit footer"><svg aria-hidden="true" class="octicon octicon-pencil" height="16" version="1.1" viewbox="0 0 14 16" width="14"><path d="M0 12v3h3l8-8-3-3L0 12z m3 2H1V12h1v1h1v1z m10.3-9.3l-1.3 1.3-3-3 1.3-1.3c0.39-0.39 1.02-0.39 1.41 0l1.59 1.59c0.39 0.39 0.39 1.02 0 1.41z"></path></svg></a><br />
<ul style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px !important; padding-left: 2em;">
<li style="box-sizing: border-box;"><a href="http://advantageous.github.io/qbit/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit Reactive Microservices</a></li>
<li style="box-sizing: border-box;"><a href="http://advantageous.github.io/reakt" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt</a></li>
<li style="box-sizing: border-box;"><a href="http://advantageous.github.io/reakt-guava/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt Guava Bridge</a></li>
</ul>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceEndpointServer#further-reading" id="user-content-further-reading" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Further Reading</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://advantageous.github.io/qbit/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit Website</a> <a href="http://www.mammatustech.com/microservices-architecture" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">What is Microservices Architecture?</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit Java Micorservices lib tutorials</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The Java microservice lib. QBit is a reactive programming lib for building microservices - JSON, HTTP, WebSocket, and REST. QBit uses reactive programming to build elastic REST, and WebSockets based cloud friendly, web services. SOA evolved for mobile and cloud. ServiceDiscovery, Health, reactive StatService, events, Java idiomatic reactive programming for Microservices.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Find more tutorial on QBit</a>.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://rick-hightower.blogspot.com/2015/03/reactive-programming-service-discovery.html" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactive Programming</a>, <a href="http://rick-hightower.blogspot.com/2015/03/java-microservices-architecture.html" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Java Microservices</a>, <a href="http://www.linkedin.com/in/rickhigh" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Rick Hightower</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">High-speed microservices consulting firm and authors of QBit with lots of experience with Vertx - Mammatus Technology</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.aboutobjects.com/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Highly recommended consulting and training firm who specializes in microservices architecture and mobile development that are already very familiar with QBit and Vertx as well as iOS and Android - About Objects</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/java-microservices-architecture" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Java Microservices Architecture</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/Microservice-Service-Discovery-with-Consul" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Microservice Service Discovery with Consul</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/consul-service-discovery-and-health-for-microservices-architecture-tutorial" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Microservices Service Discovery Tutorial with Consul</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/reactive-microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactive Microservices</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/high-speed-microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">High Speed Microservices</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/java-microservices-consulting" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Java Microservices Consulting</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/java-reactive-microservice-training" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Microservices Training</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Reactor-tutorial--%7C-reactively-handling-async-calls-with-QBit-Reactive-Microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactive Microservices Tutorial, using the Reactor</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://restlet.com/blog/2015/09/04/this-week-in-api-land-20/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit is mentioned in the Restlet blog</a></div>
</div>
</div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com0tag:blogger.com,1999:blog-6128525281101058933.post-72615427944647291282016-05-15T21:55:00.002-07:002016-05-15T21:55:15.484-07:00Understanding the QBit microervices lib's serviceBundle<div class="markdown-body" style="box-sizing: border-box; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 1.6; margin: 0px -30px; padding: 0px 30px; word-break: break-word; word-wrap: break-word;">
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 0px !important;">
Understanding the serviceBundle</h4>
<blockquote style="border-left-color: rgb(221, 221, 221); border-left-style: solid; border-left-width: 4px; box-sizing: border-box; color: #777777; margin: 0px 0px 16px; padding: 0px 15px;">
<div style="box-sizing: border-box;">
<a href="https://github.com/advantageous/qbit/wiki/%5BZ-Design%5D-Existing-design-of-method-dispatch-(circa-2016)-May-3rd" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">There are some details about serviceBundles</a>, and they are covered in the <a href="https://github.com/advantageous/qbit/wiki" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit microservices overview as well</a>.</div>
</blockquote>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundle</code> is a collection of services sitting behind <a href="https://github.com/advantageous/qbit/wiki/ServiceQueue" style="box-sizing: border-box; color: #4078c0; text-decoration: none;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code>'s</a>. You use a<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundle</code> when you want to share a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">response queue</code> and a response queue thread. The<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundle</code> can also share the same thread for the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">request queue</code> but that is not the default. The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceEndpointServer</code> which is used to expose <em style="box-sizing: border-box;">service actors</em> as remote<em style="box-sizing: border-box;">microservices</em> via <em style="box-sizing: border-box;">REST</em> and <em style="box-sizing: border-box;">WebSocket</em> uses the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundle</code>.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundle</code> is also used to add other forms of services, like service pools, and sharded services.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Let's walk through an example. We will use the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Todo</code> example that we used for <a href="https://github.com/advantageous/qbit/wiki/ServiceQueue" style="box-sizing: border-box; color: #4078c0; text-decoration: none;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code>'s</a>. Since we are covering <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceBundle</code>, we will add another service called <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Auditor</code> and its implementation called <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">AuditorImpl</code>. We will change the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">TodoManagerImpl</code> to use the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Auditor</code>.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Let's review our <em style="box-sizing: border-box;">Todo example</em>. The <em style="box-sizing: border-box;">Todo example</em>, has a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">TodoManagerClient</code> interface.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceBundle#todomanagerclient" id="user-content-todomanagerclient" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>TodoManagerClient</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">com.mammatustech.todo</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">io.advantageous.reakt.promise.Promise</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">java.util.List</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">interface</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoManagerClient</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">add</span>(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">todo</span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">remove</span>(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">id</span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-k" style="box-sizing: border-box;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">list</span>();
}</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
This is the interface we will use to invoke async methods.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
To this we will add a new service called <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Auditor</code>.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceBundle#auditor" id="user-content-auditor" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Auditor</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">com.mammatustech.todo</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">interface</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">Auditor</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">audit</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">operation</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">log</span>);
}</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
We will keep the implementation simple so we can focus on QBit and the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundle</code>.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceBundle#auditorimpl" id="user-content-auditorimpl" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>AuditorImpl</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">com.mammatustech.todo</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">class</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">AuditorImpl</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">implements</span> <span class="pl-e" style="box-sizing: border-box; color: #795da3;">Auditor</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">audit</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">operation</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">log</span>) {
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>out<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>printf(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>operations %s, message %s log<span class="pl-cce" style="box-sizing: border-box;">\n</span><span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
operation, log);
}
}</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Now to mix things up a bit and since we are talking about a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundle</code>, we will pass an<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Auditor</code> instance to the constructor of the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">TodoManagerImpl</code>.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceBundle#auditorimpl-1" id="user-content-auditorimpl-1" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>AuditorImpl</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">com.mammatustech.todo</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">io.advantageous.qbit.annotation.QueueCallback</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">io.advantageous.qbit.annotation.QueueCallbackType</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">io.advantageous.qbit.reactive.Callback</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import static</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">io.advantageous.qbit.service.ServiceProxyUtils.flushServiceProxy</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">class</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoManagerImpl</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Map<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span>, <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span> todoMap <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">TreeMap<></span>();
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Auditor</span> auditor;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoManagerImpl</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Auditor</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">auditor</span>) {
<span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">this</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>auditor <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> auditor;
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">add</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Callback<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">callback</span>,
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">todo</span>) {
todoMap<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>put(todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getId(), todo);
auditor<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>audit(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>add<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>added new todo<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>resolve(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">true</span>);
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">remove</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Callback<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">callback</span>,
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">id</span>) {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span> removed <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> todoMap<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>remove(id);
auditor<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>audit(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>add<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>removed new todo<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>resolve(removed <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">!=</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">null</span>);
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">list</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Callback<<span class="pl-k" style="box-sizing: border-box;">ArrayList<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span>></span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">callback</span>) {
auditor<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>audit(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>list<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>auditor added<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>accept(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">ArrayList<></span>(todoMap<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>values()));
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@QueueCallback</span>({<span class="pl-smi" style="box-sizing: border-box; color: #333333;">QueueCallbackType</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>LIMIT</span>,
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">QueueCallbackType</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>EMPTY</span>,
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">QueueCallbackType</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>IDLE</span>})
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">process</span>() {
flushServiceProxy(auditor);
}
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
}</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Note that the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">add</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">remove</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">list</code> all use the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">auditor</code> instance. Unlike the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code>there is no auto flush feature. This is typically because <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundles</code>s contain many<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code>s. If you wanted to get auto-flush going with a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> in a bundle, then you add the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> to the bundle or you look up the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> from the bundle and then use the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> to create the auto flush client proxy. This is usually not needed as manually flushing at the right time is better for thread hand off performance and IO performance. QBit uses micro-batching to optimize sending operations to other local and remote <em style="box-sizing: border-box;">service actors</em>.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceBundle#queuecallbacks" id="user-content-queuecallbacks" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>QueueCallbacks</h4>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Since the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">TodoManagerImpl</code> is using another <em style="box-sizing: border-box;">service actor</em>, we will flush operations to that actor when the processing queue for the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">TodoManagerImpl</code> is idle, empty or reached its limit.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceBundle#todomanager-using-queuecallbacks" id="user-content-todomanager-using-queuecallbacks" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>TodoManager using QueueCallbacks</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">com.mammatustech.todo</span>;
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">io.advantageous.qbit.annotation.QueueCallback</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">io.advantageous.qbit.annotation.QueueCallbackType</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import static</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">io.advantageous.qbit.service.ServiceProxyUtils.flushServiceProxy</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">class</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoManagerImpl</span> {
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@QueueCallback</span>({<span class="pl-smi" style="box-sizing: border-box; color: #333333;">QueueCallbackType</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>LIMIT</span>,
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">QueueCallbackType</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>EMPTY</span>,
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">QueueCallbackType</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>IDLE</span>})
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">process</span>() {
flushServiceProxy(auditor);
}
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span></pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
You can do this with annotaitons. (You can also do this without using annotations, which will show later.). The above <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">@QueueCallback</code> annotation says if the processing queue is empty (<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">QueueCallbackType.EMPTY</code>, no more requests or events in the queue), or if the request processing queue is idle (<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">QueueCallbackType.IDLE</code>, not busy at all), or if we have hit the queue limit (QueueCallbackType.LIMIT can only happen under heavy load or if you set the limit very low). A queue limit of ten would have ten times less thread handoff time than a queue limit of size 1 (under heavy load). If the auditor were a remote service, having a larger batch size than 1 would save on the cost of the IO operations.</div>
<blockquote style="border-left-color: rgb(221, 221, 221); border-left-style: solid; border-left-width: 4px; box-sizing: border-box; color: #777777; margin: 0px 0px 16px; padding: 0px 15px;">
<div style="box-sizing: border-box;">
You can turn off micro-batching by setting the processing queue to 1.</div>
</blockquote>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Later when we introduce the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Reactor</code> you can set up a reoccurring job that fires every 10ms or 100ms to flush collaborating services like the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">auditor</code>.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
You can use <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">QueueCallbacks</code> with any <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> and with any <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundle</code>.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
There are other QueueCallbacks to get notified with the services has shutdown and when it has started.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceBundle#queuecallback-for-init-and-shutdown" id="user-content-queuecallback-for-init-and-shutdown" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>QueueCallback for init and shutdown</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">class</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoManagerImpl</span> {
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@QueueCallback</span>({<span class="pl-smi" style="box-sizing: border-box; color: #333333;">QueueCallbackType</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>INIT</span>})
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">init</span>() {
auditor<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>audit(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>init<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>init service<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@QueueCallback</span>({<span class="pl-smi" style="box-sizing: border-box; color: #333333;">QueueCallbackType</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>SHUTDOWN</span>})
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">shutdown</span>() {
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">System</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>out<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>println(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>operation shutdown, shutdown service<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
flushServiceProxy(auditor);
}
</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">init</code> operation would get called once when the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> for the <em style="box-sizing: border-box;">microservice actor</em>starts up. The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">shutdown</code> operation would get called once when the when the <em style="box-sizing: border-box;">microservice actor</em>shuts down.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Let's create a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundle</code> and add the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">auditor</code> and <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">todoManager</code> services to it, and run them.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceBundle#using-the-service-bundle-with-the-auditor-and-todomanager-services" id="user-content-using-the-service-bundle-with-the-auditor-and-todomanager-services" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using the service bundle with the auditor and todoManager services</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Object address to the todoManagerImpl service actor. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span> todoAddress <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>todoService<span class="pl-pds" style="box-sizing: border-box;">"</span></span>;
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Object address to the auditorService service actor. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span> auditorAddress <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>auditorService<span class="pl-pds" style="box-sizing: border-box;">"</span></span>;
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Service Bundle */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">ServiceBundle</span> serviceBundle;
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Client service proxy to the todoManager */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">TodoManagerClient</span> client;
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Client service proxy to the auditor. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Auditor</span> auditor;
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Create the serviceBundleBuilder. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">ServiceBundleBuilder</span> serviceBundleBuilder <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> serviceBundleBuilder();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Create the service bundle. */</span>
serviceBundle <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> serviceBundleBuilder<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>build();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Add the AuditorImpl instance to the serviceBundle. */</span>
serviceBundle<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>addServiceObject(auditorAddress, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">AuditorImpl</span>());
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Create a service client proxy for the auditor. */</span>
auditor <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> serviceBundle<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>createLocalProxy(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Auditor</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class, auditorAddress);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Create a todo manager and pass the </span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> client proxy of the auditor to it. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">TodoManagerImpl</span> todoManager <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">TodoManagerImpl</span>(auditor);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Add the todoManager to the serviceBundle.</span>
serviceBundle
.addServiceObject(todoAddress, todoManager);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Create a client proxy to communicate </span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> with the service actor. */</span>
client <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> serviceBundle
.createLocalProxy(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">TodoManagerClient</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class,
todoAddress);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Start the service bundle.</span>
serviceBundle<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>start();</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Above we create the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundleBuilder</code> which can be used to the response and request queue size, types, batch size, and more. Then we create the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundle</code>. Next we add the<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">auditor</code> <em style="box-sizing: border-box;">microservice actor</em> to the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundle</code> under the address specified by<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">auditorAddress</code>. Next we create a <em style="box-sizing: border-box;">service client proxy</em> for the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">auditor</code> <em style="box-sizing: border-box;">microservice actor</em> that we can pass to the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">TodoManagerImpl</code>. We then add the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">TodoManagerImpl</code> to form the<em style="box-sizing: border-box;">microservice actor</em> for the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">TodoManager</code> Service. Next we create a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">client</code> of the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">TodoManager</code>Service to test with. Then we start the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundle</code>.</div>
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; word-wrap: normal;"><code style="background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; word-break: normal; word-wrap: normal;"> To use the `todoManager` service proxy client aka `client`, the code is much like it was before with the `serviceQueue` example except now we will flush (since by default the queue batch size is greater than 1).
</code></pre>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceBundle#using-the-todomanager-microservice-client-proxy" id="user-content-using-the-todomanager-microservice-client-proxy" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using the todoManager microservice client proxy</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> promise <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Promises</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>blockingPromiseBoolean();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Add the todo item.</span>
client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>write<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Write tutorial<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, timer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>time()))
.invokeWithPromise(promise);
flushServiceProxy(client);
assertTrue(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>The call was successful<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, promise<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>success());
assertTrue(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>The return from the add call<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, promise<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get());
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-k" style="box-sizing: border-box;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span>></span> promiseList <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Promises</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>blockingPromiseList(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Get a list of todo items.</span>
client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>list()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>invokeWithPromise(promiseList);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Call flush since this is not an auto-flush. */</span>
flushServiceProxy(client);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// See if the Todo item we created is in the listing.</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span> todoList <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> promiseList<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>stream()
.filter(todo <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getName()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>equals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>write<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">&&</span> todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getDescription()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>equals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Write tutorial<span class="pl-pds" style="box-sizing: border-box;">"</span></span>))<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>collect(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Collectors</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>toList());
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Make sure we found it.</span>
assertEquals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Make sure there is one<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1</span>, todoList<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>size());
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Remove promise</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> removePromise <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span>
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Promises</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>blockingPromiseBoolean();
client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>remove(todoList<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getId())
.invokeWithPromise(removePromise);
flushServiceProxy(client);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-k" style="box-sizing: border-box;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span>></span> promiseList2 <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span>
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Promises</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>blockingPromiseList(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Make sure it is removed.</span>
client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>list()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>invokeWithPromise(promiseList2);
flushServiceProxy(client);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// See if the Todo item we created is removed.</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span> todoList2 <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> promiseList2<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>stream()
.filter(todo <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getName()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>equals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>write<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">&&</span> todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getDescription()
.equals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Write tutorial<span class="pl-pds" style="box-sizing: border-box;">"</span></span>))
.collect(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Collectors</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>toList());
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Make sure we don't find it.</span>
assertEquals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Make sure there is one<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>, todoList2<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>size());
flushServiceProxy(client);
</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
We can also repeat the async example were we executed more than one operation at a time.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceBundle#making-async-calls-and-coordinating-with-promises" id="user-content-making-async-calls-and-coordinating-with-promises" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Making async calls and coordinating with Promises</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-c" style="box-sizing: border-box; color: #969896;">/* A list of promises for things we </span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> want to do all at once. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-k" style="box-sizing: border-box;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span>></span> promises <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">ArrayList<></span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">3</span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">CountDownLatch</span> latch <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">CountDownLatch</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1</span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">AtomicBoolean</span> success <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">AtomicBoolean</span>();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Add a todoItem to the client add method */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span> todo <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>write<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Write tutorial<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
timer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>time());
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> promise
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(todo);
promises<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(promise);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Add two more. */</span>
promises<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>callMom<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Call Mom<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, timer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>time())));
promises<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>callSis<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Call Sister<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, timer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>time())));
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Now async wait for them all to come back. */</span>
<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Promises</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>all(promises)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>then(done <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
success<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>set(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">true</span>);
latch<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>countDown();
})<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>catchError(e <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
success<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>set(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">false</span>);
latch<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>countDown();
});
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Invoke the promises. */</span>
promises<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>forEach(<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Promise</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">::</span>invoke);
flushServiceProxy(client);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** They are all going to come back async. */</span>
latch<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>await();
assertTrue(success<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get());</pre>
</div>
<div style="box-sizing: border-box;">
Please note that you can explicitly flush an <em style="box-sizing: border-box;">client microservice proxy</em>, it will also flush if you go over the <em style="box-sizing: border-box;">limit</em> for the <em style="box-sizing: border-box;">request queue</em>, or you can set the batch size to 1.</div>
</div>
<div class="wiki-footer gollum-markdown-content boxed-group" id="wiki-footer" style="border-radius: 3px; box-sizing: border-box; clear: none; margin: 30px 30px 0px; position: relative;">
<div class="boxed-group-inner wiki-auxiliary-content wiki-writable markdown-body" style="background: rgb(246, 246, 246); border-radius: 3px; border: 1px solid rgb(216, 216, 216); box-shadow: rgba(0, 0, 0, 0.0588235) 0px 1px 2px; box-sizing: border-box; color: #666666; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 13px; line-height: 1.6; margin: 0px -30px; padding: 10px 15px; word-break: break-word; word-wrap: break-word;">
<a class="wiki-edit-link" href="https://github.com/advantageous/qbit/wiki/_Footer/_edit" style="background-color: transparent; box-sizing: border-box; color: #767676; float: right; line-height: 20.8px; margin-top: 0px !important; opacity: 0.2; position: relative; right: -5px; text-decoration: none; transition: opacity 0.2s ease-in-out; z-index: 2;" title="Edit footer"><svg aria-hidden="true" class="octicon octicon-pencil" height="16" version="1.1" viewbox="0 0 14 16" width="14"><path d="M0 12v3h3l8-8-3-3L0 12z m3 2H1V12h1v1h1v1z m10.3-9.3l-1.3 1.3-3-3 1.3-1.3c0.39-0.39 1.02-0.39 1.41 0l1.59 1.59c0.39 0.39 0.39 1.02 0 1.41z"></path></svg></a><ul style="box-sizing: border-box; line-height: 20.8px; margin-bottom: 16px; margin-top: 0px !important; padding-left: 2em;">
<li style="box-sizing: border-box;"><a href="http://advantageous.github.io/qbit/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit Reactive Microservices</a></li>
<li style="box-sizing: border-box;"><a href="http://advantageous.github.io/reakt" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt</a></li>
<li style="box-sizing: border-box;"><a href="http://advantageous.github.io/reakt-guava/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt Guava Bridge</a></li>
</ul>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceBundle#further-reading" id="user-content-further-reading" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Further Reading</h2>
<div style="box-sizing: border-box; line-height: 20.8px; margin-bottom: 16px;">
<a href="http://advantageous.github.io/qbit/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit Website</a> <a href="http://www.mammatustech.com/microservices-architecture" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">What is Microservices Architecture?</a></div>
<div style="box-sizing: border-box; line-height: 20.8px; margin-bottom: 16px;">
<a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit Java Micorservices lib tutorials</a></div>
<div style="box-sizing: border-box; line-height: 20.8px; margin-bottom: 16px;">
The Java microservice lib. QBit is a reactive programming lib for building microservices - JSON, HTTP, WebSocket, and REST. QBit uses reactive programming to build elastic REST, and WebSockets based cloud friendly, web services. SOA evolved for mobile and cloud. ServiceDiscovery, Health, reactive StatService, events, Java idiomatic reactive programming for Microservices.</div>
<div style="box-sizing: border-box; line-height: 20.8px; margin-bottom: 16px;">
<a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Find more tutorial on QBit</a>.</div>
<div style="box-sizing: border-box; line-height: 20.8px; margin-bottom: 16px;">
<a href="http://rick-hightower.blogspot.com/2015/03/reactive-programming-service-discovery.html" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactive Programming</a>, <a href="http://rick-hightower.blogspot.com/2015/03/java-microservices-architecture.html" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Java Microservices</a>, <a href="http://www.linkedin.com/in/rickhigh" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Rick Hightower</a></div>
<div style="box-sizing: border-box; line-height: 20.8px; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">High-speed microservices consulting firm and authors of QBit with lots of experience with Vertx - Mammatus Technology</a></div>
<div style="box-sizing: border-box; line-height: 20.8px; margin-bottom: 16px;">
<a href="http://www.aboutobjects.com/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Highly recommended consulting and training firm who specializes in microservices architecture and mobile development that are already very familiar with QBit and Vertx as well as iOS and Android - About Objects</a></div>
<div style="box-sizing: border-box; line-height: 20.8px; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/java-microservices-architecture" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Java Microservices Architecture</a></div>
<div style="box-sizing: border-box; line-height: 20.8px; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/Microservice-Service-Discovery-with-Consul" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Microservice Service Discovery with Consul</a></div>
<div style="box-sizing: border-box; line-height: 20.8px; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/consul-service-discovery-and-health-for-microservices-architecture-tutorial" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Microservices Service Discovery Tutorial with Consul</a></div>
<div style="box-sizing: border-box; line-height: 20.8px; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/reactive-microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactive Microservices</a></div>
<div style="box-sizing: border-box; line-height: 20.8px; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/high-speed-microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">High Speed Microservices</a></div>
<div style="box-sizing: border-box; line-height: 20.8px; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/java-microservices-consulting" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Java Microservices Consulting</a></div>
<div style="box-sizing: border-box; line-height: 20.8px; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/java-reactive-microservice-training" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Microservices Training</a></div>
<div style="box-sizing: border-box; line-height: 20.8px; margin-bottom: 16px;">
<a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Reactor-tutorial--%7C-reactively-handling-async-calls-with-QBit-Reactive-Microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactive Microservices Tutorial, using the Reactor</a></div>
<div style="box-sizing: border-box; line-height: 20.8px; margin-bottom: 16px;">
<a href="http://restlet.com/blog/2015/09/04/this-week-in-api-land-20/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit is mentioned in the Restlet blog</a></div>
<div>
<br /></div>
</div>
</div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com0tag:blogger.com,1999:blog-6128525281101058933.post-16780158394130556342016-05-12T23:46:00.001-07:002016-05-12T23:46:13.612-07:00Understanding the QBit microservices lib's serviceQueue<div class="markdown-body" style="background-color: white; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 1.6; margin: 0px -30px; padding: 0px 30px; word-break: break-word; word-wrap: break-word;">
<div style="box-sizing: border-box; margin-bottom: 16px;">
QBit is made up of queues. There are request queues, response queues and event queues.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
A <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> is a set of three queues, namely requests (methodCalls), responses and events. The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> turns an ordinary POJO (plain old Java Object) into a <em style="box-sizing: border-box;">Service Actor</em>. A<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> is building block of <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">QBit</code>.</div>
<ul style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> turns a POJO into a <em style="box-sizing: border-box;">Service Actor</em></li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundle</code> groups <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code>s under different addresses, shares a response queue, allows for service pools, serviceSharding, etc.</li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceServer</code> exposes a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundle</code> to <span style="box-sizing: border-box; font-weight: bolder;"><em style="box-sizing: border-box;">REST</em></span> and <span style="box-sizing: border-box; font-weight: bolder;"><em style="box-sizing: border-box;">WebSocket RPC</em></span>.</li>
</ul>
<div style="box-sizing: border-box; margin-bottom: 16px;">
QBit allows you to adapt POJOs to become <em style="box-sizing: border-box;">Service Actors</em>. A <em style="box-sizing: border-box;">Service Actor</em> is a form of an <a href="https://en.wikipedia.org/wiki/Active_object" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">active object</a>. Method calls to a <em style="box-sizing: border-box;">Service Actor</em> are delivered asynchronously, and handled on one thread which can handle tens of millions or more method calls per second. Let's demonstrate by creating a simple POJO and turning it into a <em style="box-sizing: border-box;">Service Actor</em>.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceQueue#associating-pojo-with-servicequeue-to-make-a-service-actor" id="user-content-associating-pojo-with-servicequeue-to-make-a-service-actor" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Associating POJO with serviceQueue to make a service actor</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-smi" style="box-sizing: border-box;">ServiceQueue</span> serviceQueue;
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Create a serviceQueue with a serviceBuilder.</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">ServiceBuilder</span> serviceBuilder <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> serviceBuilder();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Start the serviceQueue.</span>
serviceQueue <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> serviceBuilder
.setServiceObject(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">TodoManagerImpl</span>())
.buildAndStartAll();
</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The above code registers the POJO <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">TodoManagerImpl</code> with a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> by using the method <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBuilder.setServiceObject</code>. The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> is started by the<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">buildAndStartAll</code> method of <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceBuilder</code>.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceQueue</code> is an interface (<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">io.advantageous.qbit.service.ServiceQueue</code>). The<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceQueue</code> is created with a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceBuilder</code>(<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">io.advantageous.qbit.service.ServiceBuilder</code>). You create a <em style="box-sizing: border-box;">Service Actor</em> by associating a POJO with a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code>. You make this association between the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> and your service POJO with the `ServiceBuilder.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Once started the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> can handle method calls on behalf of the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">TodoManagerImpl</code> and recieve events and deliver them to <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">TodoManagerImpl</code>. <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">TodoManagerImpl</code> can sit behind the<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code>. If you only access <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">TodoManagerImpl</code> POJO service from a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> then it will only ever be accessed by one thread. <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">TodoManagerImpl</code> can handle tens of millions of calls per second, and all of those calls will be thread safe. Here is a simple example of a POJO that we will expose as a <em style="box-sizing: border-box;">Service Actor</em>.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceQueue#implementation" id="user-content-implementation" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Implementation</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> <span class="pl-smi" style="box-sizing: border-box;">com.mammatustech.todo</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.reactive.Callback</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">java.util.ArrayList</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">java.util.Map</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">java.util.TreeMap</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">class</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoManagerImpl</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Map<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span>, <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span> todoMap <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">TreeMap<></span>();
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoManagerImpl</span>() {
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">add</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Callback<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">callback</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">todo</span>) {
todoMap<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>put(todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getId(), todo);
callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>resolve(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">true</span>);
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">remove</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Callback<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">callback</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">id</span>) {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span> removed <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> todoMap<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>remove(id);
callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>resolve(removed <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">!=</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">null</span>);
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">list</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Callback<<span class="pl-k" style="box-sizing: border-box;">ArrayList<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span>></span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">callback</span>) {
callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>resolve(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">ArrayList<></span>(todoMap<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>values()));
}
}
</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Notice that this example does not return values, instead it uses the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">callback</code> to send a response back to the client. A call to <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">callback.resolve(someValue)</code> will send that value to the<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">responseQueue</code>. Method calls come in on the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">requestQueue</code>. The responses go out on the the<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">responseQueue</code>. Let's explore this concept.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> has the following interface.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceQueue#partial-code-listing-of-servicequeue-showing-queues" id="user-content-partial-code-listing-of-servicequeue-showing-queues" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Partial code listing of serviceQueue showing queues</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Manages a service that sits behind a queue.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * created by Richard on 7/21/14.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * @author rhightower</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">interface</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">ServiceQueue</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">extends</span> ... {
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
<span class="pl-smi" style="box-sizing: border-box;">Object</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">service</span>();
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">SendQueue<<span class="pl-k" style="box-sizing: border-box;">MethodCall<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Object</span>></span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">requests</span>();
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">SendQueue<<span class="pl-k" style="box-sizing: border-box;">Event<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Object</span>></span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">events</span>();
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">ReceiveQueue<<span class="pl-k" style="box-sizing: border-box;">Response<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Object</span>></span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">responses</span>();
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span></pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
These methods are not typically accessed. They are for integration and internal usage but they can help you understand <em style="box-sizing: border-box;">QBit microservices</em> a bit better.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
You can access the POJO that the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> is wrapping with <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">service()</code>. You can send method calls directly to the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> by using the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">requests()</code> method to get a<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">sendQueue</code> (<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">SendQueue<MethodCall<Object>></code>). You can send events directly to the<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> by using the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">events()</code> method to get a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">sendQueue</code>. Note that the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">sendQueue</code>you receive will not be thread safe (they implement <a href="http://tutorials.jenkov.com/java-performance/micro-batching.html" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">micro-batching</a>), so each thread will need to get its own copy of an event or <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">methodCall</code> (request) <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">sendQueue</code>. A <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">sendQueue</code> is the client's view of the queue.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
On the receiver side (service side) <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">events</code> and <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">methodCalls</code> queues are handled by the same thread so that all events and <em style="box-sizing: border-box;">methodCall</em>s go to the POJO (e.g., <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">TodoManagerImpl</code>) on the same thread. This is what makes that POJO a <span style="box-sizing: border-box; font-weight: bolder;">Service Actor</span> (active object).</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Typically to make calls to a <em style="box-sizing: border-box;">Service Actor</em>, you use a <em style="box-sizing: border-box;">service client proxy</em>, which is just an interface. The <em style="box-sizing: border-box;">service client proxy</em> can return <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Promise</code>s or take a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Callback</code> as the first or last argument of the method. A <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">promise</code> is a deferred result that you can handle asynchronously. The<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Promise</code> interface is similar to <a href="http://advantageous.github.io/reakt/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">ES6 promises</a>.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceQueue#service-client-proxy-interface" id="user-content-service-client-proxy-interface" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Service Client Proxy interface</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> <span class="pl-smi" style="box-sizing: border-box;">com.mammatustech.todo</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.reakt.promise.Promise</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">java.util.List</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">interface</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoManagerClient</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">add</span>(<span class="pl-smi" style="box-sizing: border-box;">Todo</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">todo</span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">remove</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">id</span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-k" style="box-sizing: border-box;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">list</span>();
}</pre>
</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceQueue#todo-pojo-to-store-a-todo-item" id="user-content-todo-pojo-to-store-a-todo-item" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Todo POJO to store a Todo item</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> <span class="pl-smi" style="box-sizing: border-box;">com.mammatustech.todo</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">class</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">Todo</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">String</span> name;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">String</span> description;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">long</span> createTime;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-smi" style="box-sizing: border-box;">String</span> id;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">Todo</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">name</span>, <span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">description</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">long</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">createTime</span>) {
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
}
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//normal getters, equals, hashCode </span>
}
</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
To create an use a <em style="box-sizing: border-box;">service client proxy</em> you use the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code>.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceQueue#creating-and-using-a-service-client-proxy" id="user-content-creating-and-using-a-service-client-proxy" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Creating and using a <em style="box-sizing: border-box;">service client proxy</em></h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-smi" style="box-sizing: border-box;">TodoManagerClient</span> client;
<span class="pl-smi" style="box-sizing: border-box;">ServiceQueue</span> serviceQueue;
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Create a client proxy to communicate with the service actor.</span>
client <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> serviceQueue
.createProxyWithAutoFlush(<span class="pl-smi" style="box-sizing: border-box;">TodoManagerClient</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class,
<span class="pl-smi" style="box-sizing: border-box;">Duration</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>milliseconds(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">5</span>));
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Add an item</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> promise <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">Promises</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>blockingPromiseBoolean();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Add the todo item.</span>
client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>write<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Write tutorial<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, timer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>time()))
.invokeWithPromise(promise);
assertTrue(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>The call was successful<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, promise<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>success());
assertTrue(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>The return from the add call<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, promise<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get());
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Get a list of items</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-k" style="box-sizing: border-box;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span>></span> promiseList <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">Promises</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>blockingPromiseList(<span class="pl-smi" style="box-sizing: border-box;">Todo</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Get a list of todo items.</span>
client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>list()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>invokeWithPromise(promiseList);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// See if the Todo item we created is in the listing.</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span> todoList <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span>
promiseList<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>stream()<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Remove an item</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Remove promise</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> removePromise <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span>
<span class="pl-smi" style="box-sizing: border-box;">Promises</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>blockingPromiseBoolean();
client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>remove(todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getId())
.invokeWithPromise(removePromise);
</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Note <em style="box-sizing: border-box;">Blocking Promises</em> are great for testing and integration but not something you typically use in your reactive microserivce (sot of defeats the whole purpose).</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Here is a simple unit test showing what we have done and talked about so far, after this let's show a non-blocking example and some call coordination.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceQueue#unit-test-to-show-it-is-working" id="user-content-unit-test-to-show-it-is-working" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Unit test to show it is working</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">package</span> <span class="pl-smi" style="box-sizing: border-box;">com.mammatustech.todo</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.service.ServiceBuilder</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.service.ServiceQueue</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.time.Duration</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.util.Timer</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.reakt.promise.Promise</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.reakt.promise.Promises</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">org.junit.After</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">org.junit.Before</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">org.junit.Test</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">java.util.List</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">java.util.stream.Collectors</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import static</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.service.ServiceBuilder.serviceBuilder</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import static</span> <span class="pl-smi" style="box-sizing: border-box;">org.junit.Assert.assertEquals</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import static</span> <span class="pl-smi" style="box-sizing: border-box;">org.junit.Assert.assertTrue</span>;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">class</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">TodoManagerImplTest</span> {
<span class="pl-smi" style="box-sizing: border-box;">TodoManagerClient</span> client;
<span class="pl-smi" style="box-sizing: border-box;">ServiceQueue</span> serviceQueue;
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">Timer</span> timer <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">Timer</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>timer();
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@Before</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">setup</span>() {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Create a serviceQueue with a serviceBuilder.</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">ServiceBuilder</span> serviceBuilder <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> serviceBuilder();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Start the serviceQueue.</span>
serviceQueue <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> serviceBuilder
.setServiceObject(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">TodoManagerImpl</span>())
.buildAndStartAll();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Create a client proxy to communicate with the service actor.</span>
client <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> serviceQueue<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>createProxyWithAutoFlush(<span class="pl-smi" style="box-sizing: border-box;">TodoManagerClient</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class, <span class="pl-smi" style="box-sizing: border-box;">Duration</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>milliseconds(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">5</span>));
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@Test</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">test</span>() <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">throws</span> <span class="pl-smi" style="box-sizing: border-box;">Exception</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> promise <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">Promises</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>blockingPromiseBoolean();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Add the todo item.</span>
client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>write<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Write tutorial<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, timer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>time()))
.invokeWithPromise(promise);
assertTrue(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>The call was successful<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, promise<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>success());
assertTrue(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>The return from the add call<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, promise<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get());
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-k" style="box-sizing: border-box;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span>></span> promiseList <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">Promises</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>blockingPromiseList(<span class="pl-smi" style="box-sizing: border-box;">Todo</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Get a list of todo items.</span>
client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>list()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>invokeWithPromise(promiseList);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// See if the Todo item we created is in the listing.</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span> todoList <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> promiseList<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>stream()
.filter(todo <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getName()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>equals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>write<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">&&</span> todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getDescription()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>equals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Write tutorial<span class="pl-pds" style="box-sizing: border-box;">"</span></span>))<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>collect(<span class="pl-smi" style="box-sizing: border-box;">Collectors</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>toList());
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Make sure we found it.</span>
assertEquals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Make sure there is one<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1</span>, todoList<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>size());
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Remove promise</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> removePromise <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">Promises</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>blockingPromiseBoolean();
client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>remove(todoList<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getId())<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>invokeWithPromise(removePromise);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-k" style="box-sizing: border-box;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span>></span> promiseList2 <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">Promises</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>blockingPromiseList(<span class="pl-smi" style="box-sizing: border-box;">Todo</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Make sure it is removed.</span>
client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>list()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>invokeWithPromise(promiseList2);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// See if the Todo item we created is removed.</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span> todoList2 <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> promiseList2<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>stream()
.filter(todo <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getName()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>equals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>write<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">&&</span> todo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getDescription()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>equals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Write tutorial<span class="pl-pds" style="box-sizing: border-box;">"</span></span>))<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>collect(<span class="pl-smi" style="box-sizing: border-box;">Collectors</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>toList());
<span class="pl-c" style="box-sizing: border-box; color: #969896;">// Make sure we don't find it.</span>
assertEquals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Make sure there is one<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>, todoList2<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>size());
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">@After</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">tearDown</span>() {
serviceQueue<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>stop();
}
}</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
You can find this source code at this <a href="https://github.com/MammatusTech/qbit-microservices-examples/tree/master/serviceQueue" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">github repo</a>.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Here is a build file for the example so you can see the dependencies.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceQueue#build-file-buildgradle" id="user-content-build-file-buildgradle" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Build file build.gradle</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">group <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>qbit-ex<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
version <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>1.0-SNAPSHOT<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
apply plugin<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>java<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
apply plugin<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>application<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
mainClassName <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>com.mammatustech.todo.TodoServiceMain<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
compileJava {
sourceCompatibility <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1.8</span>
}
repositories {
mavenCentral()
mavenLocal()
}
dependencies {
testCompile group<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>junit<span class="pl-pds" style="box-sizing: border-box;">'</span></span>, name<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>junit<span class="pl-pds" style="box-sizing: border-box;">'</span></span>, version<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>4.11<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
compile <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>io.advantageous.qbit:qbit-vertx:1.8.3<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
compile <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>io.advantageous.qbit:qbit-admin:1.8.3<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
}
</pre>
</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceQueue#executing-a-bunch-of-methods-at-once" id="user-content-executing-a-bunch-of-methods-at-once" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Executing a bunch of methods at once</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
We can execute a bunch of methods at once and use <a href="https://github.com/advantageous/reakt/wiki/Promises.all" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Promises.all</code></a> to do the next item when they all succeed or <a href="https://github.com/advantageous/reakt/wiki/Promises.any" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Promises.any</code></a> to something when any of them succeed.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceQueue#executing-many-methods-on-a-service-proxy-at-once" id="user-content-executing-many-methods-on-a-service-proxy-at-once" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Executing many methods on a service proxy at once</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> @<span class="pl-smi" style="box-sizing: border-box;">Test</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> testUsingAll() throws <span class="pl-smi" style="box-sizing: border-box;">Exception</span> {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* A list of promises for things we want to do all at once. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-k" style="box-sizing: border-box;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span>></span> promises <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">ArrayList<></span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">3</span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">CountDownLatch</span> latch <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">CountDownLatch</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1</span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">AtomicBoolean</span> success <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">AtomicBoolean</span>();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Add a todoItem to the client add method */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span> todo <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>write<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Write tutorial<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, timer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>time());
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> promise
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(todo);
promises<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(promise);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Add two more. */</span>
promises<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>callMom<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Call Mom<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, timer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>time())));
promises<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(client<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>add(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">Todo</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>callSis<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Call Sister<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, timer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>time())));
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Now async wait for them all to come back. */</span>
<span class="pl-smi" style="box-sizing: border-box;">Promises</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>all(promises)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>then(done <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
success<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>set(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">true</span>);
latch<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>countDown();
})<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>catchError(e<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
success<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>set(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">false</span>);
latch<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>countDown();
});
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Invoke the promises. */</span>
promises<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>forEach(<span class="pl-smi" style="box-sizing: border-box;">Promise</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">::</span>invoke);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** They are all going to come back async. */</span>
latch<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>await();
assertTrue(success<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get());
}</pre>
</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceQueue#thread-model" id="user-content-thread-model" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Thread model</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> can be started and stopped. There are several options to start a<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code>. You can start it with two threads, one thread for response handling and another thread for request/event handling (<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">startAll()</code>). You can start the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> with just the request/event handling thread (<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">start()</code>). You can also start it with one thread managing request/event and responses. Caution must be exercised with the last way since if a callback or promise blocks then your <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> will be blocked. Typically you use <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">startAll</code> or you use a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceBundle</code> where one response queue is shared with many <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code>s. The<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> was meant to be composable so you can access the queues and provide your own thread model if needed or desired.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceQueue#exception-handling" id="user-content-exception-handling" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Exception Handling</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Typically you handle a exception from a <em style="box-sizing: border-box;">Service Actor</em> by calling <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">callback.reject(exception)</code>to pass the exception downstream to the client or you <em style="box-sizing: border-box;">catch</em> it and handle it in whatever way makes sense. If you do not catch an exception then the thread for your <em style="box-sizing: border-box;">Service Actor</em> will terminate. However, QBit will log the exception that you did not handle and restart a new thread to manage your <em style="box-sizing: border-box;">Service Actor</em>.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceQueue#handling-calls-to-other-service-actor" id="user-content-handling-calls-to-other-service-actor" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Handling calls to other <em style="box-sizing: border-box;">Service Actor</em></h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
In the QBit microservice lib it is common to call other async services, remote <em style="box-sizing: border-box;">Service Actors</em>, REST services, and async NoSQL database drivers. If you <em style="box-sizing: border-box;">Service Actor</em> is stateful (which is common with <a href="http://www.mammatustech.com/high-speed-microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">high-speed services</a>), then you will want to do use a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Reactor</code>. There is the<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Reactor</code> that comes with QBit which is EOL (since we are replacing it with the one we wrote for<a href="http://advantageous.github.io/reakt/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt</a>), and then there is the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Reactor</code> that comes <a href="http://advantageous.github.io/reakt/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt</a>. The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceQueue</code> allows events/method calls to all come to the <em style="box-sizing: border-box;">Service Actor</em> on one thread. The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">reactor</code> is a way to also allow method call callbacks to happen on the same thread, and since the callbacks happen on the same thread as the <em style="box-sizing: border-box;">Service Actor</em> access to the <em style="box-sizing: border-box;">Service Actors</em> data (fields, collaborating objects, etc.) are also thread safe. You only need to use a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Reactor</code> if you want to handle callback on the same thread as the <em style="box-sizing: border-box;">Service Actor</em>, which is not always needed. You can also use the<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Reactor</code> to handle streaming data on the same thread as the <em style="box-sizing: border-box;">Service Actor</em>. The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Reactor</code> can also be used for scheduling async tasks or just scheduling a task to be run on the <em style="box-sizing: border-box;">Service Actor</em>as soon as possible.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceQueue#getting-notified-when-you-start-stop-etc" id="user-content-getting-notified-when-you-start-stop-etc" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Getting notified when you start, stop, etc.</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
You can get notified of different <em style="box-sizing: border-box;">Service Actor</em> lifecycle events like started, stopped, when the micro batch limit was met, when the request queue is empty, and more. These lifecycle events allow you to do thing in batches and thus effectively pass data from one service to another (both remote and local). The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">reactor</code> for example has a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">process</code> method that is usually called when the request queue has reached a limit or is empty. There are two ways to do this. You can use a<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">QueueCallbackHandler</code> with a ServiceBuilder (or ServiceBundle) or you can use the annotation<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">@QueueCallback</code>.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceQueue#admin-package" id="user-content-admin-package" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Admin package</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The Admin package adds Consul discovery, and StatsD support to QBit microservices, and provides a simplified builder for creating a set of managed services which you can easily expose via <em style="box-sizing: border-box;">REST</em> or <em style="box-sizing: border-box;">WebSocket RPC</em>.</div>
<blockquote style="border-left-color: rgb(221, 221, 221); border-left-style: solid; border-left-width: 4px; box-sizing: border-box; color: #777777; margin: 0px 0px 16px; padding: 0px 15px;">
<div style="box-sizing: border-box;">
It is quite easy to build bridges into the QBit world and we have done so via Kafka, Vert.x event bus and even JMS. QBit was meant to be composeable so you can pick your messaging platform and plug QBit into it.</div>
</blockquote>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Two main packages of note in the QBit admin packages are the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ManagedServiceBuilder</code> and the<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceManagementBundle</code>. The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ManagedServiceBuilder</code> gives you access to building a group of services and then easily wiring them to the same health monitor, discovery system and metrics/stats system. Whilst the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceManagementBundle</code> allows services to interact with common QBit services like stats, health and discovery.</div>
<div style="box-sizing: border-box;">
Let's show some simple examples using these that we will continue on in our discussion of the<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceBundle</code> and the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ServiceEndpointServer</code>.</div>
</div>
<div class="wiki-footer gollum-markdown-content boxed-group" id="wiki-footer" style="background-color: white; border-radius: 3px; box-sizing: border-box; clear: none; color: #333333; font-family: Helvetica, arial, nimbussansl, liberationsans, freesans, clean, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 13px; line-height: 18.2px; margin: 30px 30px 0px; position: relative;">
<div class="boxed-group-inner wiki-auxiliary-content wiki-writable markdown-body" style="background: rgb(246, 246, 246); border-radius: 3px; border: 1px solid rgb(216, 216, 216); box-shadow: rgba(0, 0, 0, 0.0588235) 0px 1px 2px; box-sizing: border-box; color: #666666; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; line-height: 1.6; margin: 0px -30px; padding: 10px 15px; word-break: break-word; word-wrap: break-word;">
<a class="wiki-edit-link" href="https://github.com/advantageous/qbit/wiki/_Footer/_edit" style="background-color: transparent; box-sizing: border-box; color: #767676; float: right; margin-top: 0px !important; opacity: 0.2; position: relative; right: -5px; text-decoration: none; transition: opacity 0.2s ease-in-out; z-index: 2;" title="Edit footer"><svg aria-hidden="true" class="octicon octicon-pencil" height="16" version="1.1" viewbox="0 0 14 16" width="14"><path d="M0 12v3h3l8-8-3-3L0 12z m3 2H1V12h1v1h1v1z m10.3-9.3l-1.3 1.3-3-3 1.3-1.3c0.39-0.39 1.02-0.39 1.41 0l1.59 1.59c0.39 0.39 0.39 1.02 0 1.41z"></path></svg></a><ul style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px !important; padding-left: 2em;">
<li style="box-sizing: border-box;"><a href="http://advantageous.github.io/qbit/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit Reactive Microservices</a></li>
<li style="box-sizing: border-box;"><a href="http://advantageous.github.io/reakt" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt</a></li>
<li style="box-sizing: border-box;"><a href="http://advantageous.github.io/reakt-guava/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt Guava Bridge</a></li>
</ul>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/qbit/wiki/ServiceQueue#further-reading" id="user-content-further-reading" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Further Reading</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://advantageous.github.io/qbit/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit Website</a> <a href="http://www.mammatustech.com/microservices-architecture" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">What is Microservices Architecture?</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit Java Micorservices lib tutorials</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The Java microservice lib. QBit is a reactive programming lib for building microservices - JSON, HTTP, WebSocket, and REST. QBit uses reactive programming to build elastic REST, and WebSockets based cloud friendly, web services. SOA evolved for mobile and cloud. ServiceDiscovery, Health, reactive StatService, events, Java idiomatic reactive programming for Microservices.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Find more tutorial on QBit</a>.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://rick-hightower.blogspot.com/2015/03/reactive-programming-service-discovery.html" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactive Programming</a>, <a href="http://rick-hightower.blogspot.com/2015/03/java-microservices-architecture.html" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Java Microservices</a>, <a href="http://www.linkedin.com/in/rickhigh" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Rick Hightower</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">High-speed microservices consulting firm and authors of QBit with lots of experience with Vertx - Mammatus Technology</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.aboutobjects.com/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Highly recommended consulting and training firm who specializes in microservices architecture and mobile development that are already very familiar with QBit and Vertx as well as iOS and Android - About Objects</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/java-microservices-architecture" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Java Microservices Architecture</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/Microservice-Service-Discovery-with-Consul" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Microservice Service Discovery with Consul</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/consul-service-discovery-and-health-for-microservices-architecture-tutorial" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Microservices Service Discovery Tutorial with Consul</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/reactive-microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactive Microservices</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/high-speed-microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">High Speed Microservices</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/java-microservices-consulting" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Java Microservices Consulting</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/java-reactive-microservice-training" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Microservices Training</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Reactor-tutorial--%7C-reactively-handling-async-calls-with-QBit-Reactive-Microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactive Microservices Tutorial, using the Reactor</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://restlet.com/blog/2015/09/04/this-week-in-api-land-20/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit is mentioned in the Restlet blog</a></div>
<div style="box-sizing: border-box;">
<a href="https://www.jetbrains.com/idea/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">All code is written using JetBrains Idea - the best IDE ever!</a></div>
</div>
</div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com0tag:blogger.com,1999:blog-6128525281101058933.post-1720500452564840132016-05-05T21:28:00.001-07:002016-05-05T21:48:43.194-07:00Reactive Java code examples using Reakt (with links to docs for further info)<div class="markdown-body" style="background-color: white; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 1.6; margin: 0px -30px; padding: 0px 30px; word-break: break-word; word-wrap: break-word;">
<div style="box-sizing: border-box; margin-bottom: 16px;">
This is a presentation of different Reakt features in the context of a real application. I have renamed the classnames and such, but this is from an in-progress microservice. It demonstrates where you would use the Reakt pieces to build a reactive Java application.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
This covers usage of Reakt:</div>
<ul style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">blocking promises</li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Promises.all</code></li>
<li style="box-sizing: border-box;">invokable promise</li>
<li style="box-sizing: border-box;">Expected values</li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Circuit</code> breakers</li>
<li style="box-sizing: border-box;">Working with the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Reactor</code></li>
<li style="box-sizing: border-box;">Working with streams</li>
<li style="box-sizing: border-box;">Using <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">AsyncSupplier</code>s to create downstream services</li>
<li style="box-sizing: border-box;">Reakt Guava integration</li>
<li style="box-sizing: border-box;">Using <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">promise.thenMap</code></li>
</ul>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#blocking-promise-example" id="user-content-blocking-promise-example" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Blocking promise example</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Let's say you have an async microservices application, and you want to write some unit and integration tests. You want the tests to wait until the system starts up. Rather you want the system to notify the test when it is done loading.</div>
<blockquote style="border-left-color: rgb(221, 221, 221); border-left-style: solid; border-left-width: 4px; box-sizing: border-box; color: #777777; margin: 0px 0px 16px; padding: 0px 15px;">
<div style="box-sizing: border-box;">
NOTE: The examples below are written in Kotlin which is a JVM language from Idea. I use Kotlin because the example are easier to read and take up less space. If you know Java 8, you should follow no problem. Think of it like pseudo code. Kotlin works well with Java classes and such.</div>
</blockquote>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#loading-a-system-for-testing" id="user-content-loading-a-system-for-testing" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Loading a system for testing.</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.qbit.util.PortUtils</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.reakt.AsyncSupplier</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.reakt.promise.Promises</span>
object <span class="pl-smi" style="box-sizing: border-box;">DFTestUtils</span> {
val adminPort <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-smi" style="box-sizing: border-box;">AtomicInteger</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> AtomicInteger(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">9090</span>)
val eventBusPort <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-smi" style="box-sizing: border-box;">AtomicInteger</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> AtomicInteger(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">8080</span>)
fun loadSystem()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-smi" style="box-sizing: border-box;">ServicePlatform</span> {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Load System. */</span>
val loadPromise <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">Promises</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>blockingPromiseNotify(<span class="pl-smi" style="box-sizing: border-box;">Duration</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>ofSeconds(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">4</span>))
val servicePlatform <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> servicePlatform()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>withNamespace(<span class="pl-smi" style="box-sizing: border-box;">Constants</span><span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>TODO_SERVICE</span>)
.setWaitForStartup(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">true</span>)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>setAdminPort(<span class="pl-smi" style="box-sizing: border-box;">PortUtils</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>findOpenPortStartAt(adminPort<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>andIncrement))
.setEventBusPort(<span class="pl-smi" style="box-sizing: border-box;">PortUtils</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>findOpenPortStartAt(eventBusPort<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>andIncrement))
<span class="pl-smi" style="box-sizing: border-box;">Main</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>run(servicePlatform)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>invokeWithPromise(loadPromise)
loadPromise<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get() <span class="pl-c" style="box-sizing: border-box; color: #969896;">//Wait for the system to load before we start.</span>
<span class="pl-smi" style="box-sizing: border-box;">Assert</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>assertFalse(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>system loaded<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, loadPromise<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>failure())
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> servicePlatform
}
</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Notice that we create a promising using <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Promises.blockingPromiseNotify(Duration.ofSeconds(4))</code>. We call <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">loadPromise.get()</code> to wait until the system loads. <a href="https://github.com/advantageous/reakt/wiki/BlockingPromise" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Blocking promises</code></a> are good for testing.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Now we can use the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">loadSystem</code> in our tests. Here is a test that does a health check against a running server.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#using-loadsystem--blocking-promise-in-our-test" id="user-content-using-loadsystem--blocking-promise-in-our-test" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using loadSystem / blocking promise in our test</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> @<span class="pl-smi" style="box-sizing: border-box;">Test</span>
@Throws(<span class="pl-smi" style="box-sizing: border-box;">Exception</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">::</span>class)
fun mainHealthCheckTest() {
val servicePlatform <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> loadSystem()
val httpTextResponse <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">HttpClientBuilder</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>httpClientBuilder()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>setPort(servicePlatform<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>adminPort)
.buildAndStart()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/__admin/ok<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
assertNotNull(httpTextResponse)
assertEquals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>true<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, httpTextResponse<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>body())
assertEquals(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">200</span>, httpTextResponse<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>code()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>toLong())
shutdownSystem(servicePlatform)
}</pre>
</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#using-invokable-promises-to-notify-when-the-entire-system-is-done-loading" id="user-content-using-invokable-promises-to-notify-when-the-entire-system-is-done-loading" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using invokable promises to notify when the entire system is done loading</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The actual code that loads our system uses an <a href="https://github.com/advantageous/reakt/wiki/Invokable-Promise" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">invokable promise</a>.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#load-system-calls-mainrun-which-returns-an-invokable-promise" id="user-content-load-system-calls-mainrun-which-returns-an-invokable-promise" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Load system calls Main.run which returns an invokable promise</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.reakt.AsyncSupplier</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.reakt.Callback</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.reakt.Stream</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.reakt.promise.Promise</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.reakt.promise.Promises.*</span>
object <span class="pl-smi" style="box-sizing: border-box;">Main</span> {
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
var repoServiceQueue<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-smi" style="box-sizing: border-box;">ServiceQueue</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">?</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">null</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> val logger <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">LoggerFactory</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getLogger(<span class="pl-smi" style="box-sizing: border-box;">Main</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">::</span>class<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>java)
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
fun main(args<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Array<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span>></span>) {
run(servicePlatform()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>withNamespace(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">TODO_SERVICE</span>))<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>invoke()
}
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
fun run(servicePlatform<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-smi" style="box-sizing: border-box;">ServicePlatform</span>)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Void</span>></span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> invokablePromise { donePromise <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span>
val loadCollectionServicePromise <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> promiseBoolean()
val loadTodoServicePromise <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> promiseBoolean()
val loadPromise <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> all(loadCollectionServicePromise,
loadTodoServicePromise)
.thenPromise(donePromise)
createCollectionService(servicePlatform, loadPromise)
createTodoServiceService(servicePlatform, loadPromise)
servicePlatform<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>start()
}
}</pre>
</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#working-with-promisesall" id="user-content-working-with-promisesall" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Working with Promises.all</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
There is a lot going on here. The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">run</code> method is using <a href="https://www.blogger.com/null" style="background-color: transparent; box-sizing: border-box; color: inherit;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Promises.invokablePromise</code></a>. Then the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">run</code> method uses <a href="https://github.com/advantageous/reakt/wiki/Promises.all" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Promises.all</code></a> to chain the promise <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">loadCollectionServicePromise</code> and <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">loadTodoServicePromise</code> together. The method <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Promises.all</code> is used when you want all of the promises to trigger then the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">all</code> promise triggers. This way you are being notified when both the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">createCollectionService</code> and the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">createTodoServiceService</code> async reply. You don't want to start testing before the system is initialized.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#working-with-reakt-streams" id="user-content-working-with-reakt-streams" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Working with Reakt Streams</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The project that I am working on uses leader election from elekt. <a href="https://github.com/advantageous/elekt" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Elekt, leadership API lib,</a> uses <a href="https://github.com/advantageous/reakt/wiki/Stream" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt streams</a>. For testing, we just simulate the <a href="http://advantageous.github.io/elekt-consul/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Elekt consul support</a>.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The LeaderElector looks like this:</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#leaderelector-uses-reakt-streams-support" id="user-content-leaderelector-uses-reakt-streams-support" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>LeaderElector uses Reakt streams support</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">interface</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">LeaderElector</span> {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Attempt to elect this service as leader.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Returns true if successful, and false if not successful.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * @param endpoint endpoint describes the host and port of the leader.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * @param callback callback</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">selfElect</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">Endpoint</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">endpoint</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Callback<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">callback</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * This will send leadership changes as the occur over the stream.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * @param callback callback returns new leader.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">leadershipChangeNotice</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Stream<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Endpoint</span>></span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">callback</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * This will come back quickly with a new Leader.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * If no Endpoint is returned in the callback then there is no leader.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * @param callback callback returns new leader.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getLeader</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Callback<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Endpoint</span>></span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">callback</span>);
}
</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
To simulate that for integration testing, we use this mock <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">LeaderElector</code>.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#using-a-test-leaderelector-stream-for-integration-testing" id="user-content-using-a-test-leaderelector-stream-for-integration-testing" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using a test LeaderElector stream for integration testing</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> fun createLeaderElector()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Supplier<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">LeaderElector</span>></span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> <span class="pl-smi" style="box-sizing: border-box;">Supplier</span> {
object <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-smi" style="box-sizing: border-box;">LeaderElector</span> {
override fun leadershipChangeNotice(stream<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Stream<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Endpoint</span>></span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">?</span>) {
logger<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>info(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Leader notice registered<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
val idOfServer <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> Identity()
stream<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">?</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>reply(Endpoint(idOfServer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>host, idOfServer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>servicePort))
Thread({
<span class="pl-smi" style="box-sizing: border-box;">Thread</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>currentThread()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>isDaemon <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">true</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">while</span> (<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">true</span>) {
<span class="pl-smi" style="box-sizing: border-box;">Thread</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>sleep(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1000</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">*</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">10</span>)
stream<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">?</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>reply(Endpoint(idOfServer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>host, idOfServer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>servicePort))
}
})
}
override fun selfElect(endpoint<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-smi" style="box-sizing: border-box;">Endpoint</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">?</span>, callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Callback<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">?</span>) {
logger<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>info(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Self elect was called<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">?</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>resolve(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">true</span>);
}
override fun getLeader(callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Callback<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Endpoint</span>></span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">?</span>) {
logger<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>info(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Self elect was called<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
val idOfServer <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> Identity()
callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">?</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>resolve(Endpoint(idOfServer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>host, idOfServer<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>servicePort))
}
}
}
}</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Notice the call to <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">stream.reply</code> to send a stream of leader elect notifications that this server has been elected the leader.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#reakt-expected-and-circuit-breakers" id="user-content-reakt-expected-and-circuit-breakers" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Reakt Expected and Circuit breakers</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
It is important to monitor the health of your system, and sometimes it is good not to beat a dead horse. If downstream services are broken there is no point in using them until the are fixed. In Reakt we use Circuit Breakers and Expected to handle when some service is support to be there or some value is expected.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Let's demonstrate this with <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">MetricsCollectionServiceImpl</code>.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#metricscollectionserviceimpl" id="user-content-metricscollectionserviceimpl" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>MetricsCollectionServiceImpl</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.reakt.Breaker</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.reakt.Breaker.*</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.reakt.Expected</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">java.util.function.Function</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">java.util.function.Supplier</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Metrics Collection Service.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Manages finding leader with LeaderElector.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">class</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">MetricsCollectionServiceImpl</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * @param mgmt ServiceManagementBundle (from QBit)</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * @param leaderElectorSupplier leaderElectorSupplier for connecting to the leader elector.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * @param todoListServiceSupplier todoListServiceSupplier for connecting</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * * to the todo service.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * @param metricRepository metric repo for storing metrics</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
(
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Service management bundle which includes stats collection, Reakt reactor, QBit health management, and more.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> val mgmt: ServiceManagementBundle,
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Supplies a leader elector interface. LeaderElector is from Elekt.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> val leaderElectorSupplier: Supplier<LeaderElector>,
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * This is used to create a supplier.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> val todoListServiceSupplier: Function<Endpoint, TodoServiceClient>,
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Metric repository for saving repositories.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> val metricRepository: MetricRepositoryClient)
: MetricsCollectionService {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * The current leaderEndpoint which starts out empty.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> var leaderEndpoint <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">Expected</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>empty<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;"><</span><span class="pl-smi" style="box-sizing: border-box;">Endpoint</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span>()
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * The actual todoService wrapped in a Reakt Circuit breaker.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> var todoService <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">Breaker</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>opened<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;"><</span><span class="pl-smi" style="box-sizing: border-box;">TodoServiceClient</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span>()
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * The leader elector we are using, wrapped in Reakt Circuit breaker</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> var leaderElectorBreaker<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Breaker<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">LeaderElector</span>></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">Breaker</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>opened()
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Call count per second.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> var callCount<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-smi" style="box-sizing: border-box;">Long</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>
init {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/*</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Check circuit breaker health every 10 seconds.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>reactor()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>addRepeatingTask(seconds(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">10</span>)) {
healthCheck()
}
createLeaderElector()
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>reactor()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>addRepeatingTask(seconds(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1</span>)) { throughPut() }
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>reactor()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>addRepeatingTask(millis(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">100</span>), { flushService(metricRepository) })
}</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Notice that we are using Reakt <a href="https://github.com/advantageous/reakt/wiki/Breaker" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">circuit breakers</a>. Notice we are using Reakt reactor's <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Reactor.addRepeatingTask</code> to periodically check the health of our repo. <a href="https://github.com/advantageous/reakt/wiki/Reactor" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt's Reactor</a> is used to manage callbacks so they execute on this thread, callback timeouts, and repeating tasks.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Let's look at the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">healthCheck</code> that runs every 10 seconds to see how circuit breakers work.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#healthcheck-that-runs-every-10-seconds-via-a-reakt-reactor-task" id="user-content-healthcheck-that-runs-every-10-seconds-via-a-reakt-reactor-task" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>healthCheck that runs every 10 seconds via a Reakt Reactor task</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> fun healthCheck() {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">if</span> (mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>isFailing) {
logger<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>warn(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>CollectionService Health is suspect<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
} <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">else</span> {
logger<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>debug(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>CollectionService is Healthy<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
}
leaderElectorBreaker<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>ifBroken {
createLeaderElector()
}
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* If the TODO service is broken, i.e. the circuit is open then do... */</span>
todoService<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>ifBroken {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Check to see if we have a leaderEndpoint. */</span>
leaderEndpoint
.ifPresent { leaderEndpoint <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span>
<span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">this</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>handleNewLeader(leaderEndpoint)
}
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* If we don't have a leaderEndpoint, then look it up. */</span>
.ifEmpty {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Look up the endpoint if the elector is not broken. */</span>
leaderElectorBreaker
.ifOk { elector <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span>
<span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">this</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>lookupLeader(elector)
}
.ifBroken {
logger<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>warn(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>We have no leader and the leader elector is down<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
}
}
}
}</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">leaderEndpoint</code> is an <a href="https://github.com/advantageous/reakt/wiki/Expected" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">expected</a> value that might not exist. The methods <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ifOk</code> and <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ifBroken</code> are from circuit breaker. The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ifOk</code> means the fuse it not burned out. The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ifBroken </code>means the fuse blew. As you can see combining <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Expected</code> values and services wrapped in <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Breaker</code>s allows us to simplify and reasoning on what to do if things go down.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
When a fuse opens or breaks, then we can work around it. Here is how we mark a broken breaker.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#marking-a-service-breaker-as-broken-the-fuse-is-open" id="user-content-marking-a-service-breaker-as-broken-the-fuse-is-open" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Marking a service Breaker as broken (the fuse is open)</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">try</span> {
leaderElectorBreaker <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">Breaker</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>operational(leaderElectorSupplier<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get())
leaderElectorBreaker
.ifOk { <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">this</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>lookupLeader(it) }
.ifBroken {
logger<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>error(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Unable to connect to leader supplier<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">if</span> (leaderElectorBreaker<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>isOk)
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>increment(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>leader.elector.create.success<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">else</span>
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>increment(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>leader.elector.create.fail<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
leaderElectorBreaker<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>ifOk {
<span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">this</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>handleElectionStream(it)
}
} <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">catch</span> (ex<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-smi" style="box-sizing: border-box;">Exception</span>) {
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>increment(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>leader.elector.create.fail.exception<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
logger<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>error(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Unable to connect to leader supplier<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, ex)
leaderElectorBreaker <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">Breaker</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>broken<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;"><</span><span class="pl-smi" style="box-sizing: border-box;">LeaderElector</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span>()
}</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Notice the use of <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Breaker.operational</code> to denote that we have a new service to work with that should work. Then if the service fails, we mark it has broken with <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Breaker.broken</code>.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#working-with-reakt-streams-1" id="user-content-working-with-reakt-streams-1" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Working with Reakt Streams</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Here is us handling the election stream that we showed a mock-up of earlier.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#working-with-reakt-streams-2" id="user-content-working-with-reakt-streams-2" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Working with Reakt Streams</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> fun handleElectionStream(leaderElector<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-smi" style="box-sizing: border-box;">LeaderElector</span>) {
leaderElector<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>leadershipChangeNotice { result <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span>
result
.catchError { error <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> <span class="pl-c" style="box-sizing: border-box; color: #969896;">// Run on this service thread</span>
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>reactor()
.deferRun {
logger<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>error(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Error handling election stream<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>increment(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>leader.stream.elect.error<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
<span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">this</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>leaderElectorBreaker <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> broken<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;"><</span><span class="pl-smi" style="box-sizing: border-box;">LeaderElector</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span>()
createLeaderElector()
}
}
.then { endpoint <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> <span class="pl-c" style="box-sizing: border-box; color: #969896;">// Run on this service thread</span>
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>reactor()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>deferRun {
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>increment(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>leader.stream.elect.notify<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
logger<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>info(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>New Leader Notify {} {}<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, endpoint<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>host, endpoint<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>port)
handleSuccessfulLeaderLookupOrStream(endpoint)
}
}
}
}</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Notice that we use <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">reactor.deferRun</code> so we can handle this stream on this services thread.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Now let's show another example of <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Promises.all</code>. We have a Cassandra service that wants to write a heap of records to the DB. It wants to write the records in parallel.</div>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Stores Metric data and results into Cassandra.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
internal <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">class</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">CassandraMetricRepository</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * @param sessionAsyncSupplier supplier to supply Cassandra session.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * @param serviceMgmt serviceMgmt to manage callbacks and repeating tasks.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * @param promise returns when cassandra initializes.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> *</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
(
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Cassandra Session supplier.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> val sessionAsyncSupplier: AsyncSupplier<Session>,
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * QBit serviceMgmt for repeating tasks, stats, time and callbacks that execute on the caller's thread.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> val serviceMgmt: ServiceManagementBundle,
promise: Promise<Boolean>) : MetricRepositoryService {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * generate the sequence for backup.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> val sequenceGen <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> AtomicLong(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">2</span>)
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Reference to the cassandra session which get connected to async.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> var sessionBreaker <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">Breaker</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>opened<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;"><</span><span class="pl-smi" style="box-sizing: border-box;">Session</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span>()
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Error counts from Cassandra driver for the last time period.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> val errorCount <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> AtomicLong()
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Notice that we create our <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">sessionBreaker</code>, which is our reference to Cassandra as an opened Circuit. We define a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">sessionAsyncSupplier</code> An <a href="https://github.com/advantageous/reakt/wiki/Full-example-using-Promises.all,-promises,-AsyncCallback-and-Reakt-Guava-bridge-to-implement-Cassandra-repository." style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">AsyncSupplier</code></a> is also from Reakt. It is like a regular Supplier except it is async.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
We use the <a href="https://github.com/advantageous/reakt/wiki/Reactor" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">reactor</a> to define a repeating task to check the health of the Cassandra connection.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#using-the-reactor" id="user-content-using-the-reactor" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using the reactor</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> init {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Connect the Cassandra session. */</span>
connectSession(promise)
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/*</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> This makes sure we are connected.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> It provides circuit breaker if sessionBreaker is down to auto reconnect.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
serviceMgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>reactor()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>addRepeatingTask(<span class="pl-smi" style="box-sizing: border-box;">Duration</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>ofSeconds(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">5</span>)) </pre>
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> { <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">this</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>cassandraCircuitBreaker() }
}</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
There we check for the health of our Cassandra session and if it goes down, we try to reconnect just like before.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
We use the circuit breaker to do alternative logic if our connection goes down.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#using-alternative-breaker-logic" id="user-content-using-alternative-breaker-logic" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>using alternative Breaker logic</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> override fun recordMetrics(callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Callback<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span>, metrics<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Metric</span>></span>) {
sessionBreaker()
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* if we are not connected, fail fast. */</span>
.ifBroken { callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>reject(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Not connected to Cassandra<span class="pl-pds" style="box-sizing: border-box;">"</span></span>) }
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* If we are connected then call cassandra. */</span>
.ifOk { session <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> doStoreMetrics(session, callback, metrics) }
}</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Note the use of <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ifBroken</code> and <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ifOk</code>. This way we can control the reconnect.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The method <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">doStoreMetrics</code> stores many records to Cassandra asynchronously, and even though it saves records in parallel it does not let its caller know via a callback, unless all of the records were stored.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#using-reactorall-to-coordinate-many-async-calls" id="user-content-using-reactorall-to-coordinate-many-async-calls" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using reactor.all to coordinate many async calls</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-c" style="box-sizing: border-box; color: #969896;">/**</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Does the low level cassandra storage.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> fun doStoreMetrics(session<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-smi" style="box-sizing: border-box;">Session</span>,
callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Callback<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span>,
metrics<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Metric</span>></span>) {
logger<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>debug(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Storing metrics {}<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, metricss<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>size)
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Make many calls to cassandra using its async lib to recordMetrics</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> each imprint. */</span>
val promises <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> metrics<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>map({ metric <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> doStoreMetric(session, metric) })<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>toList()
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/*</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * Create a parent promise to contain all of the promises we</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> * just created for each imprint.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> */</span>
serviceMgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>reactor()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>all(promises)
.then {
serviceMgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>increment(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>bulk.store.success);</span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;"> logger.info(<span class="pl-pds" style="box-sizing: border-box;">"</span></span>metrics were stored {}<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>, metrics.size)</span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;"> callback.resolve(true)</span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;"> }</span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;"> .catchError { error -></span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;"> serviceMgmt.increment(<span class="pl-pds" style="box-sizing: border-box;">"</span></span>bulk<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>store<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>error);
logger<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>error(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Problem storing metrics ${metrics.size}<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, error)
callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>reject(error)
}
}</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
It does this call coordination by using <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">reactor.all</code> to create a promise that only replies if all of the other promise reply. The method <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">doStoreMetric</code> returns a single promise. We use Kotlin streams (just like Java streams but more concise) to turn the list of metrics into a list of calls to <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">doStoreMetric</code> into a list of <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Promises</code> which we then pass to <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">reactor.all</code> to make all of those promises into a single promise.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">doStoreMetric</code> uses Reakt Guava/Cassandra integration to turn a ListableFuture into a Reakt promise.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#working-with-reakt-cassandra--guava-support-and-using-thenmap" id="user-content-working-with-reakt-cassandra--guava-support-and-using-thenmap" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Working with Reakt Cassandra / Guava support, and using thenMap</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.reakt.guava.Guava.registerCallback</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> fun doStoreMetric(session<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-smi" style="box-sizing: border-box;">Session</span>,
metric <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-smi" style="box-sizing: border-box;">Metric</span>)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> {
val resultSetFuture <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> session<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>executeAsync(<span class="pl-smi" style="box-sizing: border-box;">QueryBuilder</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>insertInto(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">METRICS_TABLE</span>)
.value(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>employeeId<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, metric<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>employeeId)
.value(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>metricType<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, metric<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>metricType<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>name<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>toLowerCase())
.value(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>metricName<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, metric<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>metricName)
.value(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>provider<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, metric<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>provider)
.value(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>externalId<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, metric<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>externalId)
.value(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>value<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, metric<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>value)
.value(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>surrogateKey<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, metric<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>surrogateKey)
.value(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>created_at<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, metric<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>timestamp))
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> createPromiseFromResultSetFutureForStore(resultSetFuture, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Storing Metric<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
}
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> fun createPromiseFromResultSetFutureForStore(resultSetFuture<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-smi" style="box-sizing: border-box;">ResultSetFuture</span>,
message<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-smi" style="box-sizing: border-box;">String</span>)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Promise<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span> {
val resultSetPromise <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> serviceMgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>reactor()<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>promise<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;"><</span><span class="pl-smi" style="box-sizing: border-box;">ResultSet</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span>()
val promise <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> resultSetPromise<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>thenMap({ it<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>wasApplied() })<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>catchError { error <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">if</span> (error is <span class="pl-smi" style="box-sizing: border-box;">DriverException</span>) {
callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>ifPresent { callback1 <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> callback1<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>reject(error<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>message, error) }
logger<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>error(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Error <span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">+</span> message, error)
errorCount<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>incrementAndGet()
}
}
registerCallback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;"><</span><span class="pl-smi" style="box-sizing: border-box;">ResultSet</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span>(resultSetFuture, resultSetPromise)
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">return</span> promise
}
</pre>
</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#using-thenmap-to-convert-a-promise-into-another-type-of-promise" id="user-content-using-thenmap-to-convert-a-promise-into-another-type-of-promise" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using thenMap to convert a promise into another type of Promise</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Notice we use <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">registerCallback</code> from the <a href="http://advantageous.github.io/reakt-guava/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt Guava integration</a> to convert the future into a promise. We also use <a href="https://github.com/advantageous/reakt/wiki/Promise.thenMap" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">promise.thenMap</code></a> to convert a Promise into a Promise.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#using-invokable-promises-inside-of-an-actor-or-managed-event-loop" id="user-content-using-invokable-promises-inside-of-an-actor-or-managed-event-loop" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using invokable Promises inside of an Actor or Managed event loop</h2>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#using-invokewithreactor" id="user-content-using-invokewithreactor" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using invokeWithReactor</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">override fun collectTodo(callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Callback<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Boolean</span>></span>,
todoList<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Todo</span>></span>) {
callCount<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">++</span>
todoRepo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>recordTodoList(todoList)
.then { ok <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span>
todoService
.ifOk { todoService1 <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span>
doCollectWithCallback(callback, todoList, todoService1)
}
.ifBroken {
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>increment(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>collect.call.df.service.broken<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
logger<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>error(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Connection to todoService is down.<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>increment(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>collect.broken<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
}
}
.catchError { error <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span>
mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>setFailing()
logger<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>error(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Connection to cassandra is down.<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, error)
callback<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>reject(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Connection to cassandra is down. <span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">+</span> error<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>message, error)
}
.invokeWithReactor(mgmt<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>reactor())
}</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
You can invoke invokable promises in the context of a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Reactor</code> by using <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">invokeWithReactor(mgmt.reactor())</code>. This allows the callback handlers from the promises to run in the same thread as the service actor or event loop.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#conclusion" id="user-content-conclusion" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Conclusion</h2>
<div style="box-sizing: border-box;">
I hope you enjoyed this article. It links back to areas of the <a href="https://github.com/advantageous/reakt/wiki" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt documentation</a> where you can find more details. If you are new to Reakt and what to understand the question <a href="https://github.com/advantageous/reakt/wiki/Overview-of-Reakt" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Why Reakt and What is Reakt I suggest reading this</a>. Also this <a href="https://github.com/advantageous/reakt/wiki/Interview-about-Reakt-Reactive-Java-Programming-Part-1" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">interview about Reakt</a> might help.</div>
</div>
<div class="wiki-footer gollum-markdown-content boxed-group" id="wiki-footer" style="background-color: white; border-radius: 3px; box-sizing: border-box; clear: none; color: #333333; font-family: Helvetica, arial, nimbussansl, liberationsans, freesans, clean, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 13px; line-height: 18.2px; margin: 30px 30px 0px; position: relative;">
<div class="boxed-group-inner wiki-auxiliary-content wiki-writable markdown-body" style="background: rgb(246, 246, 246); border-radius: 3px; border: 1px solid rgb(216, 216, 216); box-shadow: rgba(0, 0, 0, 0.0588235) 0px 1px 2px; box-sizing: border-box; color: #666666; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; line-height: 1.6; margin: 0px -30px; padding: 10px 15px; word-break: break-word; word-wrap: break-word;">
<a class="wiki-edit-link" href="https://github.com/advantageous/reakt/wiki/_Footer/_edit" style="background-color: transparent; box-sizing: border-box; color: #767676; float: right; margin-top: 0px !important; opacity: 0.2; position: relative; right: -5px; text-decoration: none; transition: opacity 0.2s ease-in-out; z-index: 2;" title="Edit footer"><svg aria-hidden="true" class="octicon octicon-pencil" height="16" version="1.1" viewbox="0 0 14 16" width="14"><path d="M0 12v3h3l8-8-3-3L0 12z m3 2H1V12h1v1h1v1z m10.3-9.3l-1.3 1.3-3-3 1.3-1.3c0.39-0.39 1.02-0.39 1.41 0l1.59 1.59c0.39 0.39 0.39 1.02 0 1.41z"></path></svg></a><br />
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://advantageous.github.io/reakt" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt Reactive Java Website</a></div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Case-Studies-in-Reakt-Reactive-Java#docs" id="user-content-docs" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Docs</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Java Promises</div>
<ul style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/Promise" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Promise</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/Promise.then*-and-catchError" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Promise then*() and catchError()</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/Promise.thenMap" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Promise thenMap()</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/Promises.all" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Promise all()</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/Promises.any" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Promise any()</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/BlockingPromise" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Blocking Promise</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/Invokable-Promise" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Invokable Promise</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/ReplayPromise" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactor Replay Promises</a></li>
</ul>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Reactor, Stream, Results</div>
<ul style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/Reactor" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactor</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/Stream" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Stream</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/Result" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Result</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/StreamResult" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">StreamResult</a></li>
</ul>
</div>
</div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com0tag:blogger.com,1999:blog-6128525281101058933.post-69271268838699133482016-05-04T01:48:00.001-07:002016-05-04T01:55:38.572-07:00Reactive Java with Reakt<div class="markdown-body" style="background-color: white; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 1.6; margin: 0px -30px; padding: 0px 30px; word-break: break-word; word-wrap: break-word;">
<h1 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 2.25em; line-height: 1.2; margin-bottom: 16px; margin-left: 0px; margin-right: 0px; margin-top: 0px !important; padding-bottom: 0.3em;">
Reactive Java with Reakt</h1>
<div style="box-sizing: border-box; margin-bottom: 16px;">
It might seem like <a href="http://advantageous.github.io/reakt/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt</a> is brand new. But it is not brand new. Most of what is in <span style="box-sizing: border-box; font-weight: bolder;"><em style="box-sizing: border-box;">Reakt</em></span> existed in <a href="http://advantageous.github.io/qbit/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit</a> for years. But after working with <a href="https://nodejs.org/en/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Node.js</a> and <a href="http://www.html5rocks.com/en/tutorials/es6/promises/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">JavaScript promises</a>, we realized we could write a lot cleaner interface. Instead of QBit's <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">CallbackBuilder</code> (QBits original Promise library) and <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Reactor</code> we are moving towards <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Reakt</code> promises, streams and the Reakt <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Reactor</code>.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<em style="box-sizing: border-box;">The trick for async and reactive programming is not the streams, it is the call coordination.</em></div>
<blockquote style="border-left-color: rgb(221, 221, 221); border-left-style: solid; border-left-width: 4px; box-sizing: border-box; color: #777777; margin: 0px 0px 16px; padding: 0px 15px;">
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 0px;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Overview-of-Reakt#what-is-reakt-again" id="user-content-what-is-reakt-again" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>What is Reakt again?</h4>
<div style="box-sizing: border-box;">
If you are not familiar with <em style="box-sizing: border-box;">Reakt</em>. There was an <a href="http://www.mammatustech.com/reactive-microservices/q-and-a-interview-reactive-java-reakt-and-qbit-microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">informative interview about <em style="box-sizing: border-box;">Reakt</em></a>, and <a href="http://www.mammatustech.com/reactive-microservices/qbit-microservices-lib-supports-reakt-promises" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit has started to support Reakt as a first class citizen</a>. You should start with the <a href="http://advantageous.github.io/reakt/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt website</a>and the <a href="http://advantageous.github.io/reakt/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt documentation</a>. You can use Reakt with any async lib including async NoSQL drivers.</div>
</blockquote>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Overview-of-Reakt#understanding-managing-callbacks" id="user-content-understanding-managing-callbacks" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Understanding managing callbacks</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
You want to async call <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceA</code>, then <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceB</code>, take the results of <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceA</code> & <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceB</code>, then call <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">serviceC</code>. Then based on the results of call <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">C</code>, call <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">D</code> or <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">E</code> and then return the results to the original caller. Calls to <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">A</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">B</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">C</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">D</code> and <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">E</code> are all async calls and none should take longer than 10 seconds, if they do, then return a timeout to the original caller.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Let's say that the whole async call sequence should timeout in 20 seconds if it does not complete, and also check for circuit breakers, and provide back pressure feedback so the system does not have cascading failures. <em style="box-sizing: border-box;">QBit is really good at this</em>, but creating the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Reactor</code> (QBit's callback and task manager is called <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Reactor</code>) was trial by fire. It was created while using it at scale, and was created while needing something like it and not finding anything. Which did great for the quick evolution of QBit's Reactor but its design could improve as well as its ease-of-use (although these were goals). QBit Reactor is quite good, but we could do better. <em style="box-sizing: border-box;">QBit call coordination is good</em>. <em style="box-sizing: border-box;">Reakt's will be better</em>, and it is already useful.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Overview-of-Reakt#continous-improvement" id="user-content-continous-improvement" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Continous improvement</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<em style="box-sizing: border-box;">Reakt</em> gives us a chance to work on QBit 2 call coordination and pull out <em style="box-sizing: border-box;">async call coordination</em>into a lib that can use be used with other projects that don't use <em style="box-sizing: border-box;">QBit</em>. This is the goal with <em style="box-sizing: border-box;">Reakt</em>.<em style="box-sizing: border-box;">Reakt</em> already has extention libraries for <a href="http://advantageous.github.io/reakt-guava/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;"><em style="box-sizing: border-box;">Guava/Cassandra</em></a> and <a href="http://advantageous.github.io/reakt-vertx/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;"><em style="box-sizing: border-box;">Vert.x</em></a>. <em style="box-sizing: border-box;">Reakt</em> is also supported by<em style="box-sizing: border-box;">QBit</em>, a reactive microservice lib. The <em style="box-sizing: border-box;">Reakt</em> library can work with any JVM async framework. It is not tied to <em style="box-sizing: border-box;">QBit</em> microservices lib.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Overview-of-Reakt#next-steps-with-reakt" id="user-content-next-steps-with-reakt" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Next steps with Reakt</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<em style="box-sizing: border-box;">Reakt IO</em>, which is in progress, will sit on top of <em style="box-sizing: border-box;">Vertx</em> or <em style="box-sizing: border-box;">Conekt</em> (a lightweight Netty IO lib that we are working on). <em style="box-sizing: border-box;">Reakt IO</em> provides a common interface for lightweight IO libs. <em style="box-sizing: border-box;">Reakt</em> provides promises and streams that are Java 8 lambda expression friendly (Groovy closure friendly, Kotlin closures friendly, Scala closure friendly, and Jython lambda friendly too).</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Overview-of-Reakt#when-can-i-use-reakt" id="user-content-when-can-i-use-reakt" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>When can I use Reakt</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Now. <em style="box-sizing: border-box;">QBit</em>, which is a Java microservice lib, can handle call coordination like this at scale. It works. But this call coordination does not need to live in <em style="box-sizing: border-box;">QBit</em>. Thus <em style="box-sizing: border-box;">Reakt</em> was born. <em style="box-sizing: border-box;">Reakt</em> is already very useful and innovative and can be used with <em style="box-sizing: border-box;">Vert.x</em>, <em style="box-sizing: border-box;">Guava</em>, <em style="box-sizing: border-box;">Cassandra</em> and more.<em style="box-sizing: border-box;">Reakt</em> is laser focused on async call coordination on the JVM, and making that experience clean and enjoyable.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Overview-of-Reakt#reactive-programming" id="user-content-reactive-programming" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Reactive programming</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="https://en.wikipedia.org/wiki/Reactive_programming" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactive programming</a> is not a new paradigm. Reactive programming centers around data event flows and propagation of change. There are many example of reactive programming like spreadsheet updates, the Swing event loop, the Vert.x event bus, <a href="https://nodesource.com/blog/understanding-the-nodejs-event-loop/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Node.js event loop</a> and the <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/EventLoop" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">JavaScript browser event loop</a>.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Reactive programming often is at the core of <em style="box-sizing: border-box;">interactive user interfaces</em>, <em style="box-sizing: border-box;">Model-view-controller</em>, simulations, real time animations, but can also be used to for <a href="http://www.mammatustech.com/reactive-microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">reactive microservice development</a>. It is a general technique with many applications.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
You can even manage complex data flows and transformations using tools like Spark. You can do amazing near real time big data transformation using reactive streams, and tools like <a href="http://www.mammatustech.com/introduction-to-apache-spark" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Spark</a>. <a href="https://en.wikipedia.org/wiki/Functional_reactive_programming" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Functional reactive programming</a> has its place, and is increasingly used to process large streams of data into useable information in near real time. However, there is a much more common and mundane use of <em style="box-sizing: border-box;">reactive programming</em>.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Overview-of-Reakt#object-oriented-reactive-programming" id="user-content-object-oriented-reactive-programming" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Object-oriented reactive programming</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Object-oriented reactive programming (OORP) combines <em style="box-sizing: border-box;">object oriented programming</em> with<em style="box-sizing: border-box;">reactive programming</em>. This has become a very popular model with tools like <em style="box-sizing: border-box;">Angular</em>, <em style="box-sizing: border-box;">React</em> and<em style="box-sizing: border-box;">jQuery</em>. <em style="box-sizing: border-box;">jQuery</em> and other libs also manage call coordination with <em style="box-sizing: border-box;">Promises</em>. Promises are a very common way to manage streams of event data into actionable responses. <em style="box-sizing: border-box;">Promises</em> have become so popular in the JavaScript/Node.js world that <em style="box-sizing: border-box;">Promises</em> are part of <a href="http://es6-features.org/#PromiseUsage" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">ES6</a>.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Overview-of-Reakt#promise-vs-streams" id="user-content-promise-vs-streams" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Promise vs streams</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Events and Streams are great for things that can happen multiple times — keyup, touchstart, or event a user action stream from Kafka, etc.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
With those events you don't really care about what happened before when you attached the listener.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
But often times when dealing with service calls and data repositories, you want to handle a response with a specific next action, and a different action if there was an error or timeout from the responses. <em style="box-sizing: border-box;">You essentially want to call and handle a response asynchronously</em> and that is what <em style="box-sizing: border-box;">promises allow</em>.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
At their most basic level, promises are like event listeners except:</div>
<blockquote style="border-left-color: rgb(221, 221, 221); border-left-style: solid; border-left-width: 4px; box-sizing: border-box; color: #777777; margin: 0px 0px 16px; padding: 0px 15px;">
<div style="box-sizing: border-box;">
A promise can only succeed or fail once. A promise cannot succeed or fail twice, neither can it switch from success to failure. Once it enters its completed state, then it is done.</div>
</blockquote>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Overview-of-Reakt#reakt-promises" id="user-content-reakt-promises" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Reakt Promises</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Reakt promises are very similar in concept to <a href="http://www.html5rocks.com/en/tutorials/es6/promises/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">ES6 promises</a>, which have become standardized in the JavaScript/TypeScript/ES5/ES6/ES7 pantheon on languages.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
A promise can be:</div>
<ul style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><span style="box-sizing: border-box; font-weight: bolder;">fulfilled</span> The callback relating to the promise succeeded</li>
<li style="box-sizing: border-box;"><span style="box-sizing: border-box; font-weight: bolder;">rejected</span> The callback/action relating to the promise failed</li>
<li style="box-sizing: border-box;"><span style="box-sizing: border-box; font-weight: bolder;">pending</span> The callback has not been fulfilled or rejected yet</li>
<li style="box-sizing: border-box;"><span style="box-sizing: border-box; font-weight: bolder;">completed</span> The callback/action has been fulfilled/resolved or rejected</li>
</ul>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Java is not single threaded, meaning that two bits of code can run at the same time, so the design of this promise and streaming library takes that into account. We came out with many tools that adapt the Promise constructs (single threaded event loop in JavaScript) to the Java world (multi-threaded, race condition possibilities, thread visibility, etc.).</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
There are three types of promises:</div>
<ul style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">Callback promises (async)</li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/BlockingPromise" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Blocking promises</a> (for testing and legacy integration)</li>
<li style="box-sizing: border-box;">Replay promises (allow promises to be handled on the same thread as caller, managed by a<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Reactor</code>)</li>
</ul>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Unlike their JavaScript cousins that do not have to worry about thread safety. Promises in Reakt can be <a href="https://github.com/advantageous/reakt/wiki/Invokable-Promise" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">invokable</a>.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Promises can be very fluent.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Overview-of-Reakt#passing-a-promise-as-a-callback-handler" id="user-content-passing-a-promise-as-a-callback-handler" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Passing a promise as a callback handler</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> employeeService<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>lookupEmployee(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">33</span>, result <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
result<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>then(e <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> saveEmployee(e))
.catchError(error <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {
logger<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>error(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Unable to lookup<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, error);
});
});</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Promises in Java become even more fluent when you use <a href="https://github.com/advantageous/reakt/wiki/Invokable-Promise" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">invokable promises</a>.</div>
<h4 style="box-sizing: border-box; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Overview-of-Reakt#using-an-invokable-promise" id="user-content-using-an-invokable-promise" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using an invokable promise</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> employeeService<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>lookupEmployee(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>123<span class="pl-pds" style="box-sizing: border-box;">"</span></span>)
.then((employee)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">-</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span> {<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>})<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>catchError(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>invoke();</pre>
</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<em style="box-sizing: border-box;">Replay promises</em> are the most like their JS cousins. <em style="box-sizing: border-box;">Replay promises</em> are usually managed by the<em style="box-sizing: border-box;">Reakt Reactor</em> and supports environments like <em style="box-sizing: border-box;">Vert.x</em>, <em style="box-sizing: border-box;">Akka</em>, <em style="box-sizing: border-box;">Reactor</em>, <em style="box-sizing: border-box;">Netty</em>, async noSQL drivers and <em style="box-sizing: border-box;">QBit</em>.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
It is common to make async calls to store data in a NoSQL store or to call a remote REST interface or deal with a distributed cache or queue. Also Java is strongly typed so the library that mimics JS promises is going to look a bit different. Reakt uses similar terminology to ES6 promises where it makes sense.</div>
<blockquote style="border-left-color: rgb(221, 221, 221); border-left-style: solid; border-left-width: 4px; box-sizing: border-box; color: #777777; margin: 0px 0px 16px; padding: 0px 15px;">
<div style="box-sizing: border-box;">
We have been on projects where we wrote libs in JS and Java that did very similar things and the promise code for ES6 and Java looks close to the point where we have to take a double look to decide which language we are working with.</div>
</blockquote>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Overview-of-Reakt#conclusion" id="user-content-conclusion" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Conclusion</h2>
<div style="box-sizing: border-box;">
<span style="box-sizing: border-box; font-weight: bolder;"><em style="box-sizing: border-box;">QBit</em></span> has had <span style="box-sizing: border-box; font-weight: bolder;"><em style="box-sizing: border-box;">Promises</em></span> for a few years now, but they were called them <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">CallbackBuilders </code>instead and were not as easy-to-work with. <span style="box-sizing: border-box; font-weight: bolder;"><em style="box-sizing: border-box;">Reakt</em></span> focuses using standard terminology and ease-of-use. With <span style="box-sizing: border-box; font-weight: bolder;"><em style="box-sizing: border-box;">Reakt</em></span> you can use the same terminology and modeling on projects that do not use <a href="http://advantageous.github.io/qbit/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit, reactive microservices lib</a> like Conekt, <a href="http://vertx.io/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Vert.x</a>, <a href="https://github.com/ReactiveX/RxJava" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">RxJava</a>, <a href="https://projectreactor.io/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Project Reactor</a>, <a href="https://www.lightbend.com/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Lightbend</a>, and <a href="http://www.reactive-streams.org/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">reactive streams</a>.</div>
</div>
<div class="wiki-footer gollum-markdown-content boxed-group" id="wiki-footer" style="background-color: white; border-radius: 3px; box-sizing: border-box; clear: none; color: #333333; font-family: Helvetica, arial, nimbussansl, liberationsans, freesans, clean, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 13px; line-height: 18.2px; margin: 30px 30px 0px; position: relative;">
<div class="boxed-group-inner wiki-auxiliary-content wiki-writable markdown-body" style="background: rgb(246, 246, 246); border-radius: 3px; border: 1px solid rgb(216, 216, 216); box-shadow: rgba(0, 0, 0, 0.0588235) 0px 1px 2px; box-sizing: border-box; color: #666666; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; line-height: 1.6; margin: 0px -30px; padding: 10px 15px; word-break: break-word; word-wrap: break-word;">
<a class="wiki-edit-link" href="https://github.com/advantageous/reakt/wiki/_Footer/_edit" style="background-color: transparent; box-sizing: border-box; color: #767676; float: right; margin-top: 0px !important; opacity: 0.2; position: relative; right: -5px; text-decoration: none; transition: opacity 0.2s ease-in-out; z-index: 2;" title="Edit footer"><svg aria-hidden="true" class="octicon octicon-pencil" height="16" version="1.1" viewbox="0 0 14 16" width="14"><path d="M0 12v3h3l8-8-3-3L0 12z m3 2H1V12h1v1h1v1z m10.3-9.3l-1.3 1.3-3-3 1.3-1.3c0.39-0.39 1.02-0.39 1.41 0l1.59 1.59c0.39 0.39 0.39 1.02 0 1.41z"></path></svg></a><br />
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://advantageous.github.io/reakt" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt Reactive Java Website</a></div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Overview-of-Reakt#docs" id="user-content-docs" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Docs</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Java Promises</div>
<ul style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/Promise" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Promise</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/Promise.then*-and-catchError" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Promise then*() and catchError()</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/Promise.thenMap" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Promise thenMap()</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/Promises.all" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Promise all()</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/Promises.any" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Promise any()</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/BlockingPromise" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Blocking Promise</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/Invokable-Promise" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Invokable Promise</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/ReplayPromise" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactor Replay Promises</a></li>
</ul>
<div style="box-sizing: border-box; margin-bottom: 16px;">
Reactor, Stream, Results</div>
<ul style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/Reactor" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactor</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/Stream" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Stream</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/Result" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Result</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/reakt/wiki/StreamResult" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">StreamResult</a></li>
</ul>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Overview-of-Reakt#related-projects" id="user-content-related-projects" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Related Projects</h2>
<ul style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><a href="http://advantageous.github.io/qbit/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit Reactive Microservices</a></li>
<li style="box-sizing: border-box;"><a href="http://advantageous.github.io/reakt" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt Reactive Java</a></li>
<li style="box-sizing: border-box;"><a href="http://advantageous.github.io/reakt-guava/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reakt Guava Bridge</a></li>
<li style="box-sizing: border-box;"><a href="https://github.com/advantageous/qbit-extensions" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit Extensions</a></li>
<li style="box-sizing: border-box;"><a href="http://advantageous.github.io/elekt-consul/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Elekt Consul Leadership election</a></li>
<li style="box-sizing: border-box;"><a href="http://advantageous.github.io/elekt/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Elekt Leadership election</a></li>
<li style="box-sizing: border-box;"><a href="http://www.mammatustech.com/reactive-microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactive Microservices</a></li>
</ul>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/reakt/wiki/Overview-of-Reakt#further-reading" id="user-content-further-reading" style="background-color: transparent; box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Further reading</h2>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/microservices-architecture" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">What is Microservices Architecture?</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit Java Micorservices lib tutorials</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
The Java microservice lib. QBit is a reactive programming lib for building microservices - JSON, HTTP, WebSocket, and REST. QBit uses reactive programming to build elastic REST, and WebSockets based cloud friendly, web services. SOA evolved for mobile and cloud. ServiceDiscovery, Health, reactive StatService, events, Java idiomatic reactive programming for Microservices.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Find more tutorial on QBit</a>.</div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://rick-hightower.blogspot.com/2015/03/reactive-programming-service-discovery.html" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactive Programming</a>, <a href="http://rick-hightower.blogspot.com/2015/03/java-microservices-architecture.html" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Java Microservices</a>, <a href="http://www.linkedin.com/in/rickhigh" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Rick Hightower</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">High-speed microservices consulting firm and authors of QBit with lots of experience with Vertx - Mammatus Technology</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.aboutobjects.com/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Highly recommended consulting and training firm who specializes in microservices architecture and mobile development that are already very familiar with QBit and Vertx as well as iOS and Android - About Objects</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/java-microservices-architecture" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Java Microservices Architecture</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/Microservice-Service-Discovery-with-Consul" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Microservice Service Discovery with Consul</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/consul-service-discovery-and-health-for-microservices-architecture-tutorial" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Microservices Service Discovery Tutorial with Consul</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/reactive-microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactive Microservices</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/high-speed-microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">High Speed Microservices</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/java-microservices-consulting" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Java Microservices Consulting</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://www.mammatustech.com/java-reactive-microservice-training" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Microservices Training</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="https://github.com/MammatusTech/qbit-microservices-examples/wiki/Reactor-tutorial--%7C-reactively-handling-async-calls-with-QBit-Reactive-Microservices" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactive Microservices Tutorial, using the Reactor</a></div>
<div style="box-sizing: border-box; margin-bottom: 16px;">
<a href="http://restlet.com/blog/2015/09/04/this-week-in-api-land-20/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">QBit is mentioned in the Restlet blog</a></div>
<div style="box-sizing: border-box;">
<a href="https://www.jetbrains.com/idea/" style="background-color: transparent; box-sizing: border-box; color: #4078c0; text-decoration: none;">All code is written using JetBrains Idea - the best IDE ever!</a></div>
</div>
</div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com0tag:blogger.com,1999:blog-6128525281101058933.post-32708266186630023822016-05-02T01:21:00.003-07:002016-05-02T01:21:32.135-07:00Integrating TypeSafe Config and Konf.<div style="background-color: #fbfaf7; border: 0px; color: #232323; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; font-stretch: inherit; line-height: 25.2px; margin-bottom: 22px; padding: 0px; vertical-align: baseline;">
<a href="http://advantageous.github.io/konf/" style="border: 0px; color: #c30000; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">Konf Website</a></div>
<h1 style="background-color: #fbfaf7; border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-width: 0px 0px 1px; color: #232323; font-family: Arvo, Monaco, serif; font-size: 30px; font-stretch: inherit; font-weight: normal; line-height: 1.3; margin: 36px 0px 10px; padding: 0px 0px 5px; vertical-align: baseline;">
<a aria-hidden="true" class="anchor" href="http://advantageous.github.io/konf-typesafe-config/#konf---typed-java-config-integration" id="konf---typed-java-config-integration" style="border: 0px; color: #c30000; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"><span aria-hidden="true" class="octicon octicon-link" style="border: 0px; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"></span></a>Konf - Typed Java Config Integration</h1>
<div style="background-color: #fbfaf7; border: 0px; color: #232323; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; font-stretch: inherit; line-height: 25.2px; margin-bottom: 22px; padding: 0px; vertical-align: baseline;">
This allows you to combine TypeSafe config and Konf. You can have TypeSafe config be a fallback for Konf or the other way around.</div>
<div style="background-color: #fbfaf7; border: 0px; color: #232323; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; font-stretch: inherit; line-height: 25.2px; margin-bottom: 22px; padding: 0px; vertical-align: baseline;">
You can load TypeSafe config as a Konf <code style="border: 0px; color: black; font-family: Monaco, 'Bitstream Vera Sans Mono', 'Lucida Console', Terminal, monospace; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">Config</code> instance as follows:</div>
<h4 style="background-color: #fbfaf7; border: 0px; color: #232323; font-family: Arvo, Monaco, serif; font-size: 14px; font-stretch: inherit; line-height: 25.2px; margin: 36px 0px 10px; padding: 0px; vertical-align: baseline;">
<a aria-hidden="true" class="anchor" href="http://advantageous.github.io/konf-typesafe-config/#loading-typesafe-config-as-a-konf-config-object" id="loading-typesafe-config-as-a-konf-config-object" style="border: 0px; color: #c30000; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: 200; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"><span aria-hidden="true" class="octicon octicon-link" style="border: 0px; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"></span></a>Loading Typesafe config as a Konf Config object</h4>
<div class="highlight highlight-source-java" style="background-color: #fbfaf7; border: 0px; color: #232323; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; font-stretch: inherit; line-height: 25.2px; margin: 0px; padding: 0px; vertical-align: baseline;">
<pre style="background: rgb(253, 254, 251); border-radius: 4px; border: 1px solid rgb(215, 216, 200); color: black; font-family: Monaco, 'Bitstream Vera Sans Mono', 'Lucida Console', Terminal, monospace; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin-bottom: 32px; overflow-x: auto; overflow-y: hidden; padding: 4px 12px; vertical-align: baseline;"> <span class="pl-smi" style="border: 0px; color: #333333; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">Config</span> config <span class="pl-k" style="border: 0px; color: #a71d5d; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">=</span> <span class="pl-smi" style="border: 0px; color: #333333; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">TypeSafeConfig</span><span class="pl-k" style="border: 0px; color: #a71d5d; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">.</span>typeSafeConfig();
<span class="pl-k" style="border: 0px; color: #a71d5d; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">final</span> <span class="pl-smi" style="border: 0px; color: #333333; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">String</span> abc <span class="pl-k" style="border: 0px; color: #a71d5d; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">=</span> config<span class="pl-k" style="border: 0px; color: #a71d5d; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">.</span>getString(<span class="pl-s" style="border: 0px; color: #183691; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"><span class="pl-pds" style="border: 0px; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">"</span>abc<span class="pl-pds" style="border: 0px; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">"</span></span>);
assertEquals(<span class="pl-s" style="border: 0px; color: #183691; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"><span class="pl-pds" style="border: 0px; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">"</span>abc<span class="pl-pds" style="border: 0px; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">"</span></span>, abc);</pre>
</div>
<div style="background-color: #fbfaf7; border: 0px; color: #232323; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; font-stretch: inherit; line-height: 25.2px; margin-bottom: 22px; padding: 0px; vertical-align: baseline;">
You can also chain TypeSafe config as fallback or Konfig as a fallback for TypeSafe config as follows:</div>
<h4 style="background-color: #fbfaf7; border: 0px; color: #232323; font-family: Arvo, Monaco, serif; font-size: 14px; font-stretch: inherit; line-height: 25.2px; margin: 36px 0px 10px; padding: 0px; vertical-align: baseline;">
<a aria-hidden="true" class="anchor" href="http://advantageous.github.io/konf-typesafe-config/#konf-as-a-fallback-for-typesafe-config" id="konf-as-a-fallback-for-typesafe-config" style="border: 0px; color: #c30000; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: 200; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"><span aria-hidden="true" class="octicon octicon-link" style="border: 0px; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"></span></a>Konf as a fallback for TypeSafe config.</h4>
<div class="highlight highlight-source-java" style="background-color: #fbfaf7; border: 0px; color: #232323; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; font-stretch: inherit; line-height: 25.2px; margin: 0px; padding: 0px; vertical-align: baseline;">
<pre style="background: rgb(253, 254, 251); border-radius: 4px; border: 1px solid rgb(215, 216, 200); color: black; font-family: Monaco, 'Bitstream Vera Sans Mono', 'Lucida Console', Terminal, monospace; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin-bottom: 32px; overflow-x: auto; overflow-y: hidden; padding: 4px 12px; vertical-align: baseline;"><span class="pl-k" style="border: 0px; color: #a71d5d; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">import static</span> <span class="pl-smi" style="border: 0px; color: #333333; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">io.advantageous.config.ConfigLoader.config</span>;
<span class="pl-k" style="border: 0px; color: #a71d5d; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">import static</span> <span class="pl-smi" style="border: 0px; color: #333333; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">io.advantageous.config.ConfigLoader.configs</span>;
<span class="pl-k" style="border: 0px; color: #a71d5d; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">import static</span> <span class="pl-smi" style="border: 0px; color: #333333; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">io.advantageous.config.ConfigLoader.load</span>;
<span class="pl-c1" style="border: 0px; color: #0086b3; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">...</span>
<span class="pl-smi" style="border: 0px; color: #333333; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">Config</span> config;
<span class="pl-c1" style="border: 0px; color: #0086b3; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">...</span>
config <span class="pl-k" style="border: 0px; color: #a71d5d; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">=</span> configs(<span class="pl-smi" style="border: 0px; color: #333333; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">TypeSafeConfig</span><span class="pl-k" style="border: 0px; color: #a71d5d; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">.</span>typeSafeConfig(), config(<span class="pl-s" style="border: 0px; color: #183691; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"><span class="pl-pds" style="border: 0px; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">"</span>test-config.js<span class="pl-pds" style="border: 0px; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">"</span></span>));</pre>
</div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com0tag:blogger.com,1999:blog-6128525281101058933.post-52813497904102418092016-05-01T22:57:00.002-07:002016-05-01T22:57:21.351-07:00Konf - Java config system that supports YAML, JSON, Java properties, Java pojos, List, Maps and JavaScript<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<a href="http://advantageous.github.io/konf/" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">Konf Website</a></div>
<h1 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 2.25em; line-height: 1.2; margin: 1em 0px 16px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#konf---typed-java-config-system" id="user-content-konf---typed-java-config-system" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Konf - Typed Java Config system</h1>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
Java configuration library similar in concept to TypeSafe config, but uses full YAML or JSON or JavaScript for configuration (and more).</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
Konf allows you to easily create your own config DSLs.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#using-konf-on-your-project" id="user-content-using-konf-on-your-project" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using Konf on your project</h2>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
Konf is in the <a href="http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22io.advantageous.konf%22" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">public maven repo</a>.</div>
<h3 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.5em; line-height: 1.43; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#using-konf-from-maven" id="user-content-using-konf-from-maven" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using konf from maven</h3>
<div class="highlight highlight-text-xml" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">dependency</span>>
<<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">groupId</span>>io.advantageous.konf</<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">groupId</span>>
<<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">artifactId</span>>konf</<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">artifactId</span>>
<<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">version</span>>1.3.0.RELEASE</<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">version</span>>
</<span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">dependency</span>></pre>
</div>
<h3 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.5em; line-height: 1.43; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#using-konf-from-gradle" id="user-content-using-konf-from-gradle" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using konf from gradle</h3>
<div class="highlight highlight-source-java" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">compile <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>io.advantageous.konf:konf:1.3.0.RELEASE<span class="pl-pds" style="box-sizing: border-box;">'</span></span></pre>
</div>
<h3 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.5em; line-height: 1.43; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#using-konf-from-scala-sbt" id="user-content-using-konf-from-scala-sbt" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using konf from scala sbt</h3>
<div class="highlight highlight-source-java" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">libraryDependencies <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">+=</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>io.advantageous.konf<span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">%</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>konf<span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">%</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>1.3.0.RELEASE<span class="pl-pds" style="box-sizing: border-box;">"</span></span></pre>
</div>
<h3 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.5em; line-height: 1.43; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#using-konf-from-clojure-leiningen" id="user-content-using-konf-from-clojure-leiningen" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using konf from clojure leiningen</h3>
<div class="highlight highlight-source-lisp" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">[io.advantageous.konf/konf <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>1.3.0.RELEASE<span class="pl-pds" style="box-sizing: border-box;">"</span></span>]</pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
Here is an example config for JavaScript.</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
Konf expects the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">config</code> variable to be set to a JavaScript object with properties.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#javascript-based-configuration-for-java" id="user-content-javascript-based-configuration-for-java" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>JavaScript based configuration for Java</h4>
<div class="highlight highlight-source-js" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">var</span> config <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> {
myUri<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">uri</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>http://host:9000/path?foo=bar<span class="pl-pds" style="box-sizing: border-box;">"</span></span>),
someKey<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> {
nestedKey<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">234</span>,
other<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>this text<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
}
};</pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
You can use full JavaScript for configuration as long as you define a variable called <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">config</code> that results in a JavaScript object which equates to a Java map.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#defining-your-own-dsl" id="user-content-defining-your-own-dsl" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Defining your own DSL</h2>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
You can define you own config DSL for your environment. We have a <a href="https://github.com/advantageous/konf/wiki/Config-Logic---creating-your-own-config-DSL" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">full example that shows you how to create a custom config DSL</a> for your internal projects. The example uses Mesosphere and Docker PORT look ups and it is from a real project.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#defining-your-own-config-dsl" id="user-content-defining-your-own-config-dsl" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Defining your own config DSL</h4>
<div class="highlight highlight-source-js" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">var</span> config <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> {
platform<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> {
statsd<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>udp://<span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">+</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getDockerHost</span>() <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">+</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>:8125<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
servicePort<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">mesosPortAt</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">8080</span>),
adminPort<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">mesosPortAt</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1</span>, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">9090</span>),
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">...</span></pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
See the real world for <a href="https://github.com/advantageous/konf/wiki/Config-Logic---creating-your-own-config-DSL" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">example that uses Konf to find ports under Mesosphere</a> (running in stating or prod) or under Docker (running on a local developers box).</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#overview" id="user-content-overview" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Overview</h2>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
Overview</div>
<ul style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">implemented in plain Java SDK almost no dependencies (sl4j, and reflekt with no others)</li>
<li style="box-sizing: border-box;">supports files in : YAML, JSON, JSON LAX, JavaScript, Java properties or any tree of Map/List basic types and POJOs</li>
<li style="box-sizing: border-box;">allows you to easily create your own config DSL</li>
<li style="box-sizing: border-box;">merges multiple configs across all formats</li>
<li style="box-sizing: border-box;">can load from configs, from classpath, http, file or just an Java Object tree</li>
<li style="box-sizing: border-box;">great support for "nesting" (treat any subtree of the config the same as the whole config)</li>
<li style="box-sizing: border-box;">users can override the config with Java system properties, java -Dmyapp.foo.bar=10 and sysProp</li>
<li style="box-sizing: border-box;">users can override the config with OS environment variables</li>
<li style="box-sizing: border-box;">supports configuring an app, with its framework and libraries, all from a single file such as application.yaml</li>
<li style="box-sizing: border-box;">parses duration and size settings, "512k" or "10 seconds"</li>
<li style="box-sizing: border-box;">converts types, so if you ask for a boolean and the value is the string "yes", or you ask for a float and the value is an int, it will figure it out.</li>
<li style="box-sizing: border-box;">API based on immutable Config instances, for thread safety and easy reasoning about config transformations</li>
<li style="box-sizing: border-box;">extensive test coverage</li>
</ul>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
This library limits itself to config. If you want to load config from another source, e.g., database or Redis or MongoDB, then you would need to write some custom code. The library has nice support for merging configurations (Configs with fall-backs) so if you build a custom Config from a custom source it's easy to merge it in. Just implement Config and then use<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">config(config...)</code> to configure your config into a chain of other configs. This is described at length below see "Loading config files with fallbacks".</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#license" id="user-content-license" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>License</h2>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
The license is Apache 2.0.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#release-notes" id="user-content-release-notes" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Release Notes</h2>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
Please see <a href="https://github.com/advantageous/konf/releases" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">Release Notes</a>, and <a href="https://github.com/advantageous/konf/wiki/Release-Notes-Draft" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">Release Notes In Progress</a> for the latest releases.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#build" id="user-content-build" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Build</h2>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
The build uses gradle and the tests are written in Java; and, the library itself is plain Java.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#using-the-library" id="user-content-using-the-library" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using the Library</h2>
<div class="highlight highlight-source-java" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.config.ConfigLoader</span>;
<span class="pl-smi" style="box-sizing: border-box;">Config</span> conf <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">ConfigLoader</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>load(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>myconfig.js<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>reference.js<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">int</span> bar1 <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> conf<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getInt(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>foo.bar<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
<span class="pl-smi" style="box-sizing: border-box;">Config</span> foo <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> conf<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getConfig(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>foo<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">int</span> bar2 <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> foo<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getInt(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>bar<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);</pre>
</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#longer-examples" id="user-content-longer-examples" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Longer Examples</h2>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
You can see longer examples in <a href="https://github.com/advantageous/konf/blob/master/src/test/java/io/advantageous/config/JsLoadTest.java" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">tests</a> along with <a href="https://github.com/advantageous/konf/blob/master/src/test/resources/test-config.js" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">sample config</a>. You can run these examples by <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">git cloning</code> this project and <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">running gradle test</code>.</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
In brief, as shown in the examples:</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
You create a Config instance provided by your application. You use <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ConfigLoader.load()</code> and you can define your own config system. You could setup default <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">reference.yaml</code> or <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">reference.json</code> but you don't have to. You could just load a single level of config. Config is as complex or as simple as you need.</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
A <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Config</code> can be created with the parser methods in <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ConfigLoader.load</code> or built up from any POJO object tree or tree of Map/List/Pojos basic value. It is very flexible. Examples are shown below and linked to below that use JSON, YAML and allow you to define your own <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">DSL</code> like config. It is very simple and easy to use.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#immutability" id="user-content-immutability" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Immutability</h2>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
Objects are immutable, so methods on Config which transform the configuration return a new <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Config</code>. There is no complex tree of <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Config</code> objects. Just <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Config</code>. It is pretty simple to use and understand.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#java-interface-for-konf-is-config" id="user-content-java-interface-for-konf-is-config" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Java interface for Konf is Config.</h2>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
The Java interface for Konf is Config. You can get a sub Config from Config (<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getConfig(path)</code>). The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">path</code> is always in dot notation (<code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">this.that.foo.bar</code>). You can also use:</div>
<ul style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">hasPath(path)</code></li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getInt(path)</code></li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getLong(path)</code></li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getDouble(path)</code></li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getBoolean(path)</code> can be true, false, "yes", "no", "on", "off", yes, no, off, on</li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getString(path)</code></li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getStringList(path)</code> gets a list of strings</li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getConfig(path)</code> gets a sub-config.</li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getMap(path)</code> gets a map which is a sub-config.</li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getConfigList(path)</code> gets a list of configs at the location specified.</li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getIntList(path)</code></li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getLongList(path)</code></li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getDoubleList(path)</code></li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getBooleanList(path)</code></li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getDuration(path)</code> gets <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">java.time.Duration</code> useful for timeouts</li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getDurationList(path)</code> gets duration list</li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getUri(path)</code> gets <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">java.net.URI</code> useful for connecting to downstream services</li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getUriList(path)</code> useful for connecting to downstream services</li>
</ul>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getMap</code> works with JavaScript objects (or Java maps see below for loading config from Java objects, YAML or JSON). The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getStringList</code> and <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getConfigList</code> works with JavaScript array of string and a JavaScript array of JavaScript objects.</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
Note you get an exception if the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">path</code> requested is not found. Use <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">hasPath(path)</code> if you think the config path might be missing.</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
Here is partial glimpse at the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Config</code> interface.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#config-interface" id="user-content-config-interface" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Config interface</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">interface</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">Config</span> {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Get string at location. */</span>
<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getString</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">path</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Checks to see if config has the path specified. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">boolean</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">hasPath</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">path</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Get int at location. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">int</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getInt</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">path</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Get float at location. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">float</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getFloat</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">path</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Get double at location. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">double</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getDouble</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">path</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Get long at location. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">long</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getLong</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">path</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Get list of strings at location. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getStringList</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">path</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Get map at location. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Map<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span>, <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Object</span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getMap</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">path</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Get a sub-config at location. */</span>
<span class="pl-smi" style="box-sizing: border-box;">Config</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getConfig</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">path</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Get list of sub-configs at location. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Config</span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getConfigList</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">path</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Get a single POJO out of config at path. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;"><<span class="pl-smi" style="box-sizing: border-box; color: #333333;">T</span>></span> <span class="pl-smi" style="box-sizing: border-box;">T</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">get</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">path</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Class<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">T</span>></span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">type</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Get a list of POJOs. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;"><<span class="pl-smi" style="box-sizing: border-box; color: #333333;">T</span>></span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">T</span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getList</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">path</span>, <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Class<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">T</span>></span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">componentType</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Get duration. Good for timeouts */</span>
<span class="pl-smi" style="box-sizing: border-box;">Duration</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getDuration</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">path</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Get duration list. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Duration</span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getDurationList</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">path</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/** Get int list. */</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Integer</span>></span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">getIntegerList</span>(<span class="pl-smi" style="box-sizing: border-box;">String</span> <span class="pl-v" style="box-sizing: border-box; color: #ed6a43;">path</span>);
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
}
</pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getX</code> methods work like you would expect. Given this config file.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#javascript-functions-for-config" id="user-content-javascript-functions-for-config" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>JavaScript functions for config</h2>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#javascript-functions-that-we-support" id="user-content-javascript-functions-that-we-support" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>JavaScript functions that we support</h4>
<ul style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">sysProp(propName)</code> to read a sysProp as in <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">fooSize : sysProp("my.foo.size")</code></li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">sysPropOrDefault(propName, defaultValue)</code> to read a sysProp or a default</li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">isWindowsOS()</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">isMacOS()</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">isUnix()</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">isLinux()</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">isSolaris()</code></li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">env()</code> as in <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">fooSize : env('MY_FOO_SIZE')</code> or even <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">fooSize : sysPropOrDefault("my.foo.size", env('MY_FOO_SIZE'))</code></li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">uri()</code> which creates a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">java.net.URI</code> as in <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">fooURI : uri ("http://localhost:8080/")</code></li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">java.time.Duration</code> is imported as <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">duration</code></li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">java.lang.System</code> is imported as <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">system</code></li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">seconds(units)</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">minutes(units)</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">hours(units)</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">days(units)</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">millis(units)</code> and <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">milliseconds(units</code>) define a <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">Duration</code> which is useful for configuring timeouts and interval jobs</li>
<li style="box-sizing: border-box;">constants <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">yes</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">no</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">on</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">off</code> for boolean config</li>
</ul>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#sample-config-for-testing-and-showing-how-config-works" id="user-content-sample-config-for-testing-and-showing-how-config-works" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Sample config for testing and showing how config works</h4>
<div class="highlight highlight-source-js" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">var</span> config <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> {
myUri<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">uri</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>http://host:9000/path?foo=bar<span class="pl-pds" style="box-sizing: border-box;">"</span></span>),
someKey<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> {
nestedKey<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">234</span>,
other<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>this text<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
},
int1<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1</span>,
float1<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1.0</span>,
double1<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1.0</span>,
long1<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1</span>,
string1<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>rick<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
stringList<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> [<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>Foo<span class="pl-pds" style="box-sizing: border-box;">'</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>Bar<span class="pl-pds" style="box-sizing: border-box;">'</span></span>],
configInner<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> {
int2<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">2</span>,
float2<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">2.0</span>
},
uri<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">uri</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>http://localhost:8080/foo<span class="pl-pds" style="box-sizing: border-box;">"</span></span>),
myClass<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>java.lang.Object<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
myURI<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>http://localhost:8080/foo<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
employee<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> {<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>id<span class="pl-pds" style="box-sizing: border-box;">"</span></span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">123</span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>name<span class="pl-pds" style="box-sizing: border-box;">"</span></span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Geoff<span class="pl-pds" style="box-sizing: border-box;">"</span></span>},
employees<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> [
{id<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">123</span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>name<span class="pl-pds" style="box-sizing: border-box;">"</span></span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Geoff<span class="pl-pds" style="box-sizing: border-box;">"</span></span>},
{id<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">456</span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>name<span class="pl-pds" style="box-sizing: border-box;">"</span></span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Rick<span class="pl-pds" style="box-sizing: border-box;">"</span></span>},
{id<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">789</span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">'</span>name<span class="pl-pds" style="box-sizing: border-box;">'</span></span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Paul<span class="pl-pds" style="box-sizing: border-box;">"</span></span>}
]
};
</pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
We can do the following operations.</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
First we load the config.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#loading-the-config" id="user-content-loading-the-config" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Loading the config.</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-smi" style="box-sizing: border-box;">Config</span> config;
@<span class="pl-smi" style="box-sizing: border-box;">Before</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> setUp() throws <span class="pl-smi" style="box-sizing: border-box;">Exception</span> {
config <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">ConfigLoader</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>load(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>test-config.js<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
}</pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
Note that <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ConfigLoader.load(resources...)</code> takes a variable length string array. By default a resource String can contain a valid URI, which can have the scheme <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">classpath</code>, <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">file</code>, or <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">http</code>. If you do not specify a scheme than the path is assumed to be a classpath resource.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#using-different-resources" id="user-content-using-different-resources" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using different resources</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> config <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">ConfigLoader</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>load(
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/io/mycompany/foo-classpath.js<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>classpath:test-config.js<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>classpath://foo.js<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>classpath:/bar.js<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>file://opt/app/config.js<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>file:///opt/app/config2.js<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>file:/opt/app/config.js<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>http://my.internal.server:9090/foo.js<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
);
</pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
Then we show reading basic types with the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">config</code> object using <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getX</code>.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#reading-basic-types-from-config" id="user-content-reading-basic-types-from-config" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Reading basic types from config</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> @<span class="pl-smi" style="box-sizing: border-box;">Test</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> testSimple() throws <span class="pl-smi" style="box-sizing: border-box;">Exception</span> {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//getInt</span>
assertEquals(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1</span>, config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getInt(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>int1<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//getStringList</span>
assertEquals(asList(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Foo<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Bar<span class="pl-pds" style="box-sizing: border-box;">"</span></span>),
config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getStringList(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>stringList<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//getString </span>
assertEquals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>rick<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getString(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>string1<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//getDouble</span>
assertEquals(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1.0</span>, config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getDouble(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>double1<span class="pl-pds" style="box-sizing: border-box;">"</span></span>), <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0.001</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//getLong</span>
assertEquals(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1L</span>, config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getLong(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>long1<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//getFloat</span>
assertEquals(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1.0f</span>, config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getFloat(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>float1<span class="pl-pds" style="box-sizing: border-box;">"</span></span>), <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0.001</span>);
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Basic JDK value types are supported like class.</span>
assertEquals(<span class="pl-smi" style="box-sizing: border-box;">Object</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class, config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>myClass<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-smi" style="box-sizing: border-box;">Class</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class));
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Basic JDK value types are supported like URI.</span>
assertEquals(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">URI</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>create(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>http://localhost:8080/foo<span class="pl-pds" style="box-sizing: border-box;">"</span></span>),
config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>myURI<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-smi" style="box-sizing: border-box;">URI</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class));
assertEquals(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">URI</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>create(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>http://localhost:8080/foo<span class="pl-pds" style="box-sizing: border-box;">"</span></span>),
config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>uri<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-smi" style="box-sizing: border-box;">URI</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class));
}
</pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
You can work with nested properties as well.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#reading-a-nested-config-from-the-config" id="user-content-reading-a-nested-config-from-the-config" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Reading a nested config from the config</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> @<span class="pl-smi" style="box-sizing: border-box;">Test</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> testGetConfig() throws <span class="pl-smi" style="box-sizing: border-box;">Exception</span> {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Read nested config.</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">Config</span> configInner <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getConfig(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>configInner<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
assertEquals(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">2</span>, configInner<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getInt(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>int2<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
assertEquals(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">2.0f</span>, configInner<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getFloat(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>float2<span class="pl-pds" style="box-sizing: border-box;">"</span></span>), <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0.001</span>);
}
@<span class="pl-smi" style="box-sizing: border-box;">Test</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> testGetMap() throws <span class="pl-smi" style="box-sizing: border-box;">Exception</span> {
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Read nested config as a Java map.</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">Map<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">String</span>, <span class="pl-smi" style="box-sizing: border-box; color: #333333;">Object</span>></span> map <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getMap(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>configInner<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
assertEquals(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">2</span>, (<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">int</span>) map<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>int2<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
assertEquals(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">2.0f</span>, (<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">double</span>) map<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>float2<span class="pl-pds" style="box-sizing: border-box;">"</span></span>), <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0.001</span>);
}</pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
You can read deeply nested config items as well by specifying the property path using dot notation.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#reading-nested-properties-with-dot-notation-from-config" id="user-content-reading-nested-properties-with-dot-notation-from-config" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Reading nested properties with dot notation from config</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> @<span class="pl-smi" style="box-sizing: border-box;">Test</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> testSimplePath() throws <span class="pl-smi" style="box-sizing: border-box;">Exception</span> {
assertTrue(config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>hasPath(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>configInner.int2<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
assertFalse(config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>hasPath(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>configInner.foo.bar<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
assertEquals(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">2</span>, config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getInt(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>configInner.int2<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
assertEquals(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">2.0f</span>, config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getFloat(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>configInner.float2<span class="pl-pds" style="box-sizing: border-box;">"</span></span>), <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0.001</span>);
}</pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
You can also read POJOs directly out of the config file.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#reading-a-pojo-directly-out-of-the-config-file" id="user-content-reading-a-pojo-directly-out-of-the-config-file" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Reading a pojo directly out of the config file</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> @<span class="pl-smi" style="box-sizing: border-box;">Test</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> testReadClass() throws <span class="pl-smi" style="box-sizing: border-box;">Exception</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">Employee</span> employee <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>employee<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-smi" style="box-sizing: border-box;">Employee</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class);
assertEquals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Geoff<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, employee<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>name);
assertEquals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>123<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, employee<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>id);
}
</pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
You can read a list of POJOs at once.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#reading-a-pojo-list-directly-out-of-the-config-file" id="user-content-reading-a-pojo-list-directly-out-of-the-config-file" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Reading a pojo list directly out of the config file</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> @<span class="pl-smi" style="box-sizing: border-box;">Test</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">void</span> testReadListOfClass() throws <span class="pl-smi" style="box-sizing: border-box;">Exception</span> {
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Employee</span>></span> employees <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getList(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>employees<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-smi" style="box-sizing: border-box;">Employee</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>class);
assertEquals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Geoff<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, employees<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>name);
assertEquals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>123<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, employees<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>id);
}</pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
You can also read a list of config objects out of the config as well.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#reading-a-config-list-directly-out-of-the-config-file" id="user-content-reading-a-config-list-directly-out-of-the-config-file" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Reading a config list directly out of the config file</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">List<<span class="pl-smi" style="box-sizing: border-box; color: #333333;">Config</span>></span> employees <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getConfigList(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>employees<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
assertEquals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Geoff<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, employees<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getString(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>name<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
assertEquals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>123<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, employees<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>get(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">0</span>)<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getString(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>id<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));</pre>
</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#using-config-with-yaml" id="user-content-using-config-with-yaml" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using Config with YAML</h2>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
First include a YAML to object parser like <a href="https://github.com/EsotericSoftware/yamlbeans" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">YAML Beans</a> or a library like this.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#example-yaml" id="user-content-example-yaml" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Example YAML</h4>
<div class="highlight highlight-source-yaml" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">name:</span> <span class="pl-s" style="box-sizing: border-box;">Nathan Sweet</span></span>
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;"><span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">age:</span> 28</span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">address:</span> <span class="pl-s" style="box-sizing: border-box;">4011 16th Ave S</span></span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">phone numbers:</span></span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;">- <span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">name:</span> <span class="pl-s" style="box-sizing: border-box;">Home</span></span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">number:</span> <span class="pl-s" style="box-sizing: border-box;">206-555-5138</span></span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;">- <span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">name:</span> <span class="pl-s" style="box-sizing: border-box;">Work</span></span>
<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-ent" style="box-sizing: border-box; color: #63a35c;">number:</span> <span class="pl-s" style="box-sizing: border-box;">425-555-2306</span></span></pre>
</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#using-konf-with-yaml" id="user-content-using-konf-with-yaml" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using Konf with YAML</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-c" style="box-sizing: border-box; color: #969896;">//Use YamlReader to load YAML file.</span>
<span class="pl-smi" style="box-sizing: border-box;">YamlReader</span> reader <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">YamlReader</span>(<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">new</span> <span class="pl-smi" style="box-sizing: border-box;">FileReader</span>(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>contact.yml<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Convert object read from YAML into Konf config</span>
<span class="pl-smi" style="box-sizing: border-box;">Config</span> config <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">ConfigLoader</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>loadFromObject(reader<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>read());
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Now you have strongly typed access to fields</span>
<span class="pl-smi" style="box-sizing: border-box;">String</span> address <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getString(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>address<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);</pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
You can also read Pojos from anywhere in the YAML file as well as sub configs.</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#you-can-also-use-konf-with-json-using-boon" id="user-content-you-can-also-use-konf-with-json-using-boon" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>You can also use Konf with JSON using Boon</h2>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
See <a href="https://github.com/advantageous/boon" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">Boon</a> JSON parser project, and <a href="https://github.com/boonproject/boon/wiki/Boon-JSON-in-five-minutes" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">Boon in five minutes</a></div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#using-konf-with-json" id="user-content-using-konf-with-json" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using Konf with JSON</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-smi" style="box-sizing: border-box;">ObjectMapper</span> mapper <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">JsonFactory</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>create();
<span class="pl-c" style="box-sizing: border-box; color: #969896;">/* Convert object read from YAML into Konf config.</span>
<span class="pl-c" style="box-sizing: border-box; color: #969896;"> 'src' can be File, InputStream, Reader, String. */</span>
<span class="pl-smi" style="box-sizing: border-box;">Config</span> config <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> <span class="pl-smi" style="box-sizing: border-box;">ConfigLoader</span><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>loadFromObject(mapper<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>fromJson(src));
<span class="pl-c" style="box-sizing: border-box; color: #969896;">//Now you have strongly typed access to fields</span>
<span class="pl-smi" style="box-sizing: border-box;">String</span> address <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getString(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>address<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
</pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
Boon supports LAX JSON (Json with comments, and you do not need to quote the field).</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#working-with-javatimeduration" id="user-content-working-with-javatimeduration" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Working with java.time.Duration</h4>
<ul style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getDuration(path)</code> get a duration</li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getDurationList(path)</code> get a duration list</li>
</ul>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
Konf supports "10 seconds" style config for duration as well as having built-in functions and support for ISO-8601. See documentation for <a href="https://github.com/advantageous/konf/wiki/Working-with-Durations" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">duration config</a> for more details.</div>
<h5 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#konf-can-reads-list-of-numbers" id="user-content-konf-can-reads-list-of-numbers" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Konf can reads list of numbers.</h5>
<ul style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getIntList</code> reads list of ints</li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getLongList</code> reads list of longs</li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getDoubleList</code> reads list of doubles</li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getFloatList</code> reads list of floats</li>
</ul>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
See documentation <a href="https://github.com/advantageous/konf/wiki/Working-with-lists-of-ints,-longs,-doubles," style="box-sizing: border-box; color: #4078c0; text-decoration: none;">list of number configuration</a> for more details.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#konf-can-read-memory-sizes" id="user-content-konf-can-read-memory-sizes" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Konf can read memory sizes</h4>
<ul style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getMemorySize(path)</code></li>
<li style="box-sizing: border-box;"><code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">getMemorySizeList(path)</code></li>
</ul>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
This means we support config like:</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#sizes-supported" id="user-content-sizes-supported" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Sizes supported.</h4>
<div class="highlight highlight-source-js" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> diskSpace <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span> 10 gigabytes<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
diskVolumes <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> [<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span> 10 gigabytes<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>10GB<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>10 gigabytes<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">10</span>]</pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
We support the following size Strings.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#supported-size-strings" id="user-content-supported-size-strings" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Supported size strings</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">public</span> <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">enum</span> <span class="pl-en" style="box-sizing: border-box; color: #795da3;">MemorySizeUnit</span> {
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">BYTES</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1</span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>B<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>b<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>byte<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>bytes<span class="pl-pds" style="box-sizing: border-box;">"</span></span>),
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">KILO_BYTES</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1_000</span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>kB<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>kilobyte<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>kilobytes<span class="pl-pds" style="box-sizing: border-box;">"</span></span>),
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">MEGA_BYTES</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1_000_000</span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>MB<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>megabyte<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>megabytes<span class="pl-pds" style="box-sizing: border-box;">"</span></span>),
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">GIGA_BYTES</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1_000_000_000</span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>GB<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>gigabyte<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>gigabytes<span class="pl-pds" style="box-sizing: border-box;">"</span></span>),
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">TERA_BYTES</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1_000_000_000</span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>TB<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>terabyte<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>terabytes<span class="pl-pds" style="box-sizing: border-box;">"</span></span>),
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">PETA_BYTES</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1_000_000_000_000L</span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>PB<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>petabyte<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>petabytes<span class="pl-pds" style="box-sizing: border-box;">"</span></span>),
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">EXA_BYTES</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1_000_000_000_000_000L</span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>EB<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>exabyte<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>exabytes<span class="pl-pds" style="box-sizing: border-box;">"</span></span>),
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">ZETTA_BYTES</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">1_000_000_000_000_000_000L</span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>ZB<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>zettabyte<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>zettabytes<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
</pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
You can also specify the sizes with built-in functions if you don't want to use strings.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#using-built-in-functions-to-create-sizes" id="user-content-using-built-in-functions-to-create-sizes" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Using built-in functions to create sizes.</h4>
<div class="highlight highlight-source-js" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"> diskVolumes<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> [<span class="pl-en" style="box-sizing: border-box; color: #795da3;">kilobytes</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">10</span>), <span class="pl-en" style="box-sizing: border-box; color: #795da3;">megabytes</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">10</span>), <span class="pl-en" style="box-sizing: border-box; color: #795da3;">bytes</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">10</span>), <span class="pl-en" style="box-sizing: border-box; color: #795da3;">gigabytes</span>(<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">10</span>)]</pre>
</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#loading-config-files-with-fallbacks" id="user-content-loading-config-files-with-fallbacks" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Loading config files with fallbacks</h2>
<div class="highlight highlight-source-java" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import static</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.config.ConfigLoader.*</span>;
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">private</span> <span class="pl-smi" style="box-sizing: border-box;">Config</span> config;
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
config <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> configs(config(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>test-config.js<span class="pl-pds" style="box-sizing: border-box;">"</span></span>), config(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>reference.js<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
</pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
You can load config. The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">config</code> method is an alias for <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">load(resources...)</code>. The <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">configs(config...)</code> creates a series of configs where the configs are search from left to right. The first config that has the object (starting from the left or 0 index) will return the object.</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
Give the following two configs (from the above example).</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#test-configjs" id="user-content-test-configjs" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>test-config.js</h4>
<div class="highlight highlight-source-js" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">var</span> config <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> {
abc <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>abc<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,</pre>
</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#referencejs" id="user-content-referencejs" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>reference.js</h4>
<div class="highlight highlight-source-js" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">var</span> config <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> {
abc <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>abcFallback<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
def <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>def<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
}</pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
You could run this test.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#testing-the-referencejs-is-a-fallback-for-test-configjs" id="user-content-testing-the-referencejs-is-a-fallback-for-test-configjs" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Testing the reference.js is a fallback for test-config.js.</h4>
<div class="highlight highlight-source-java" style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">import static</span> <span class="pl-smi" style="box-sizing: border-box;">io.advantageous.config.ConfigLoader.*</span>;
<span class="pl-c1" style="box-sizing: border-box; color: #0086b3;">...</span>
config <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> configs(config(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>test-config.js<span class="pl-pds" style="box-sizing: border-box;">"</span></span>), config(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>reference.js<span class="pl-pds" style="box-sizing: border-box;">"</span></span>));
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">String</span> value <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getString(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>abc<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
assertEquals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>abc<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, value);
<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">final</span> <span class="pl-smi" style="box-sizing: border-box;">String</span> value1 <span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">=</span> config<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">.</span>getString(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>def<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);
assertEquals(<span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>def<span class="pl-pds" style="box-sizing: border-box;">"</span></span>, value1);</pre>
</div>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
You can load your config anyway you like. The String <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">abc</code> is found when looking up the key <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">abc</code> because it is in the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">test-config.js</code> which gets read before the value <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">abcFallback</code> which is in <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">reference.js</code>. Yet the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">def</code> key yields the <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">"def"</code>because it is defined in <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">reference.js</code> but not <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">test-config.js</code>. You can implement the same style config reading and fallback as is in Type Safe Config but with your DSL.</div>
<h4 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.25em; line-height: 1.4; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/advantageous/konf#thanks" id="user-content-thanks" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Thanks</h4>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">
If you like our configuration project, please try our <a href="https://github.com/advantageous/reakt" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">Reactive Java project</a> or our <a href="https://github.com/advantageous/qbit" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">Actor based microservices lib</a>.</div>
RickHighhttp://www.blogger.com/profile/15451049723511388409noreply@blogger.com0