Setting up Ansible for our Cassandra Database Cluster to do DevOps/DBA tasks

Setting up Ansible for our Cassandra Database Cluster to do DevOps/DBA tasks

Ansible is an essential DevOps/DBA tool for managing backups and rolling upgrades to the Cassandra cluster in AWS/EC2. An excellent aspect of Ansible is that it uses ssh, so you do not have to install an agent to use Ansible.

This article series centers on how DevOps/DBA tasks with the Cassandra Database. However the use of Ansible for DevOps/DBA transcends its use with the Cassandra Database so this article is good information for any DevOps/DBA or Developer that needs to manage groups of instances, boxes, hosts whether they be on-prem bare-metal, dev boxes, or in the Cloud. You don’t need to be setting up Cassandra to get use of this article.

Cassandra Tutorial Series on DevOps/DBA Cassandra Database

The first article in this series was about setting up a Cassandra cluster with Vagrant (also appeared on DZone with some additional content DZone Setting up a Cassandra Cluster with Vagrant. The second article in this series was about setting up SSL for a Cassandra cluster using Vagrant (which also appeared with more content as DZone Setting up a Cassandra Cluster with SSL). You don’t need those articles to follow along, but they might provide a lot of contexts. You can find the source for the first and second article at our Cloudurable Cassandra Image for Docker, AWS, and Vagrant. In later articles, we will use Ansible to create more complicated playbooks like doing a rolling Cassandra upgrade, and we will cover using Ansible/ssh with AWS EC2.

Source code for Vagrant, and ansbile

We continue to evolve the cassandra-image GitHub project. In an effort for the code to match the listings in the article, we created a new branch where the code was when this article was written (more or less): Article 3 Ansible Cassandra Vagrant.

Let’s get to it. Let’s start by creating a key for our DevOps/DBA test Cassandra cluster.

Create key for test cluster to do Cassandra Database DevOps/DBA tasks with Ansible

To use Ansible for DevOps/DBA, we will need to setup ssh keys as Ansible uses ssh instead of running an agent on each server like Chef and Puppet.

The tool ssh-keygen manages authentication keys for ssh(secure shell). The utility ssh-keygen generates RSA or DSA keys for SSH (secure shell) protocol version 1 and 2. You can specify the key type with the -t option.

setup key script bin/setupkeys-cassandra-security.sh

CLUSTER_NAME=test
...
ssh-keygen -t rsa -C "your_email@example.com" -N "" -C "setup for cloud" \
 -f "$PWD/resources/server/certs/${CLUSTER_NAME}_rsa"

chmod 400 "$PWD/resources/server/certs/"*
cp "$PWD/resources/server/certs/"* ~/.ssh
...

Let’s break that down.

We use ssh-keygen to create a private key that we will use to log into our boxes.

In this article those boxes are Vagrant boxes (VirtualBox), but in the next article, we will use the same key to manage EC2 instances.

Use ssh-keygen to create private key for ssh to log into Cassandra Database nodes

ssh-keygen -t rsa -C "your_email@example.com" -N "" -C "setup for cloud" \
 -f "$PWD/resources/server/certs/${CLUSTER_NAME}_rsa"

Then we restrict the access to the file of the key otherwise, ansible, ssh and scp (secure copy) will not let us use it.

Change the access of the key

chmod 400 "$PWD/resources/server/certs/"*

The above chmod 400 changes the cert files so only the owner can read the file. This file change mod makes sense. The certification file should be private to the user (and that is what 400 does).

Copy keys to area where it will be copied by Cassandra node provisioning

cp "$PWD/resources/server/certs/"* ~/.ssh

The above just puts the files where our provisioners (Packer and Vagrant) can pick them up and deploy them with the image.

Locally we are using Vagrant to launch a cluster to do some tests on our laptop.

We also use Packer and aws command line tools to create EC2 AMIs (and Docker images), but we don’t cover aws in this article (it is in the very next which is sort of part 2 to this article).

Create a bastion server to do ansible DevOps/DBA tasks for Cassandra Cluster

Eventually, we would like to use a bastion server that is a public subnet to send commands to our Cassandra Database nodes that are in a private subnet in EC2. For local testing, we set up a bastion server, which is well explained in this guide to Vagrant and Ansible.

We used Learning Ansible with Vagrant (Part 2⁄4) as a guide for some of the setup performed in this article. It is a reliable source of Ansible and Vagrant knowledge for DevOps/DBA. Their mgmt node corresponds to what we call a bastion server. A notable difference is we are using CentOS 7 not Ubuntu, and we made some slight syntax updates to some of the Ansible commands that we are using (we use a later version of Ansible).

We added a bastion server to our Vagrant config as follows:

Vagrantfile to set up the bastion for our Cassandra Cluster

  # Define Bastion Node
  config.vm.define "bastion" do |node|
            node.vm.network "private_network", ip: "192.168.50.20"
            node.vm.provider "virtualbox" do |vb|
                   vb.memory = "256"
                   vb.cpus = 1
            end


            node.vm.provision "shell", inline: <<-SHELL
                yum install -y epel-release
                yum update -y
                yum install -y  ansible

                mkdir /home/vagrant/resources
                cp -r /vagrant/resources/* /home/vagrant/resources/

                mkdir -p ~/resources
                cp -r /vagrant/resources/* ~/resources/

                mkdir  -p  /home/vagrant/.ssh/
                cp /vagrant/resources/server/certs/*  /home/vagrant/.ssh/

                sudo  /vagrant/scripts/002-hosts.sh

                ssh-keyscan node0 node1 node2  >> /home/vagrant/.ssh/known_hosts


                mkdir ~/playbooks
                cp -r /vagrant/playbooks/* ~/playbooks/
                sudo cp /vagrant/resources/home/inventory.ini /etc/ansible/hosts
                chown -R vagrant:vagrant /home/vagrant
            SHELL

The bastion server which could be on a public subnet in AWS in a VPC uses the ssh-keyscan to add nodes that we setup in the host file into /home/vagrant/.ssh/known_hosts.

Running ssh-keyscan

ssh-keyscan node0 node1 node2  >> /home/vagrant/.ssh/known_hosts

This utility is to get around the problem of needing to verify nodes, and getting this error message: The authenticity of host ... can't be established. ... Are you sure you want to continue connecting (yes/no)? no when we are trying to run ansible command line tools.

Modify the Vagrant provision script

Since we are using provision files to create different types of images (Docker, EC2 AMI, Vagrant/VirtualBox), then we use a provisioning script specific to vagrant.

In this vagrant provision script, we call another provision script to setup a hosts file.

000-vagrant-provision.sh

mkdir  -p  /home/vagrant/.ssh/
cp /vagrant/resources/server/certs/*  /home/vagrant/.ssh/
...

scripts/002-hosts.sh
echo RUNNING TUNE OS

Setting up sshd on our Cassandra Database nodes in our DevOps Cassandra Cluster

The provision script 002-hosts.sh configures /etc/ssh/sshd_config/sshd_config to allow public key auth. Then it restarts the daemon for ssh communication sshd. (The other provisioning scripts it invokes was covered in the first two articles).

Let’s look at the 002-hosts.sh provision script. You can see some remnants from the last article where we setup csqlsh, and then it gets to business setting up sshd (secure server shell daemon).

scripts/002-hosts.sh - sets up sshd and hosts file

#!/bin/bash
set -e



## Copy csqlshrc file that controls csqlsh connections to ~/.cassandra/cqlshrc.
mkdir ~/.cassandra
cp ~/resources/home/.cassandra/cqlshrc ~/.cassandra/cqlshrc

## Allow pub key login to ssh.
sed -ie 's/#PubkeyAuthentication no/PubkeyAuthentication yes/g' /etc/ssh/sshd_config

## System control restart sshd daemon to take sshd_config into effect.
systemctl restart sshd

# Create host file so it is easier to ssh from box to box
cat >> /etc/hosts <<EOL

192.168.50.20  bastion

192.168.50.4  node0
192.168.50.5  node1
192.168.50.6  node2
192.168.50.7  node3
192.168.50.8  node4
192.168.50.9  node5
EOL

This setup is fairly specific to our Vagrant setup at this point. To simplify access to the servers that hold the different Cassandra Database nodes, the 002-hosts.sh creates an \etc\hosts\ file on the bastion server.

With our certification keys added to sshd config and our hosts configured (and our inventory.ini file shipped), we can start using ansible from our bastion server.

This reminds me, we did not talk about the ansible inventory.ini file.

Ansible config on bastion for Cassandra Database Cluster

Ansible has a ansible.cfg file, and an inventory.ini file. When you run ansible, it checks for ansible.cfg in the current working directory, then your home directory, and then for a master config file (/etc/ansible). We created an inventory.ini file which lives under ~\github\cassandra-image\resources\home, which gets mapped to \vagrant\resources\home on the virtual machines (node0, bastion, node1, and node2). A provision script moves the inventory.ini file to its proper location (sudo cp /vagrant/resources/home/inventory.ini /etc/ansible/hosts).

The inventory.ini contains servers that you want to manage with Ansible. A couple of things are going on here, we have a bastiongroup, this is for our bastion server, next we have the nodes group, and it is made up of node0, node1, and node2.

Let’s see what the inventory.ini file actually looks like.

inventory.ini that gets copied to Ansible master list on Bastion

[bastion]
bastion


[nodes]
node0
node1
node2

Once we provision our cluster, we can log into bastion and start executing ansible commands.

Installing cert key for test DevOps/DBA Cassandra Cluster on all nodes using an ansible playbook

To make this happen, we had to tell the other servers about our certification keys.

We did this with an ansible playbook as follows:

Ansible playbook getting invoked from Vagrant on each new Cassandra Database node

Vagrant.configure("2") do |config|


  config.vm.box = "centos/7"


  # Define Cassandra Nodes
  (0..numCassandraNodes-1).each do |i|

        port_number = i + 4
        ip_address = "192.168.50.#{port_number}"
        seed_addresses = "192.168.50.4,192.168.50.5,192.168.50.6"
        config.vm.define "node#{i}" do |node|
            node.vm.network "private_network", ip: ip_address
            node.vm.provider "virtualbox" do |vb|
                   vb.memory = "2048"
                   vb.cpus = 4
            end
            ...

            node.vm.provision "ansible" do |ansible|
                  ansible.playbook = "playbooks/ssh-addkey.yml"
            end
        end
  end

Notice the line node.vm.provision "ansible" do |ansible| and ansible.playbook = "playbooks/ssh-addkey.yml".

If you are new to Vagrant and the above just is not making sense, please watch Vagrant Crash Course. It is by the same folks (guy) who created the Ansible series.

Ansible playbooks are like configuration playbooks. You can perform tons of operations that are important for DevOps (like yum installing software, specific tasks to Cassandra, etc.).

Playbooks are Ansible’s configuration, deployment, and orchestration language. They can describe a policy you want your remote systems to enforce, or a set of steps in a general IT process. –Ansible Playbook documentation.

Here is the ansible playbook to add the RSA public key to the cassandra nodes as follows.

Ansible playbook ssh-addkey.yml to add test_rsa.pub to all Cassandra Database node servers

---
- hosts: all
  become: true
  gather_facts: no
  remote_user: vagrant

  tasks:

  - name: install ssh key
    authorized_key: user=vagrant
                    key="{{ lookup('file', '../resources/server/certs/test_rsa.pub') }}"
                    state=present

The trick here is that Vagrant supports running Ansible playbooks as well.

The Vagrant Ansible provisioner allows you to provision the guest using Ansible playbooks by executing ansible-playbook from the Vagrant host. –(Vagrant Ansible documentation)[https://www.vagrantup.com/docs/provisioning/ansible.html]

For users who did not read any of the first articles on setting up the Cassandra Cluster

If you have not done so already navigate to the project root dir (which is ~/github/cassandra-image on my dev box), download the binaries. The source code is at Github Cassandra Image project.

Running setup scripts

## cd ~; mkdir github; cd github; git clone https://github.com/cloudurable/cassandra-image
$ cd ~/github/cassandra-image
$ pwd
~/github/cassandra-image
## Setup keys
$ bin/setupkeys-cassandra-security.sh
## Download binaries
$ bin/prepare_binaries.sh
## Bring Vagrant cluster up
$ vagrant up

Even if you read the first article note that bin/prepare_binaries.sh is something we added after the first two articles. It downloads the binaries needed for the provisioning, does a checksum of the files and then installs them as part of the provisioning process.

Where do you go if you have a problem or get stuck?

We set up a google group for this project and set of articles. If you just can’t get something to work or you are getting an error message, please report it here. Between the mailing list and the github issues, we can support you with quite a few questions and issues.

Running ansible commands from bastion

Let’s log into bastion and run ansible commands against the cassandra nodes.

Working with ansible from bastion and using ssh-agent

$ vagrant ssh bastion

So we don’t have to keep logging in, and passing our cert key, let’s start up an ssh-agent and add our cert key ssh-add ~/.ssh/test_rsa to the agent.

The ssh-agent is a utility to hold private keys used for public key authentication (RSA, DSA, ECDSA, Ed25519) so you don’t have to keep passing the keys around. The ssh-agent is usually started in the beginning of a login session. Other programs (scp, ssh, ansible) are started as clients to the ssh-agent utility.

Mastering ssh is essential for DevOps and needed for ansible.

First set up ssh-agent and add keys to it with ssh-add.

Start ssh-agent and add keys

$ ssh-agent bash
$ ssh-add ~/.ssh/test_rsa

Now that the agent is running and our keys are added, we can use ansible without passing it the RSA private key.

Let’s verify connectivity, by pinging some of these machines. Let’s ping the node0 machine. Then let’s ping all of the nodes.

Let’s use the ansible ping module to ping the node0 server.

Ansible Ping the Cassandra Database node

$ ansible node0 -m ping

Output

node0 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

To learn more about DevOps with ansible see this video on Ansible introduction. It covers a lot of the basics of ansible.

Now let’s ping all of the nodes.

Ansible Ping all Cassandra Database Cluster nodes

$ ansible nodes  -m ping

Output

node0 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
node2 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
node1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

Looks like bastion can run ansible against all of the servers.

Setting up my MacOSX to run Ansible against Cassandra Database Cluster nodes

The script ~/github/cassandra-image/bin/setupkeys-cassandra-security.sh copies the test cluster key for ssh (secure shell) over to ~/.ssh/ (cp "$PWD/resources/server/certs/"* ~/.ssh). It was Run from the project root folder which is ~/github/cassandra-image on my box.

Move to the where you checked out the project.

cd ~/github/cassandra-image

In this folder is an ansible.cfg file and an inventory.ini file for local dev. Before you use these first modify your /etc/hosts file to configure entries for bastion, node0, node1, node2 servers.

Add bastion, node0, etc. to /etc/hosts

$ cat /etc/hosts

### Used for ansible/ vagrant
192.168.50.20  bastion
192.168.50.4  node0
192.168.50.5  node1
192.168.50.6  node2
192.168.50.7  node3
192.168.50.8  node4
192.168.50.9  node5

We can use ssh-keyscan just like we did before to add these hosts to our known_hosts file.

Add keys to known_hosts to avoid prompts

$ ssh-keyscan node0 node1 node2  >> ~/.ssh/known_hosts

Then just like before we can start up an ssh-agent and add our keys.

Start ssh-agent and add keys

$ ssh-agent bash
$ ssh-add ~/.ssh/test_rsa

Notice that the ansible.cfg and inventory.ini files are a bit different than on our bastion server because we have to add the user name.

Notice the ansible.cfg file and inventory.ini file in the project dir

$ cd ~/github/cassandra-image

$ cat ansible.cfg
[defaults]
hostfile = inventory.ini

cat inventory.ini
[nodes]
node0 ansible_user=vagrant
node1 ansible_user=vagrant
node2 ansible_user=vagrant

Ansible will use these.

From the project directory, you should be able to ping node0 and all of the nodes just like before.

Ping node0 with ansible.

Ansible Ping Cassandra Database node

$ ansible node0 -m ping

Output

node0 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

Ping all of the Cassandra nodes with ansible.

Ansible Ping All Cassandra Database Cluster nodes

$ ansible nodes  -m ping

Output

node0 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
node2 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
node1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

In the next article, we cover how to setup ~.ssh/config so you don’t have to remember to use ssh-agent.

Using ansible to run nodetool on Cassandra Cluster nodes

You may recall from the first article that we would log into the servers (vagrant ssh node0) and then check that they could see the other nodes with nodetool describecluster. We could run this command with all three servers (from bastion or on our dev laptop) with ansible.

Let’s use ansible to run describecluster against all of the nodes.

Ansible running nodetool describecluster against all Cassandra Cluster nodes

$ ansible nodes -a "/opt/cassandra/bin/nodetool describecluster"

This command allows us to check the status of every node quickly.

Let’s say that we wanted to update a schema or do a rolling restart of our Cassandra cluster nodes, which could be a very common task. Perhaps before the update, we want to decommission the node and back things up. To do this sort of automation, we could create an Ansible playbook.

Ansible Playbooks are more powerful than executing ad-hoc task execution and is especially powerful for managing a cluster of Cassandra servers.

Playbooks allow for configuration management and multi-machine deployment to manage complex tasks like a rolling upgrade or schema updates or perhaps a weekly backup.

Playbooks are declarative configurations. Ansible Playbooks also orchestrate steps into a simpler task. This automation gets rid of a lot of manually ordered process and allows for an immutable infrastructure.

Our describe-cluster playbook for Cassandra Database Cluster nodes

Creating a complicated playbook is beyond the scope of this article. But let’s create a simple playbook and execute it. This playbook will run the nodetool describecluster on each node.

Here is our playbook that runs Cassandra nodetool describecluster on each Cassandra node in our cluster.

playbooks/descibe-cluster.yml - simple ansible playbook that runs Cassandra nodetool describecluster

---
- hosts: nodes
  gather_facts: no
  remote_user: vagrant

  tasks:

  - name: Run NodeTool Describe Cluster command against each Cassandra Cluster node
    command: /opt/cassandra/bin/nodetool describecluster

To run this, we use ansible-playbook as follow.

Running describe-cluster playbook

$ ansible-playbook playbooks/describe-cluster.yml --verbose

Between this article and the last, we modified our Vagrantfile quite a bit. It now uses a for loop to create the Cassandra nodes, and it uses ansible provisioning.

Here is our new Vagrantfile with updates:

Complete code listing of Vagrantfile that sets up our DevOps/DBA Cassandra Database Cluster

# -*- mode: ruby -*-
# vi: set ft=ruby :

numCassandraNodes = 3

Vagrant.configure("2") do |config|


  config.vm.box = "centos/7"


  # Define Cassandra Nodes
  (0..numCassandraNodes-1).each do |i|

        port_number = i + 4
        ip_address = "192.168.50.#{port_number}"
        seed_addresses = "192.168.50.4,192.168.50.5,192.168.50.6"
        config.vm.define "node#{i}" do |node|
            node.vm.network "private_network", ip: ip_address
            node.vm.provider "virtualbox" do |vb|
                   vb.memory = "2048"
                   vb.cpus = 4
            end


            node.vm.provision "shell", inline: <<-SHELL

                sudo /vagrant/scripts/000-vagrant-provision.sh



                sudo /opt/cloudurable/bin/cassandra-cloud -cluster-name test \
                -client-address     #{ip_address} \
                -cluster-address    #{ip_address} \
                -cluster-seeds      #{seed_addresses}

            SHELL

            node.vm.provision "ansible" do |ansible|
                  ansible.playbook = "playbooks/ssh-addkey.yml"
            end
        end
  end


  # Define Bastion Node
  config.vm.define "bastion" do |node|
            node.vm.network "private_network", ip: "192.168.50.20"
            node.vm.provider "virtualbox" do |vb|
                   vb.memory = "256"
                   vb.cpus = 1
            end


            node.vm.provision "shell", inline: <<-SHELL
                yum install -y epel-release
                yum update -y
                yum install -y  ansible

                mkdir /home/vagrant/resources
                cp -r /vagrant/resources/* /home/vagrant/resources/

                mkdir -p ~/resources
                cp -r /vagrant/resources/* ~/resources/

                mkdir  -p  /home/vagrant/.ssh/
                cp /vagrant/resources/server/certs/*  /home/vagrant/.ssh/

                sudo  /vagrant/scripts/002-hosts.sh

                ssh-keyscan node0 node1 node2  >> /home/vagrant/.ssh/known_hosts


                mkdir ~/playbooks
                cp -r /vagrant/playbooks/* ~/playbooks/
                sudo cp /vagrant/resources/home/inventory.ini /etc/ansible/hosts
                chown -R vagrant:vagrant /home/vagrant
            SHELL


  end



  #
  # View the documentation for the provider you are using for more
  # information on available options.

  # Define a Vagrant Push strategy for pushing to Atlas. Other push strategies
  # such as FTP and Heroku are also available. See the documentation at
  # https://docs.vagrantup.com/v2/push/atlas.html for more information.
  config.push.define "atlas" do |push|
     push.app = "cloudurable/cassandra"
  end


end

Conclusion

We set up Ansible for our Cassandra Database Cluster to do automate common DevOps/DBA tasks. We created an ssh key and then set up our instances with this key so we could use ssh, scp, and ansible. We set up a bastion server with Vagrant. We used ansibleplaybook (ssh-addkey.yml) from Vagrant to install our test cluster key on each server. We ran ansible ping against a single server. We ran ansible ping against many servers (nodes). We set up our local dev machine with ansible.cfg and inventory.ini so we could use ansible commands direct to node0 and nodes. We ran nodetool describecluster against all of the nodes from our dev machine. Lastly, we created a very simple playbook that can run nodetool describecluster. Ansible is a very powerful tool that can help you manage a cluster of Cassandra instances. In later articles, we will use Ansible to create more complex playbooks like backing up Cassandra nodes to S3.

Cassandra Tutorial: Cassandra Cluster DevOps/DBA series

The first tutorial in this Cassandra tutorial series focused on setting up a Cassandra Cluster. The first Cassandra tutorial setting up a Cassandra cluster with Vagrant (also appeared on DZone with some additional content DZone Setting up a Cassandra Cluster with Vagrant. The second article in this series was about setting up SSL for a Cassandra cluster using Vagrant (which also appeared with more content as DZone Setting up a Cassandra Cluster with SSL). The third article in this series was about configuring and using Ansible (building on the first two articles). This article (the 4th) will cover applying the tools and techniques from the first three articles to produce an image (EC2 AMI to be precise) that we can deploy to AWS/EC2. To do this explanation, we will use Packer, Ansible, and the Aws Command Line tools. The AWS command line tools are essential for doing DevOps with AWS.

Check out more information about the Cassandra Database

• Cassandra Consulting: Architecture Analysis
• Cassandra Consulting: Quick Start
• Cassandra Course
• Amazon Cassandra Support