Manage your OpenStack cloud with Ansible: Day two operations

Managing an application on OpenStack presents a host of challenges for the system administrator, and finding ways to reduce complexity and produce consistency are key ingredients to achieving success. By using Ansible, an agentless IT automation technology, a system administrator can create Ansible playbooks that provide consistency and reduce complexity.

OpenStack provides a rich API to manage resources that has led to the creation of dozens of Ansible modules that can easily fit into any automation workflow. Combined with the ability to automate tasks in OpenStack instances, an operator can work both inside and out to coordinate complex operations against an environment.

“Day one” operations refer to tasks that are executed during the initial configuration and deployment of an environment. The index of OpenStack modules for Ansible lists many of the common modules used to complete tasks during day one. Creating all manner of resources, such as networks, volumes, and instances are covered. “Day two” deals with what happens next:

  • How will upgrades happen?
  • How are backups maintained?
  • How does the environment scale up with demand?

Ansible can easily handle these use cases.

For example, consider a cluster of web servers that need to be upgraded, all sitting behind an OpenStack load balancer. With the ability to manage both the infrastructure and tasks within the VMs themselves, an operator can ensure the sequence of events executed always happens in a particular order. Here is a simple example of a playbook to perform a rolling upgrade:

– hosts: web
gather_facts: true
user: centos
serial: 1  # ensures only one server will update/reboot at a time
tasks:
– name: check for pending updates
yum:
list: updates
register: yum_update # check if there are updates before going any further
– block:
– name: remove web server from pool
os_member:
state: absent
name: ‘{{ ansible_hostname }}’
pool: weblb_80_pool
delegate_to: localhost
– name: update packages
package:
name: ‘*’
state: latest
become: true
– name: reboot server
shell: sleep 5 && reboot &
async: 1
poll: 0
– name: wait for server
wait_for_connection:
connect_timeout: 20
sleep: 5
delay: 5
timeout: 600
become: true
– name: put server back in pool
os_member:
state: present
name: ‘{{ ansible_hostname }}’
pool: weblb_80_pool
address: ‘{{ ansible_default_ipv4.address }}’
protocol_port: 80
delegate_to: localhost
when:
– yum_update.results | length > 0 # only execute the block if there are updates

This playbook first checks to see whether there are any updates to apply. If so, the playbook removes the node from the pool, applies the updates, and reboots the node. Once the node is back online, it gets added back into the pool. The Ansible playbook uses the serial keyword to ensure only one node is removed from the pool at a time.

If a database is running in the OpenStack cloud, occasionally a backup will have to be restored—either to refresh some test data or perhaps in the event of a data corruption incident. Orchestrating tasks between the database server and Cinder is easily accomplished with Ansible:

– hosts: db
gather_facts: true
user: centos
tasks:
– name: stop database
systemd:
name: mongod
state: stopped
become: true
– name: unmount db volume
mount:
path: /var/lib/mongodb
state: unmounted
become: true
– name: detach volume from server
os_server_volume:
state: absent
server: db0
volume: dbvol
delegate_to: localhost
– name: restore cinder backup
command: openstack volume backup restore dbvol_backup dbvol
delegate_to: localhost
register: vol_restore
failed_when:
– vol_restore.rc > 0
– “‘VolumeBackupsRestore’ not in vol_restore.stderr”
– name: wait for restore to finish
command: openstack volume show -c status -f value dbvol
register: restore_progress
until: restore_progress.stdout is search(“available”)
retries: 60
delay: 5
delegate_to: localhost
– name: reattach volume to server
os_server_volume:
state: present
server: db0
volume: dbvol
device: /dev/vdb
delegate_to: localhost
– name: mount db volume
mount:
path: /var/lib/mongodb
state: mounted
src: LABEL=dbvol
fstype: xfs
become: true
– name: start database
systemd:
name: mongod
state: started
become: true

Looking closely at the playbook, you may have noticed that the restore is done via the OpenStack command line and not a proper Ansible module. In some cases, a module for a task might not exist, but Ansible is flexible enough to allow calling arbitrary commands within a playbook until a module is developed. Feel like you could write the missing module? Consider creating it by contributing to the Ansible project.

These are just a couple of day-two operations a system administrator may need to orchestrate in their cloud. Roger Lopez and I will offer a hands-on lab at OpenStack Summit in Berlin with real-world scenarios and associated Ansible playbooks to automate them. We’ll also upload our examples and materials to GitHub the week of the conference for the benefit of anyone who can’t attend.


Roger Lopez and David Critch will present Simplifying Day Two Operations with Ansible (A Hands-on Lab) at the OpenStack Summit, November 13-15 in Berlin.

Facebook Comments
Spread the love

Posted by News Monkey