How to provision a local VM and remote amazon EC2 instance with the same Chef and Vagrant setup

In the previous article we learned how to create a local virtual machine for development and a similar live server on Amazon EC2 with Vagrant.

That helped us setup the servers and get going with Vagrant but we didn’t install anything on them. So let’s do that now!

First a recap of the tools we are using:

Vagrant – The glue that holds the whole process together. Vagrant co-ordinates virtualisation providers to manage virtual servers and provisioners that will then load apps and tweak settings on those servers.

Virtualbox – The virtualisation provider for local virtual machines. Also the default for vagrant, but other providers can be used.

Vagrant EC2 plugin – The link to the virtualisation provider for servers on the Amazon EC2 platform.

Chef – The tool to add applications, modules and config files to the server that is controlled by Vagrant. The provisioner.

The good thing about this toolset is they all abstract their work domain well. Vagrant can work with different virtualisation providers, such as Virtualbox or VMware. It can use different provisioners such as Chef or Puppet. With whatever combinations you still use the same vagrant instructions to work – vagrant up, vagrant destroy, vagrant provision, vagrant shh.

Chef abstracts the provisioning process so the same Chef configuration can be used for whatever type of server you wish to cook up (sorry!). In theory this is true, but in practise it may need a bit of OS specific config here and there. To be fair this stuff is HARD so sometimes you have to be aware that you have a particular strain of a certain OS. There might be a way around this but in my last setup to install a certain application on Ubuntu I had to ensure apt-get update was called before the app was installed. But I could do this with Chef, so it still keeps the philosophy of the toolset.

And the philosophy of the toolset? To be able to produce a portable and reproducible development environment.

And this is what I want to do. To be able to produce a local development server and then reproduce this server on EC2. In the previous article we created a Vagrant managed server both locally and on EC2. So here we now need to feed these servers some food – in the form of apps and config.

Our shopping list of tasks will be:

  • Install Chef
  • Setup the local chef directory structure
  • Create a git repo for the whole project (the chef tools manage cookbooks via git)
  • Add some cookbooks
  • Instruct vagrant to provision the VM with Chef to install MySQL and PHP
  • Create a custom cookbook to set the root MySQL password, create a user and database and populate from a dump file.
  • Repeat on our remote EC2 server to provision with the same setup as the development machine

 

Installing chef

First step is follow instructions here:

https://wiki.opscode.com/display/chef10/Installing+Chef+Client+and+Chef+Solo

The aim is to just install chef on your machine and go no further into configuration. We will be using chef-solo, which means all configuration will be kept and managed locally. That’s fine for this project, we can keep our config close. The other types of chef are suited for managing multiple clusters of servers which sounds like an adventure for another day.

To keep things simple we won’t even have to call chef-solo commands ourself. Vagrant will do that. The one chef tool we will have to use is called ‘knife’ which is used to carve up config files, or cookbooks to use the Chef terminology.

Installing cookbooks

Before we start let’s recap the file structure of our project so far:

Let’s start by asking chef to install php and mysql for us on our VM. To do this we have to use knife to install the cookbooks for php and mysql. We will then instruct vagrant to tell chef to run recipes from those cookbooks.

One thing to be aware of with using knife (hold the blunt end?) is it requires a git repo to work with. But we were going to put our server config into a repo anyway, so let’s do it now.

Now we can start using knives:

Ideally we would only be installing the php and mysql cookbooks but we need a few extras to smooth over. After all this stuff is tricky to do across such a wide range of platforms. The apt cookbook will ensure our Ubuntu server is up to date when we start installing, the iis and yum-epel keep the other cookbooks happy.

During the install your screen should show you knife doing lots of stuff. If you look in your cookbook directory you will see the downloaded cookbooks:

Cookbooks are packages that can have dependencies on other cookbooks. knife is clever enough to deal with these dependencies and load them for us, which accounts for the extras here (beyond our own extra’s we specified).

Getting Vagrant to read cookbooks

Now we can edit the Vagrantfile for our VM. Refer to the final copy of the file at the bottom of the article to see where to fit things in. Here we tell vagrant to use chef for provisioning and which recipes to run. We also need to tell vagrant where the chef cookbooks are:

Earlier chef was installed on your host machine so cookbooks could be downloaded, but we

also need chef to be installed on the virtual machine too. The chef client on the target machine is sometimes included in base boxes, so may already be there but that is not

guaranteed. Luckily there is a vagrant plugin that will ensure chef is installed on the target machine, and if not install it for us. To install the plugin run in your shell:

And then update your Vagrantfile to use the plugin:

Provision the local VM

Now from the vagrant_local directory tell vagrant to provision the server.

Again chef will fill your screen in green with it’s activity. Once completed you can

login and verify it’s installed mysql

Result! Chef has cooked up this stack for us.

Passing parameters to cookbook recipes

What we’ve done so far is run off the shelf cookbooks that install standard packages.

We haven’t yet told Chef about anything specific about our particular install. Cookbooks often contain multiple recipes so you can customise an install by selecting appropriate recipes. For example if we only needed MySQL client we would of left out the MySQL server recipe. The other way to customise chefs actions is to pass in cookbook parameters. There’s often a wide range of cookbook parameters which you can find detailed in the cookbook docs. Let’s start by specifying the root password for mysql (from a security point of view this is not a production solution, just a demo). We can do this by passing the value to the mysql cookbook in our Vagrantfile:

And ask vagrant to shake this change through to the VM

This command runs Chef on an already running VM. A core principle of Chef is it’s operation is idempotent – running it multiple times will result in the same outcome. In this case the root password gets updated but everything else stays the same. This is great for developing our setup, we can make small steps and test each time.

Creating custom cookbooks

So next something more adventurous. We will setup a database, user and then import data into the database from a dump file so our app has an initial state. To my knowledge this isn’t possible with the default cookbook so lets create our own cookbook to do this.

To create a new cookbook we again use knife but first we must create a separate directory to store our custom cookbooks. This must be done as some Chef commands that manage cookbooks can delete cookbooks that have not been downloaded. It also helps organise your cookbooks clearly. So from the chef directory run

Then we must tell Chef about the new cookbook directory by editing the Vagrantfile to describe cookbook locations relative to the Vagrantfile:

Now instruct knife to create an empty cookbook for us (run from the chef directory)

If you look inside the site_cookbooks directory you will see a dbsetup cookbook that is far from empty. Fortunately we don’t need to worry about most of this structure for the moment, we just need to edit the default recipe (site_cookbooks/dbsetup/recipes/default.rb):

This will instruct chef to run mysql commands to create a database and then a database user. This operation requires root permissions but we can fetch that here from the config we defined earlier. Note the database name, username and password are also pulled from the config. So best define that back in the Vagrantfile:

Also tell Chef to use the new cookbook:

And kick it off again with (from vagrant directory)

This time you might see some unpleasant red text in the Chef output:

As we created the new cookbook while the VM was running the directory could not be mounted. No problem, we can switch off and switch on again to fix:

Excellent! We can login with our new user and that user can see the new database.

Using Chef to restore a database from a dump file

If only we could fill that database with data from a database dump so our VM has data to work with out of the box. Again Chef makes that pretty simple. First we need to generate the dumpfile. As we ‘backup like a boss‘ around here use this command on whichever server contains the populated database: (substituting your db credentials):

Copy this file to site_cookbooks/dbsetup/files

Now add lines in the recipe to copy this file to the VM and restore the db (site_cookbooks/dbsetup/recipes/default.rb)

And again get chef to work

Now inspect the database to check your data is there.

I really like this setup. With our custom cookbook added to version control we have a setup that can from nothing create a VM, install core applications and also populate and configure our database. These methods can be used to setup Apache, PHP or whatever stack you require. This setup is also going to payoff for our EC2 server that we setup in the previous article. As we have done all the hard work creating the cookbook we only need to update the EC2 Vagrantfile with the cookbooks to run and the config. What’s nice here is we can use the config to set different parameters for the different environments when required.

Here’s the completed Vagrantfile for the local VM (mydomain/vagrant_local/Vagrantfile)

And here’s the complete Vagrantfile for the remote EC2 server (mydomain/vagrant_ec2/Vagrantfile)

So there we have the basics of a project that can create a local VM and similar instance on EC2; provision both with applications via the same Chef setup and deploy databases. Now it’s a matter of building on this structure to fill in the gaps and add the rest of the stack. For a start install a webserver and configure the virtual hosts files using Chef templates (maybe a future article). Also for production secure and then prepare methods to deploy codebases and databases. Happy dev-oping!

How to setup an EC2 instance and similar local development virtual machine with Virtualbox, Vagrant and Chef

I’ve finally done it and taken the plunge into the world of devops. I’ve been meaning to automate the build of my live server out for a while but recent changes to the EC2 pricing structure have given me extra motivation. Financial motivation! What I wanted to achieve was:

  • Automate creating, starting and stopping an Amazon EC2 instance using Vagrant
  • Automate creating a similar local virtual machine using Vagrant
  • Provisioning both with Chef to install packages such as Apache, MySQL, etc
  • Deploy base codebases and databases for all my sites

The holy grail for me would be to run one command and bang! – an EC2 instance with all my sites would appear. Then run another command and boom! – a local virtual machine would appear with all the sites running locally for development. And of course all the deployment and setup would be shared so there would be no duplication.

There were many problems found along the way pursuing this dream but in the end it turns out Virtualbox, Vagrant and Chef can deliver the goods. And deploy the goods. And provision the goods!

The benefits for this process are plenty:

  • Recreate the environment quickly and easily, recreate locally.
  • Test changes to the environment locally then deploy live.
  • Migration to another OS is simple. Where possible the tools are platform agnostic and where this is not possible platform specific work arounds can be implemented.
  • This toolset is widely accepted so would be simple to migrate to another hosting platform
  • All config is kept in one place under version control. It’s entirely possible to work on all the config files like virtual host files, db config in your favourite IDE locally and deploy changes via Chef so you don’t need to fiddle around with vi inside a ssh tunnel. (Although I do like that type of thing!)

Creating a local development virtual machine with Vagrant

So to get started we need Vagrant https://docs.vagrantup.com/v2/installation/ and also Virtualbox if you don’t already have it.

With these in place we can start to create the configuration for our local virtual machine

vagrant init creates a file called Vagrantfile that defines everything Vagrant needs to create the VM and later the details for Chef to know how to provision your new server. The file is heavily commented with sensible defaults to help us when we need to start tweaking.

So first thing we need to consider is the ‘base box’ to use. This is the base flavour of the OS and where our journey starts. Kinda like an install DVD. Or ISO. Normally this is a matter of choosing a base box to match the intended OS, eg CentOS, Debian, Ubuntu. However we want to create a server on Amazon EC2, so we must choose an image that is both available as a Vagrant base box and as an EC2 AMI (the EC2 equivalent of a Vagrant base box)

I was already planning to run Ubuntu so our next job is to find a menu of base boxes and AMI’s.

Luckily there are excellent online resources for Ubuntu. EC2 AMI’s are listed here http://cloud-images.ubuntu.com/locator/ec2/ and Vagrant boxes listed here https://vagrantcloud.com/ubuntu

Ubuntu Server 14.04 LTS is the best match here, so let’s configure our Vagrantfile for the local VM. Fire up your favourite editor and amend the Vagrantfile to use this box and to setup networking:

Then to start the vm run this in the same directory as the Vagrantfile

Vagrant will now download the base box and instruct Virtualbox to create and start the VM. Once it’s completed we can login to our new local development server

So that’s our local development server created with a single command. Later we will introduce Chef to install applications like MySQL and Apache. Here’s the full Vagrantfile for the local VM:

(mydomain/vagrant_local/Vagrantfile)

Creating a remote EC2 server with Vagrant

Next to setup the live EC2 server. For this we need to start with installing a Vagrant plugin to do the talking with EC2.

And then setup the Vagrant environment

And again we have to configure the new Vagrantfile. But before we can we have do some some work in our AWS Management Console.

We need to:

  • Setup an IAM access key to allow Vagrant to talk to EC2
  • Setup a SSH key pair to use to login to the EC2 instance once it’s created
  • Choose the location to launch the instance
  • Setup a security group for the instance
  • Choose the type of instance
  • Choose the AMI

That sounds like a lot to do! And there’s more, and this was a real gotcha for me. I wanted to take advantage of the new t2.micro pricing. It’s cheaper AND better spec than the t1.micro. No brainer. However it turns out that t2 instances only run under Amazon VPC. I thought this would be end of the road, with either VPC not working with the vagrant-ec2 plugin or it costing too much. Turns out VPC has no cost and it does work with vagrant-ec2. Phew!

So the final item for the AWS Management Console list is:

  • Setup a VPC network

So off to work. To obtain the IAM access key follow the instructions here:

http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html

You will end up with an Access key ID (example: AKIAIOSFODNN7EXAMPLE) and Secret access key (example: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY) These keys identify the vagrant-ec2 plugin with an Amazon IAM user/role/group. In the IAM console you set the permission policy to only let these keys access what is neccessary to create, boot and halt an instance. However as we are keen to get started we can allow all actions by creating a group assigned to our user with the following policy:

With this in place enter the key details into your Vagrantfile (the full Vagrantfile is listed at the end of the article, refer to it to see where to insert each snippet)

It’s probably a good idea next to decide which EC2 region you want your server to live in. We have much to do in the EC2 console so should make sure we are setting things up in the right region. I’ve selected ‘US East’ for my example.

Next task is the ssh keys to access the instance once it’s created. This is not managed in the IAM console but the EC2 console, selecting ‘Key Pairs’ from the navigation menu. Once your keypair is setup, enter the details in the Vagrantfile

Amazon Ubuntu AMI’s require ssh using the ubuntu user, which we specify with the ‘override.ssh.username’ parameter.

Now we need to setup the VPC, as this needs to be in place for the other items on our todo list. Again in the EC2 console select ‘Network Interfaces’ from the navigation menu and create a VPC network interface. Vagrantfile:

Then select ‘Security Groups’ from the navigation menu and create a security group for the VPC. At least add SSH, HTTP and HTTPS inbound rules for a web server. More food for our Vagrantfile. Note you must use the ‘Group ID’ value:

Now the instance type. I already know I want the cheap one:

And the AMI. In this example we want 64 bit trusty tahr for us-east-1, to fit the VPC. But which type? Turns out for the t2.micro instance we must have the ‘hvm’ type. The list at http://cloud-images.ubuntu.com/locator/ec2/ leads us to ami-9aaa1cf2 which we can enter, along with your region

Then create an elastic IP (select the VPC type) for the instance and enter it here:

Finally we have to set the base box. As said before Amazon doesn’t use Vagrant base boxes but it’s own AMI’s, but vagrant still needs a base box to do it’s stuff. So we specify a dummy box that is built to work with the Vagrant-EC2 plugin:

Now we are set. Time to see the vagrant-ec2 plug work it’s magic (note the extra provider option calling Vagrant to tell it to talk to EC2)

Check in the EC2 console to see the instance spark into life. In my testing the elastic IP didn’t always connect so I needed to connect it by hand, but that’s a small step to put right.

Again once booted we can login

Another win, we are now half way to our objective. A live EC2 server and corresponding local development server all controlled via Vagrant. In true devops style you can create, start, stop, destroy and recreate repeatedly from the same config. Infact I suggest you next run

just because you can

As promised the Amazon EC2 Vagrantfile in full:

Next article: Use Chef to provision both the local VM and the EC2 instance. It’s nice to have created these servers but they really need applications installing so they can do some work for us!