Then came the concept of EBS backed instances. In those days we were still experimenting and HBase was releasing new versions very frequently. We were already few versions ahead pf our original AMI for Hadoop and HBase. We were also in the process of tuning our HBase/Hadoop cluster. The process of documenting all the changes after the changes are done to the installation or creating a new image everytime you changed something was very cumbersome. Instead, we thought if we converted our nodes to EBS backed instances, we won’t have to do any of it. We simply have to take a snapshot of the root device and then restore it incase the volume fails.

And this worked happily for few months. One day it suddenly stopped working.

There are many wayas to restore EBS backed instances from their snapshots. Here are all of the ways I knew:
1) Register the snapshot as an AMI and start an instance from the image.
2) Create a volume from your snapshot. Start a similar EBS backed instance, stop the instance and swap the root device.
3) Create an AMI from a running instance. This causes the instance to reboot immediately. It wasn’t an option for us. There is no way were could afford to reboot our master!

You have to know kernel and ramdisk ids if you want to go for option 1 and 2. You may think it’s a no brainer – just use the meta data query tool and find out kernel and ramdisk of the running instances. But not all instances have that meta data available to them! Our instances did not have a ramdisk meta data available! When we contacted Amazon support they told us that the instance is very old and there is simply no way to know which ramdisk it is using. That means you need to choose a ramdisk yourself. If the kernel or ramdisk you are using to create AMI from the snapshot is not compaitable, your instance will not boot up correctly. And this is especially true in case of Ubuntu images.

That’s what happened with us. It stopped working – somehow the kernel files were not available. Even though ramdisk information was not available, it was the kernel that caused us a problem. Here is what Amazon support had to say on our problems:

“Your practice of taking snapshots and starting instances from those machines can work, as it has in the past, but will always be susceptible to kernel/ramdisk mismatches.”

“Our standard practice of creating an image (AMI) from a running instance (option 3 as described above) and launching instances from that AMI would avoid the problem you’re seeing with the mismatched/incompatible kernels.”

When we told Amazon that it’s not an option for us as it causes the instance to reboot immidiately, here is what they suggested:

“Have you considered writing data to an EBS volume that is separate from your root EBS volume? I’m just wondering if that’s a viable option as it wouldn’t require stopping or rebooting the instance.”

There lies the answer! We have a requirement of recreating the cluster in case we accidently delete entire data or if we loose our master. In such a case the reliable backup can only be taken if your HDFS data does not reside on the root devices. A reliable backup of the root device cannot be taken without rebooting the device. Furthermore it’s stored as an AMI which mean you have to create a new AMI every day and delete the old one. This means to solve all of our problems we need HBase installation and data both stored on attached EBS volumes that are not the root devices.

It was news to us.

We had no choice. We decided to invest time to convert our architecture to use attached EBS volumes rather than waking up in the middle of a night and realizing that we are not able to restore our backup!

It’s a python script. Change the extension to be .py. Ubuntu comes with python installed. You will need to install aws python library boto. To install boto all ob ubuntu you can simply execute

sudo apt-get install python-boto

You will need to change the following lines with your aws access key and aws secret key. Replace MY_ACCESS_KEY_HERE with your access key and MY_SECRET_KEY_HERE with your secret key.

# Substitute your access key and secret key here
aws_access_key = 'MY_ACCESS_KEY_HERE'
aws_secret_key = 'MY_SECRET_KEY_HERE'

Here is how you can use the script:
The script takes three arguments:
1) volume-id (required) – This is Amazon’s volume id
2) number of snapshots to preserve (required) – An integer. If you specify 2 it will keep 2 newest snapshots (including the one it just created). If you specify 0, the script will delete all the snapshots (including the one it just created).
3) description (optional) – Description you want to use for your snapshot.

Example:
python manage_snaphots.py vol-dkls343e 2 'log server daily snapshot'

This execution will create a snapshot of vol-dkls343e and delete anything but that snapshot and the one before it. It will make sure that only two latest snapshots are kept. You can simply add a cron per volume.

SDK or a java api is very much needed – especially if you are writing your automation scripts in groovy. We have tried multiple java apis in our scripts including JetS3t and Typica. These apis were really helpful, but they only supported some of the AWS services and were not up to date (for obvious reasons). Having one java api that can support all of AWS technologies was definitely the need of the hour. I am sure Amazon will keep it updated as new services are released. They have the necessary resources to do so.

Furthermore, Amazon has also uploaded the SDK to the maven repository.
You can use the following dependency in your pom.xml:

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk</artifactId>
    <version>1.0.002</version>
</dependency>

The java doc for the SDK is hosted at http://docs.amazonwebservices.com/AWSJavaSDK/latest/javadoc/index.html

Amazon has also opened the SDK source code for all. They have mirrored the SDK code repository at github. You can look at the SDK code at http://github.com/amazonwebservices/aws-sdk-for-java

Keep machines in the same availability zone
Don’t scatter your machines that talk to each other across multiple availability zones. You will end up paying for the bandwidth. Off course this does not apply to the people who purposely keep their machines in different availibilty zones to ensure high availibility.

Use cnames instead of A records
In other words do not map your domain name to an elastic ip. Map it to the public domain name of the instance. Let’s say you have a machine with name splunk.gumgum.com. All the web servers (within the same availability zone) send considerable data to this machine. If you setup splunk.gumgum.com as an A record, your data will go out and come back in. But if you map it as cname, your data will always remain within the ec2 cluster. To read more about this visit Eric Hammond’s Using Elastic IP to Identify Internal Instances on Amazon EC2 post.

Use spot instances
Use spot instances whenever possible. Spot instances cost less than on demand instances. You can read more about spot instances on Amazon’s web page. There is a website that shows spot history price graphs. It will tell you an average price of various types of instances. The average price of a m1.small sport instance is 3 cents an hour. You pay 8.5 cents an hour for the same type of on demand instance.

Choose your instance types wisely
A smallest possible instance may not be the right choice to save money. For example c1.medium costs twice as much as m1.small. But it offers 5 times more compute power than m1.small. Thus in some compute heavy jobs it might be cheaper to use c1.medium instead of the least expensive m1.small.

Choose the smallest possible storage
Choose the smallest possible EBS volumes and RDS instances. It is very easy to expand an ebs volume when needed. Same is the case with a RDS instance. Furthermore by choosing smaller drives, you will save money needed for the backup(snapshot) storage too.

Use Autoscaling
Autoscaling can start additional instances when needed. It can shut down the additional instances when the need disappears. This is especailly useful for hosting websites/web services that have a specific traffic/usage pattern.

Reserve your instances
You can reduce your costs significantly by reserving number of instances for a year or for three years. Reservation is only a billing concept and thus you don’t have to designate any istances as reserved. If you reserve 4 m1.small instances, Amazon will make 4 of your running m1.small instances as reserved. For more information visit Werner Vogels’s (Amazon CTO) blog.

There might be numerous other ways you can save money. Please feel free to leave a comment to share your ways. The more the merrier!

First detach the volume and take a snapshot of the volume. If you already have a snapshot and haven’t made any changes after that, no need to take a new snapshot.

No create a new volume from this snapshot. Let’s say your original volume was 10G and you wanted to increase its size to 20G.

ec2-create-volume -z us-east-1a --size 20 --snapshot snap-xxxxxxxx

The command will output id of the new volume. Use that id in the following command to attach the volume to any instance you have up.

ec2-attach-volume vol-yyyyyyyy --instance i-xxxxxxxx --device /dev/sdh

Make sure that ec2-describe-volumes show the state of the instance as attached. Once the volume is attached, mount it at a path say /mnt/data. I am assuming here that your volume had ext3 filesystem.

mkdir /mnt/data
echo '/dev/sdh /mnt/data ext3 defaults,noatime 0 0' >> /etc/fstab
mount /mnt/data

Mounting the volume is necessary as otherwise you wont’ be able to execute the following (resize2fs) command.
Now execute the following command to resize the volume

resize2fs /dev/sdh

You should see the following output (sizes you see might be different):

Filesystem at /dev/sdh is mounted on /mnt/data; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 2
Performing an on-line resize of /dev/sdh to 5505024 (4k) blocks.
The filesystem on /dev/sdh is now 5505024 blocks long.

The command should execute in a minute or two for 20G size. That’s it, you can verify your new size by executing:

df -h

This should show you that the volume’s new size. That’s it!

We have found a solution. Ylastic provides an amazing and inexpensive web based interface for various AWS services. It allows you to name your ec2 instances and remembers it. If you have an autoscaling setup it will automatically name your autoscaled instances when they start e.g. webserver001, webserver002 etc.

Ylastic does much more than just naming the servers. It has a very user friendly user interface and also provides an iphone website from which you can start, stop instances etc. It also has a concept of locking an instance so that you don’t accidentally terminate it.

ylastic ec2 instances table

It has nice graphs for cloudwatch data.

ylastic cloudwatch graphs

It supports most of the AWS services such as EC2, RDS, Auto scaling, S3, Simple DB.

ylastic dashboard

It also lets you schedule tasks such as taking a snapshot of your ebs volumes etc. It keep tabs on your instances by receiving alerts when the state of the launched instance changes.

Ylastic costs only $25 per month per Amazon account. It allows you to create multiple users for the same Amazon account. Furthermore ylastic guys are amazingly fast. Their customer service usually comes back with an answer within the first hour. We have hardly had any complaints though. Most of the times we have suggested new features and these people have implemented them in few days.

I totally recommend ylastic. We are very happy with it at Bedrock. It has made our life easier.

Make sure that you have the latest ec2 api tools before you start creating and ebs type of image. Support for some of the commands (such as to register a snapshot as an image) is only available in the version 46266 onwards.

1) Launch an ec2 instance from the image you want to convert to an ebs image. Make the changes (if any) you want to make to this instance. Let’s say that your instance id is i-xxxxxxxx

2) Launch an 10G ebs volume and attach it to this instance. You can create a bigger volume too. But currently the default size of Amazon’s ephemeral storage is 10G and hence a 10G volume should be sufficient for most of the people.
The following code will launch an ebs volume and attach it to your instance. The code assumes that us-east-1a is the availability zone of your instance.

ec2-create-volume --size 10 --availability-zone us-east-1a

The command should give you an volume id in the form of vol-yyyyyyyy. Note it down and use it in the next command to attach this volume to your instance.

ec2-attach-volume vol-yyyyyyyy --instance i-xxxxxxxx --device /dev/sdh

Create ext3 filesystem on this volume and mount the volume

yes | mkfs -t ext3 /dev/sdh
mkdir /mnt/ebsimage
echo '/dev/sdh /mnt/ebsimage ext3 defaults,noatime 0 0' >> /etc/fstab
mount /mnt/ebsimage

3) Now that the volume is ready and mounted at /mnt/ebsimage, we are going to copy your entire disk to this volume using dd (a linux disc imaging utility).

nohup dd if=/dev/sda1 of=/dev/sdh &

The command above will start copying the disc to the ebs volume in background. This could take really long time (e.g. 20 minutes) depending upon your data. To check status of copying you will first need to get pid of the dd process by executing

ps -aef | grep dd

And then use the pid obtained in the following command

kill -SIGUSR1 pid

The above command will clearly print a message similar to

531694080 bytes (532 MB) copied, 11.6338 seconds, 45.7 MB/s

Please note that the message is printed by the dd process and not by the kill process. This means if you have your output redirected to somewhere else, the message will be printed there instead of your screen.

3) Once dd finishes, do a spot check on the data copied in /mnt/ebsimage. If you find everything ok, unmount the volume and detach the volume from the ec2 instance. You can use the following commands to do so:

umount /mnt/ebsimage
ec2-detach-volume vol-yyyyyyyy --instance i-xxxxxxxx


4) Now take the snapshot of the ebs volume.

ec2-create-snapshot vol-yyyyyyyy


5) Register this snapshot as an image. Use the snapshot id obtained from the output of the above command.

ec2reg -s snap-zzzzzzzz -a x86_64 -d Description -n imagename

Make sure that you replace the snapshot id, image name, description and the image type (x86_86 or x86_64) in the command above.

That’s it! Your ebs image is ready. Simply launch your instance from this image exactly the same way you launch any other ec2 instance!