Then came the concept of EBS backed instances. In those days we were still experimenting and HBase was releasing new versions very frequently. We were already few versions ahead pf our original AMI for Hadoop and HBase. We were also in the process of tuning our HBase/Hadoop cluster. The process of documenting all the changes after the changes are done to the installation or creating a new image everytime you changed something was very cumbersome. Instead, we thought if we converted our nodes to EBS backed instances, we won’t have to do any of it. We simply have to take a snapshot of the root device and then restore it incase the volume fails.

And this worked happily for few months. One day it suddenly stopped working.

There are many wayas to restore EBS backed instances from their snapshots. Here are all of the ways I knew:
1) Register the snapshot as an AMI and start an instance from the image.
2) Create a volume from your snapshot. Start a similar EBS backed instance, stop the instance and swap the root device.
3) Create an AMI from a running instance. This causes the instance to reboot immediately. It wasn’t an option for us. There is no way were could afford to reboot our master!

You have to know kernel and ramdisk ids if you want to go for option 1 and 2. You may think it’s a no brainer – just use the meta data query tool and find out kernel and ramdisk of the running instances. But not all instances have that meta data available to them! Our instances did not have a ramdisk meta data available! When we contacted Amazon support they told us that the instance is very old and there is simply no way to know which ramdisk it is using. That means you need to choose a ramdisk yourself. If the kernel or ramdisk you are using to create AMI from the snapshot is not compaitable, your instance will not boot up correctly. And this is especially true in case of Ubuntu images.

That’s what happened with us. It stopped working – somehow the kernel files were not available. Even though ramdisk information was not available, it was the kernel that caused us a problem. Here is what Amazon support had to say on our problems:

“Your practice of taking snapshots and starting instances from those machines can work, as it has in the past, but will always be susceptible to kernel/ramdisk mismatches.”

“Our standard practice of creating an image (AMI) from a running instance (option 3 as described above) and launching instances from that AMI would avoid the problem you’re seeing with the mismatched/incompatible kernels.”

When we told Amazon that it’s not an option for us as it causes the instance to reboot immidiately, here is what they suggested:

“Have you considered writing data to an EBS volume that is separate from your root EBS volume? I’m just wondering if that’s a viable option as it wouldn’t require stopping or rebooting the instance.”

There lies the answer! We have a requirement of recreating the cluster in case we accidently delete entire data or if we loose our master. In such a case the reliable backup can only be taken if your HDFS data does not reside on the root devices. A reliable backup of the root device cannot be taken without rebooting the device. Furthermore it’s stored as an AMI which mean you have to create a new AMI every day and delete the old one. This means to solve all of our problems we need HBase installation and data both stored on attached EBS volumes that are not the root devices.

It was news to us.

We had no choice. We decided to invest time to convert our architecture to use attached EBS volumes rather than waking up in the middle of a night and realizing that we are not able to restore our backup!

It’s a python script. Change the extension to be .py. Ubuntu comes with python installed. You will need to install aws python library boto. To install boto all ob ubuntu you can simply execute

sudo apt-get install python-boto

You will need to change the following lines with your aws access key and aws secret key. Replace MY_ACCESS_KEY_HERE with your access key and MY_SECRET_KEY_HERE with your secret key.

# Substitute your access key and secret key here
aws_access_key = 'MY_ACCESS_KEY_HERE'
aws_secret_key = 'MY_SECRET_KEY_HERE'

Here is how you can use the script:
The script takes three arguments:
1) volume-id (required) – This is Amazon’s volume id
2) number of snapshots to preserve (required) – An integer. If you specify 2 it will keep 2 newest snapshots (including the one it just created). If you specify 0, the script will delete all the snapshots (including the one it just created).
3) description (optional) – Description you want to use for your snapshot.

Example:
python manage_snaphots.py vol-dkls343e 2 'log server daily snapshot'

This execution will create a snapshot of vol-dkls343e and delete anything but that snapshot and the one before it. It will make sure that only two latest snapshots are kept. You can simply add a cron per volume.

In my opinion, people who build their systems in AWS cloud should be prepared for such a failure. They need to design their systems in a way that can accommodate such failures. Here are my suggestions to minimize the damage caused by such failures:

1) Architect your system to operate without most of the components

Some pieces might force the entire system to go down, but you may realize that a lot of them don’t. For example, in our case we cache all the configuration data in memory. Permanent home for this data is our RDS database. Because of the cache we get better performance and if our database goes down, our app can keep functioning. There are some inserts done at real time, but we have a JMX based switch that can switch it off. That way the app can keep functioning as much as it could without needing the database. Also note that many systems have pieces that can go down without causing any damage. Identify them and be aware of them. That way you can avoid panic when they go down.

2) Use elastic load balancer for critical services.

Elastic load balancer + auto scaling ensures that the service behind a load balancer is up all the time.

3) Use EBS volumes wherever data on the disk is important.

Use EBS backed images wherever possible. It’s very easy to take backups of EBS volumes (snapshot) and it’s very easy to boot up a new instance with a backup. Furthermore in many cases we have found that if an instance is unreachable, simply rebooting the instance is sufficient to bring it up again. You can’t reboot a s3 backed instance without loosing disk data.

4) Duplicate your data (unless it’s in S3) on multiple machines.

Use distributed file systems. We use HBase which uses Hadoop internally to make sure that the data is distributed and duplicated.

5) Create Snapshots often

Do not hesitate to take snapshots every hour if you need to. Snapshots are incremental backups. Every time you take a snapshot, only change are backed up to save space (and cost).

6) Use Amazon’s services wherever possible instead of using software installed on EC2 instances.

For example use SQS and SNS for your asynchronous and publish-subscribe communication wherever possible instead of installing and taking care of queue systems yourself.

7) Follow best practices

Follow EC2 best practices mentioned in the following document - Architecting for Cloud: Best Practices. It’s a very useful guide.

But there is a web service version of this metadata tool too. This version is not so much advertised. Whenever I google ‘aws metadata query tool’, I never find it. I like the webservice version better becasue I don’t have to go through installation steps everytime I want get metadata of an instance.

Here is how you can use the webservice version:

curl http://169.254.169.254/latest/meta-data/

This will print all the metadata fields available for querying. It should print something like:

ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4
public-keys/
reservation-id
security-groups


Now, to get instance-id,you simply have to execute the following command:

curl http://169.254.169.254/latest/meta-data/instance-id

It will simply print the instance id of the machine you are executing this command on.

Please note that some metadata fields listed above have a / in front of them. The webservice url for those fields must have forward slash at the end. For example

curl http://169.254.169.254/latest/meta-data/public-keys

will not print anything. You must execute the command as:

curl http://169.254.169.254/latest/meta-data/public-keys/

To begin with let’s look at an autoscaling trigger creation command:

as-create-or-update-trigger my-cpu --auto-scaling-group myautoscalinggroup --dimensions "AutoScalingGroupName=myautoscalinggroup" --measure CPUUtilization --period 60 --statistic Average --lower-threshold 20 --upper-threshold 50 --breach-duration 300 --upper-breach-increment 1 --lower-breach-increment 1

The dimensions parameter indicates that this is a trigger created on myautoscalinggroup autoscaling group. The measure indicates that it’s a trigger based on CPU Utilization. Period indicates that the trigger will be evaluated every 60 seconds. if upper threshold (average cpu utilization goes above 50%) is breached for 300 seconds then start 1 additional instance. If lower threshold (average cpu utilization goes below 20%) is breached for 300 seconds then 1 instance is scaled down.

The metrics on which a autoscaling trigger is built (Cloudwatch measure) is the most important decision you make when you set up an autoscaling cluster.

There are many metrics you can choose to scale up or down your cluster. CPU utilization, latency, network out bytes, disk reads in bytes etc. You can set up autoscaling based on whatever data Cloudwatch offers.

Latency or CPU utilization were obvious choices for our web cluster. We started out with latency as the metrics for autoscaling. Quickly we realized that latency was not the right metrics for us. Our application was making calls to some third party web services. Sometimes third party webservices problems caused our latency to go up. In that case, adding more instances to the cluster didn’t help much. Furthermore, third party calls made our latency spike unevenly. We wanted our cluster to go up smoothly as the traffic increases and come down smoothly as the traffic decreased.

We decided to switch to CPU Utilization as the metric for autoscaling. We started thinking about our upper and lower thresholds. We chose 50 as the upper threshold. It can take autoscaling few minutes to kick in. It was important for us to keep enough room (50%) for CPU to go up in those few minutes. In fact by setting a breach duration of 300 seconds we ensured that autoscaling did not kick in immediately. This adjustment accommodated occasional traffic spikes that went down on their own. By looking at our average CPU utilization, we decided to keep the lower threshold at 20% CPU utilization. This worked well when we had a relatively small cluster. Our traffic pattern caused dramatic increase in our average CPU utilization in a smaller cluster (3 to 5 instances). CPU went up to 50% quickly and came back down to 20% when traffic subsided.

Recently our traffic increased dramatically and as a result our cluster increased tenfold! Now we were running 40 instances! Our CTO observed that the autoscaling was adding instances when they are necessary, but the cluster was not scaling down quickly enough. In a bigger cluster, average cpu was fairly stable and was not undergoing dramatic changes as in a smaller cluster. So our CTO decided to adjust the lower threshold to 40% of CPU Utilization. This worked and the autoscaling cluster started scaling down.

You can see the trigger scaling instances up and down throughout a day below. Time is on x axis and number of instances is on y axis.

We hope our experiences are helpful to others implementing autoscaling. I am sure we will have to revisit these numbers when our cluster grows to 100s of instances! I will write a new blog post describing my experiences then.

1) Use private dns addresses in config files such as hdfs-site.xml, hbase-site.xml. On ec2 Ubuntu instances java’s getHost() gets resolved to the private dns addresses.

2) Use c1.xlarge or bigger node to start with. I have seen Andrew Purtell (HBase committer) recommending this on HBase mailing list. We have tried m1.large machines. It has worked well for us when our traffic was small. We are hitting HBase in real time. As traffic started increasing we started getting CPU maxouts. Currently we use c1.xlarge machines.

3) Define a security group for all of your Hadoop/HBase nodes. Hadoop, HBase, Zookeeper nodes need to talk with each other on many ports. It’s better to assign one security group to all the nodes and give that group permission to talk to it self.

4) Use dfs.host property in hdfs-site.xml. This property allows you to specify a path of a file with a list of dns addresses of allowable nodes. Only nodes listed in the file will be allowed to join the Hadoop cluster. This becomes important in case you are spanning a QA cluster in ec2. You don’t want your QA node to join your production cluster accidentally. If you add a new node, you need execute the following command to refresh the allowable nodes list to avoid entire cluster restart:

hadoop dfsadmin -refreshNodes

I am assuming here that you are not running map reduce on the same cluster. If you want to run map reduce on the same cluster do not use a similar setting in mapred-site.xml. There is no refreshNodes option available for mradmin. It’s coming up in 0.21.

5) Use EBS backed instances for a Hadoop/HBase nodes. There are many benefits of using EBS volumes instead of the s3 backed ones. For example, let’s say you tried a different vm args on a node to test the performance. And if the very same node goes down, you don’t have to worry about loosing those settings. All you need to do is take a snapshot of that volume and spawn an instance. I generally find it easier to deal with EBS backed instances than S3 backed instances. The added cost is too small in comparison with the added convenience of not loosing data.

6) Prepare an AMI with Hadoop, HBase installation and with all the configurations. This will allow you to add a node quickly when you want to do so.

7) Use Elastic Map Reduce to run map reduce jobs with your HBase. We have found this much more cost effective. Let’s say you have a 7 node HBase cluster. But your map reduce jobs need at least 10 nodes for it to finish on time. Elastic map reduce will allow you to spawn a 10 node job with your HBase. The load on your HBase cluster will reduce and you don’t have to pay for 3 extra nodes 24 x 7. Furthermore, you can spawn cheaper instances for your EMR job than you are using for your HBase (c1.xlarge) if you want to. We use c1.mediums.

HBase guys have written scripts to spawn a cluster in ec2. The scripts makes it easier to spawn an entire cluster. Don’t hesitate to try it out. Let me know if you have any other tips that you would like to share. I am sure many people are running their HBase clusters on ec2. I am eager to learn from their experiences.

Let’s discuss an example. The example is in groovy, but I am sure the code can be understood by everybody:


def registerInstance(String instanceId, String loadBalancerName) {
  def request request = new RegisterInstancesWithLoadBalancerRequest()
  def server = new com.amazonaws.services.elasticloadbalancing.model.Instance()
  server.setInstanceId(instanceId)
  request = request.withInstances([server]).withLoadBalancerName(loadBalancerName)
  def client = new AmazonElasticLoadBalancingClient(awsCredentials)
  client.registerInstancesWithLoadBalancer(request)
  logger.info "${instance.getPublicDnsName()} registered with ${loadBalancerName} load balancer"
}


Above code simply registers a given instance with a load balancer. Now let’s try to achieve the same thing using a different API available on Google code. It’s called Typica and is written by dkavanagh and two other persons.


def registerInstance(Stirng instanceId, String loadBalancerName) {
  def loadBalancing = new LoadBalancing(accessKey, secretKey)
  loadBalancing.registerInstancesWithLoadBalancer(loadBalancerName, [instaceId])
}

Now which one is easier to read? Which one is faster to code? Which is more intuitive? Obviously the second one. I simply don’t understand the reason for long class names they have throughout the API and the request and result pattern. Every single method in Amazon’s sdk take a request object and return a result object. You are forced to create these extra long name objects! Thank god I did not write the code in java here, otherwise it wouldn’t have been even bigger as in Java you have to repeat a class name twice in a line if you want to create an object. The whole api is full of such examples.

I am not the only one screaming over the API. Steve Jin has expressed similar concerns about the API in his DoubleCloud Blog. According to Steve the API lacks consistency, clear object model and the structure of the API is flawed.

Hope enough people scream over the Internet so that Amazon can hear it.

SDK or a java api is very much needed – especially if you are writing your automation scripts in groovy. We have tried multiple java apis in our scripts including JetS3t and Typica. These apis were really helpful, but they only supported some of the AWS services and were not up to date (for obvious reasons). Having one java api that can support all of AWS technologies was definitely the need of the hour. I am sure Amazon will keep it updated as new services are released. They have the necessary resources to do so.

Furthermore, Amazon has also uploaded the SDK to the maven repository.
You can use the following dependency in your pom.xml:

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk</artifactId>
    <version>1.0.002</version>
</dependency>

The java doc for the SDK is hosted at http://docs.amazonwebservices.com/AWSJavaSDK/latest/javadoc/index.html

Amazon has also opened the SDK source code for all. They have mirrored the SDK code repository at github. You can look at the SDK code at http://github.com/amazonwebservices/aws-sdk-for-java

I played with it and found that the service is really easy to use, very robust and very extensible. It basically makes a publish/subscribe messaging service similar to JMS available in the cloud. Having a cheap, robust publish/subscribe messaging system can serve many purposes in the cloud. In an auto scaled environment servers go up and down depending upon traffic. Here are some of the usages I can think of in our environment:

Discover members of a cluster
We use JMX to switch on/off various services in our java webapp. JManage allows us to configure a cluster and call a method remotely on the entire cluster. That is how we flush caches, switch on/off services etc. JManage is not sufficient by itself in an autoscaled environment as it does not detect new nodes automatically. When a new node comes up, somebody needs to configure the node in the JManage cluster. You can create a topic and publish a message to it with the external dns name of the newly born server in it. This message can be then consumed by a piece of code that can add the new node in the JManage Cluster. It’s also possible to similarly remove a node by publishing a message when the instance is terminated by auto scaling.

Discovering members of a cluster is very useful in a cloud environment especially in an auto scaled environment. As Http is supported as a protocol, it’s possible call a servlet (or similar concept in other platforms) url. This opens up a lot of possibilities. You can pretty much execute anything you want.

Send email alerts without having to configure a SMTP server
Simply put, you can use SNS to send simple emails. All the email addresses need to be the subscribers of the topic, but that’s a trivial one time activity. No need to configure and maintain any SMTP server!

Some nice to haves features
There is one particular thing I find annoying about SNS. Let’s say you subscribed to a topic using the email protocol. You will get a long string identifier called subscription arn. When you want to unsubscribe this email, you cannot supply the email address and the topic, you must supply the subscription arn. That’s not a good idea. It forces applications to store the subscription arn. To me an email and a topic combination is unique enough for them to determine the subscription. It will be much easier for applications if they change the unsubscribe api to accept the topic and the email address you want to unsubscribe. I can understand that they probably want their api to be protocol agnostic, but it will be a nice to have feature.

Furthermore, you cannot set the subject of the notification email. That means, if I am using multiple notifications in application, all the notifications will have the same standard subject line – AWS Notification Message.

It would also be nice to have twitter as a protocol. We use twitter heavily for notification. Twitter allows us to get notifications via text messages. Pingdom got us into it.

We have a web application with nginx + tomcat. We approximately get 200k requests per minute at the peak and about 65K requests per minute at night. Since we host a webservice and not a webpage, most of our requests are servlet requests (and not the faster file serving, nginx based requests).

We began our experiment with two m1.small machines. We set the autoscaling group minimum to two and set the tomcat file handle limit to be 65535.
We based on autoscaling metric on latency. We asked our autoscaling group to increase the capacity if the latency goes beyond 0.75 seconds.

This didn’t work that well. Here is what happened:

1) We started getting cpu bursts in peak hours. CPU used to jump to 100%, there by causing latency to go beyond 0.75 seconds periodically. The autoscaling used to kick in, starting a new server. The latency use to fall back below 0.75 within few minutes and the autoscaling used to cut back the capacity to the minimum.

2) We also started getting nginx errors. Here is what the error said:

(24: Too many open files) while accepting new connection on 0.0.0.0:80


Afer some research, we realized that the default nginx settings were not meant for the kind of scale we were dealing with. We changed the following settings in the /etc/nginx/nginx.conf:

worker_processes 4;
worker_rlimit_nofile 10240;
events {
worker_connections 8192;
}

The nginx errors stopped. And the CPU bursts evened out for the non peak period. Here is how the graph looked right after we made the nginx change.

You can clearly see that the change the bursts stopped at one point (when we made the change). But the CPU bursts started coming back in the peak time. This time, nginx was fine and there were no errors in the log.

This was clearly a signal that m1.small was not performing well at that load (for our application). We decided to switch to c1.mediums. We knew that c1.mediums have 5 EC2 compute units where as the m1.smalls have 1 EC2 compute unit. But we wanted to see how far m1.smalls can take us. The switch totally worked! Cpu bursts stopped. Autoscaling stopped kicking in. We can see the cpu going from 10% to 50% smoothly from non peak to peak hours. This is what we wanted!

Obviously two c1.mediums cost more than two m1.smalls. But we belive that we will be able to cope up with much larger growth using c1.mediums as the CPU is always hovering between 10 to 40%. In long term, it will definitely save us money. We will need less number of machines and we won’t waste money on instances getting started for a few minutes and getting shutdown when the bursts subside.