1) Use private dns addresses in config files such as hdfs-site.xml, hbase-site.xml. On ec2 Ubuntu instances java’s getHost() gets resolved to the private dns addresses.

2) Use c1.xlarge or bigger node to start with. I have seen Andrew Purtell (HBase committer) recommending this on HBase mailing list. We have tried m1.large machines. It has worked well for us when our traffic was small. We are hitting HBase in real time. As traffic started increasing we started getting CPU maxouts. Currently we use c1.xlarge machines.

3) Define a security group for all of your Hadoop/HBase nodes. Hadoop, HBase, Zookeeper nodes need to talk with each other on many ports. It’s better to assign one security group to all the nodes and give that group permission to talk to it self.

4) Use dfs.host property in hdfs-site.xml. This property allows you to specify a path of a file with a list of dns addresses of allowable nodes. Only nodes listed in the file will be allowed to join the Hadoop cluster. This becomes important in case you are spanning a QA cluster in ec2. You don’t want your QA node to join your production cluster accidentally. If you add a new node, you need execute the following command to refresh the allowable nodes list to avoid entire cluster restart:

hadoop dfsadmin -refreshNodes

I am assuming here that you are not running map reduce on the same cluster. If you want to run map reduce on the same cluster do not use a similar setting in mapred-site.xml. There is no refreshNodes option available for mradmin. It’s coming up in 0.21.

5) Use EBS backed instances for a Hadoop/HBase nodes. There are many benefits of using EBS volumes instead of the s3 backed ones. For example, let’s say you tried a different vm args on a node to test the performance. And if the very same node goes down, you don’t have to worry about loosing those settings. All you need to do is take a snapshot of that volume and spawn an instance. I generally find it easier to deal with EBS backed instances than S3 backed instances. The added cost is too small in comparison with the added convenience of not loosing data.

6) Prepare an AMI with Hadoop, HBase installation and with all the configurations. This will allow you to add a node quickly when you want to do so.

7) Use Elastic Map Reduce to run map reduce jobs with your HBase. We have found this much more cost effective. Let’s say you have a 7 node HBase cluster. But your map reduce jobs need at least 10 nodes for it to finish on time. Elastic map reduce will allow you to spawn a 10 node job with your HBase. The load on your HBase cluster will reduce and you don’t have to pay for 3 extra nodes 24 x 7. Furthermore, you can spawn cheaper instances for your EMR job than you are using for your HBase (c1.xlarge) if you want to. We use c1.mediums.

HBase guys have written scripts to spawn a cluster in ec2. The scripts makes it easier to spawn an entire cluster. Don’t hesitate to try it out. Let me know if you have any other tips that you would like to share. I am sure many people are running their HBase clusters on ec2. I am eager to learn from their experiences.

Let’s discuss an example. The example is in groovy, but I am sure the code can be understood by everybody:


def registerInstance(String instanceId, String loadBalancerName) {
  def request request = new RegisterInstancesWithLoadBalancerRequest()
  def server = new com.amazonaws.services.elasticloadbalancing.model.Instance()
  server.setInstanceId(instanceId)
  request = request.withInstances([server]).withLoadBalancerName(loadBalancerName)
  def client = new AmazonElasticLoadBalancingClient(awsCredentials)
  client.registerInstancesWithLoadBalancer(request)
  logger.info "${instance.getPublicDnsName()} registered with ${loadBalancerName} load balancer"
}


Above code simply registers a given instance with a load balancer. Now let’s try to achieve the same thing using a different API available on Google code. It’s called Typica and is written by dkavanagh and two other persons.


def registerInstance(Stirng instanceId, String loadBalancerName) {
  def loadBalancing = new LoadBalancing(accessKey, secretKey)
  loadBalancing.registerInstancesWithLoadBalancer(loadBalancerName, [instaceId])
}

Now which one is easier to read? Which one is faster to code? Which is more intuitive? Obviously the second one. I simply don’t understand the reason for long class names they have throughout the API and the request and result pattern. Every single method in Amazon’s sdk take a request object and return a result object. You are forced to create these extra long name objects! Thank god I did not write the code in java here, otherwise it wouldn’t have been even bigger as in Java you have to repeat a class name twice in a line if you want to create an object. The whole api is full of such examples.

I am not the only one screaming over the API. Steve Jin has expressed similar concerns about the API in his DoubleCloud Blog. According to Steve the API lacks consistency, clear object model and the structure of the API is flawed.

Hope enough people scream over the Internet so that Amazon can hear it.

SDK or a java api is very much needed – especially if you are writing your automation scripts in groovy. We have tried multiple java apis in our scripts including JetS3t and Typica. These apis were really helpful, but they only supported some of the AWS services and were not up to date (for obvious reasons). Having one java api that can support all of AWS technologies was definitely the need of the hour. I am sure Amazon will keep it updated as new services are released. They have the necessary resources to do so.

Furthermore, Amazon has also uploaded the SDK to the maven repository.
You can use the following dependency in your pom.xml:

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk</artifactId>
    <version>1.0.002</version>
</dependency>

The java doc for the SDK is hosted at http://docs.amazonwebservices.com/AWSJavaSDK/latest/javadoc/index.html

Amazon has also opened the SDK source code for all. They have mirrored the SDK code repository at github. You can look at the SDK code at http://github.com/amazonwebservices/aws-sdk-for-java

I played with it and found that the service is really easy to use, very robust and very extensible. It basically makes a publish/subscribe messaging service similar to JMS available in the cloud. Having a cheap, robust publish/subscribe messaging system can serve many purposes in the cloud. In an auto scaled environment servers go up and down depending upon traffic. Here are some of the usages I can think of in our environment:

Discover members of a cluster
We use JMX to switch on/off various services in our java webapp. JManage allows us to configure a cluster and call a method remotely on the entire cluster. That is how we flush caches, switch on/off services etc. JManage is not sufficient by itself in an autoscaled environment as it does not detect new nodes automatically. When a new node comes up, somebody needs to configure the node in the JManage cluster. You can create a topic and publish a message to it with the external dns name of the newly born server in it. This message can be then consumed by a piece of code that can add the new node in the JManage Cluster. It’s also possible to similarly remove a node by publishing a message when the instance is terminated by auto scaling.

Discovering members of a cluster is very useful in a cloud environment especially in an auto scaled environment. As Http is supported as a protocol, it’s possible call a servlet (or similar concept in other platforms) url. This opens up a lot of possibilities. You can pretty much execute anything you want.

Send email alerts without having to configure a SMTP server
Simply put, you can use SNS to send simple emails. All the email addresses need to be the subscribers of the topic, but that’s a trivial one time activity. No need to configure and maintain any SMTP server!

Some nice to haves features
There is one particular thing I find annoying about SNS. Let’s say you subscribed to a topic using the email protocol. You will get a long string identifier called subscription arn. When you want to unsubscribe this email, you cannot supply the email address and the topic, you must supply the subscription arn. That’s not a good idea. It forces applications to store the subscription arn. To me an email and a topic combination is unique enough for them to determine the subscription. It will be much easier for applications if they change the unsubscribe api to accept the topic and the email address you want to unsubscribe. I can understand that they probably want their api to be protocol agnostic, but it will be a nice to have feature.

Furthermore, you cannot set the subject of the notification email. That means, if I am using multiple notifications in application, all the notifications will have the same standard subject line – AWS Notification Message.

It would also be nice to have twitter as a protocol. We use twitter heavily for notification. Twitter allows us to get notifications via text messages. Pingdom got us into it.

We have a web application with nginx + tomcat. We approximately get 200k requests per minute at the peak and about 65K requests per minute at night. Since we host a webservice and not a webpage, most of our requests are servlet requests (and not the faster file serving, nginx based requests).

We began our experiment with two m1.small machines. We set the autoscaling group minimum to two and set the tomcat file handle limit to be 65535.
We based on autoscaling metric on latency. We asked our autoscaling group to increase the capacity if the latency goes beyond 0.75 seconds.

This didn’t work that well. Here is what happened:

1) We started getting cpu bursts in peak hours. CPU used to jump to 100%, there by causing latency to go beyond 0.75 seconds periodically. The autoscaling used to kick in, starting a new server. The latency use to fall back below 0.75 within few minutes and the autoscaling used to cut back the capacity to the minimum.

2) We also started getting nginx errors. Here is what the error said:

(24: Too many open files) while accepting new connection on 0.0.0.0:80


Afer some research, we realized that the default nginx settings were not meant for the kind of scale we were dealing with. We changed the following settings in the /etc/nginx/nginx.conf:

worker_processes 4;
worker_rlimit_nofile 10240;
events {
worker_connections 8192;
}

The nginx errors stopped. And the CPU bursts evened out for the non peak period. Here is how the graph looked right after we made the nginx change.

You can clearly see that the change the bursts stopped at one point (when we made the change). But the CPU bursts started coming back in the peak time. This time, nginx was fine and there were no errors in the log.

This was clearly a signal that m1.small was not performing well at that load (for our application). We decided to switch to c1.mediums. We knew that c1.mediums have 5 EC2 compute units where as the m1.smalls have 1 EC2 compute unit. But we wanted to see how far m1.smalls can take us. The switch totally worked! Cpu bursts stopped. Autoscaling stopped kicking in. We can see the cpu going from 10% to 50% smoothly from non peak to peak hours. This is what we wanted!

Obviously two c1.mediums cost more than two m1.smalls. But we belive that we will be able to cope up with much larger growth using c1.mediums as the CPU is always hovering between 10 to 40%. In long term, it will definitely save us money. We will need less number of machines and we won’t waste money on instances getting started for a few minutes and getting shutdown when the bursts subside.

Keep machines in the same availability zone
Don’t scatter your machines that talk to each other across multiple availability zones. You will end up paying for the bandwidth. Off course this does not apply to the people who purposely keep their machines in different availibilty zones to ensure high availibility.

Use cnames instead of A records
In other words do not map your domain name to an elastic ip. Map it to the public domain name of the instance. Let’s say you have a machine with name splunk.gumgum.com. All the web servers (within the same availability zone) send considerable data to this machine. If you setup splunk.gumgum.com as an A record, your data will go out and come back in. But if you map it as cname, your data will always remain within the ec2 cluster. To read more about this visit Eric Hammond’s Using Elastic IP to Identify Internal Instances on Amazon EC2 post.

Use spot instances
Use spot instances whenever possible. Spot instances cost less than on demand instances. You can read more about spot instances on Amazon’s web page. There is a website that shows spot history price graphs. It will tell you an average price of various types of instances. The average price of a m1.small sport instance is 3 cents an hour. You pay 8.5 cents an hour for the same type of on demand instance.

Choose your instance types wisely
A smallest possible instance may not be the right choice to save money. For example c1.medium costs twice as much as m1.small. But it offers 5 times more compute power than m1.small. Thus in some compute heavy jobs it might be cheaper to use c1.medium instead of the least expensive m1.small.

Choose the smallest possible storage
Choose the smallest possible EBS volumes and RDS instances. It is very easy to expand an ebs volume when needed. Same is the case with a RDS instance. Furthermore by choosing smaller drives, you will save money needed for the backup(snapshot) storage too.

Use Autoscaling
Autoscaling can start additional instances when needed. It can shut down the additional instances when the need disappears. This is especailly useful for hosting websites/web services that have a specific traffic/usage pattern.

Reserve your instances
You can reduce your costs significantly by reserving number of instances for a year or for three years. Reservation is only a billing concept and thus you don’t have to designate any istances as reserved. If you reserve 4 m1.small instances, Amazon will make 4 of your running m1.small instances as reserved. For more information visit Werner Vogels’s (Amazon CTO) blog.

There might be numerous other ways you can save money. Please feel free to leave a comment to share your ways. The more the merrier!

RDS does not give you super privileges. That is why you won’t be able to change the global time zone by simply executing:

SET GLOBAL time_zone = 'US/Pacific';


The default_time_zone db parameter is not modifiable.

In this Amazon forum thread, you can clearly see the AWS guy saying that there is no way to change the time zone of a RDS instance.

It is very important to understand that unlike Java (or other programming languages), MySQL does not store time zone in its date, datetime or timestamp datatypes. That is why it’s important that all the dates are stored in one time zone. Most people prefer to store them in their own time zone for easier calculations.

Since you cannot change the database server settings, you will have to change your application. Here are some approaches you can take to make sure that the date and time stored in your database is in your desired time zone.

1) Set time zone per database connection
When ever you create a connection, set time zone using

SET time_zone = 'US/Pacific';

This time zone will be valid only for the connection. That is why it must be done for every single connection you open. If you have a java application, you can use a wrapper around the JDBC driver to accomplish this.

Please note that MySQL allows you to specify time zone by its name (such as ‘US/Pacific’) or by the difference in hours (such as ‘-8:00′). If you are in a part of United States or any other part of the world that uses day light savings, do not use the difference of hours option for obvious reasons.

Named time zones can only be used if the time zone related tables are populated in MySQL. Thankfully they seem to be populated in RDS.

2) Use convert_tz function
You can use MySQL’s convert_tz function to convert the time zone. For example:


insert into my_table (created_date) values (convert_tz(now(), 'GMT', 'US/Pacific'));


For more information on convert_tz function visit convert_tz documentation on MySQL website

3) Set the date/time pragmatically in your application code in every query.
Thus if you have created_date column in your table, do not rely on timestamp datatype’s default value. The default value will be in UTC time zone. Set the value explicitly using application code. If you are using a prepared statement this becomes very easy. All you have to make sure is to pass a date object with appropriate time zone set in it. In order for your application to calculate correct date, you have to make sure that your application instance’s time zone is set to your time zone.

Amazon ec2 instances allow you to change their default time zones. If your web server is also deployed in ec2, I would suggest you to bundle an AMI with your time zone and use it to launch all your instances. I think most of the ubuntu public AMIs have time zone set to UTC.

Here are the easy steps:

1) Install ubuntu desktop on your server instance
By default X server, GNOME etc is not installed on your ubuntu instance. You can install it by simply executing the following commands:

export DEBIAN_FRONTEND=noninteractive
sudo -E apt-get update
sudo -E apt-get install -y ubuntu-desktop

It will install lot of stuff. So don’t get surprised if it takes 10 to 15 minutes to finish it.

2) Install FreeNX server on your server
The following commands assume that you are using ubuntu 9.10 (Karmic Koala) image.

sudo add-apt-repository ppa:freenx-team
sudo apt-get update
sudo aptitude install -y freenx
sudo /usr/lib/nx/nxsetup --install


If you are using older ubuntu versions, please visit FreeNX server installation page on ubuntu website to get the instructions for installing FreeNX server.

While executing the last command, you will get the following prompt:

It is recommended that you use the NoMachine key for
easier setup. If you answer "y", FreeNX creates a custom
KeyPair and expects you to setup your clients manually.
"N" is default and uses the NoMachine key for installation.

I would say answer ‘N’ which is the default. If you answer Y, you will have to generate keys and put them at a specific location in the client.

3) Enable password authentication on the ssh server
Edit /etc/ssh/sshd_config and set PasswordAuthentication to yes. By default password authentication is disabled.
You will need to restart ssh process by executing

sudo /etc/init.d/ssh restart

Now that you have enabled password authentication, you should make sure that the the user you are going to use for using remote desktop has a password. Cannonical’s karmic images come with ‘ubuntu’ user. This user does not have a password. You can set a password by executing

sudo passwd ubuntu

4) Install and setup NX client
Now install the NoMachineNX client on your Desktop. You can download the client form http://www.nomachine.com/download.php. You can install it by simply double clicking the downloaded file. Once you install it, click on the NX Connection wizard to setup a new connection. Fill in the values as shown below. Make sure that you put in your server address in the host field.

And that’s it! NX client will create a desktop shortcut for the server connection (Only if you choose to do so). Double click the shortcut, fill in the username (ubuntu) and the password (set in step 3) and you are in!

RDS Pros:
1) No need to maintain MySQL in the cloud. Amazon will maintain it. Amazon will apply all the patches and upgrade the server software.

2) It’s very easy to scale up. All you have to do is execute rds-modify-db-instance command. You can change your db class to the bigger/better machine.

3) Snapshot based backups are very easy to manage.

4) No need to manage an ec2 volume, backup and an ec2 instance separately to run your database in the cloud. RDS combines all the three in one package.

RDS Cons:
1) The biggest problem about RDS is the 4 hour per week maintenance window. I don’t believe that this maintenance takes place every week. They only take down your database when they are applying patches or upgrading the database server. They will take down your database in your specified time only. This time can be changed any time by simply executing an api call. I also believe that in most cases the downtime will be much less than 4 hours. Nevertheless the threat of downtime exists every week!

2) Does not expose my.cnf file of the database. It instead gives you an API to modify the parameters

3) If you are an advanced MySQL user and want to use tweaked MySQL (such as by Percona http://www.percona.com/), RDS is clearly not a solution.

4) As of now, there is no replication support. This means you cannot have any read only slaves. I have read somewhere that they are working on it.

Clearly RDS is not a silver bullet. In fact in many cases where your application cannot run without a database, you may not be able to afford the weekly downtime. I talked to some friends and the 4 hour weekly downtime was their the biggest problem with RDS. I wish that Amazon would declare the downtime whenever the need arises (such as when new MySQL patches are ready) and less frequently (such as once a month).

We did not have a full time system administrator in our team. We were bunch of software engineers trying to manage our ec2 instances by ourselves. Furthermore our application wasn’t dependant on database completely. Most of the data was cached. Thus we could afford an occasional downtime. That is why we chose RDS.

First detach the volume and take a snapshot of the volume. If you already have a snapshot and haven’t made any changes after that, no need to take a new snapshot.

No create a new volume from this snapshot. Let’s say your original volume was 10G and you wanted to increase its size to 20G.

ec2-create-volume -z us-east-1a --size 20 --snapshot snap-xxxxxxxx

The command will output id of the new volume. Use that id in the following command to attach the volume to any instance you have up.

ec2-attach-volume vol-yyyyyyyy --instance i-xxxxxxxx --device /dev/sdh

Make sure that ec2-describe-volumes show the state of the instance as attached. Once the volume is attached, mount it at a path say /mnt/data. I am assuming here that your volume had ext3 filesystem.

mkdir /mnt/data
echo '/dev/sdh /mnt/data ext3 defaults,noatime 0 0' >> /etc/fstab
mount /mnt/data

Mounting the volume is necessary as otherwise you wont’ be able to execute the following (resize2fs) command.
Now execute the following command to resize the volume

resize2fs /dev/sdh

You should see the following output (sizes you see might be different):

Filesystem at /dev/sdh is mounted on /mnt/data; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 2
Performing an on-line resize of /dev/sdh to 5505024 (4k) blocks.
The filesystem on /dev/sdh is now 5505024 blocks long.

The command should execute in a minute or two for 20G size. That’s it, you can verify your new size by executing:

df -h

This should show you that the volume’s new size. That’s it!