Posts

Rackspace vs Amazon impressions

I’ve been a customer of Amazon Web Services since they have been in beta. I’ve also worked with Rackspace now and then. Lately I’ve been working extensively with Rackspace infrastructure so I have a better understanding of their products. Bellow there are a few things that bother me at Rackspace:

Instances

The thing you are going to use 99% of the time with any of the providers. Amazon offers normal instances, high cpu instances, high memory instances, huge memory and cpu instances. On Rackspace you will have a very limited offer of instances. No high cpu or high memory. They do have plans to add them in nearby future. Until then, if you have a CPU bottleneck then it’s tough luck: scale up the instance and pay double.

And since we are talking about pricing for instances, Amazon is cheaper on the long run due to Reserved Instances (you pay an upfront fee and lower price/hour for the instance after). Amazon has been cutting down prices several times. Rackspace? Never.

Block storage

Called EBS on Amazon or CBS on Rackspace, it’s the preferred way of adding extra space to running instances without upgrading them. Beware that on Rackspace it takes an horrible amount of time to do a snapshot. I had to contact RS support several times because of this.

MySQL instances

Called RDS on Amazon or Cloud Database on Rackspace, they are basically optimized instances that run MySQL, but you are not given SSH access to them. Here Rackspace has the worst offer:
no scheduled backups (RS recommends mysqldump … lol try to do mysqldump for 10GB+ of data)
no replication (really? I couldn’t replicate my database)
no hotspare (Amazon MultiA-Z)

The good thing about Cloud Database is that they are fast. Really fast. Also RS promised to address all the above things in the nearby future (yeah).

Load Balancing

Rackspace does not support LB inside private network. Come on RackSpace, do your homework. This thing has been working for AWS since ancient history.

Conclusion:

There are many other things that need to be discovered, probably not pleasant ones. So far I am disappointed with Rackspace and what it has to offer and I would recommend to any customer to use Amazon instead.

Installing Scalr 3.5 Open Source on Ubuntu 12.04

This is an update on an older post of mine, one of my first articles regarding cloud computing. Much has changed since since 2008 when I have wrote this article “How to install Scalr on Ubuntu 8.10 EC2 Instance“.

For example the Ubuntu has evolved to 12.04 LTS (I am using LTS 64bit for this howto) and Scalr is now version 3.5. One thing didn’t change: it’s still a royal PITA to get Scalr open source working. Hopefully this howto will help you to install Scalr on your server. It doesn’t cover operating Scalr and other things, which I will address in future posts, if there is enough interest.

After you have installed Ubuntu 12.04 64bit server edition to your server or virtual machine the way you like it it’s time to start the update process:

apt-get update && apt-get upgrade

Now you are ready to run tasksel and select the following roles for your server: OpenSSH, DNS server, LAMP server

You will need to install some dev packages before going anything else:

apt-get install libcurl4-gnutls-dev make librrd-dev

Now it’s time for PHP5 related extensions:

apt-get install php5-curl php-gettext php-net-socket php5-mcrypt php-xml-serializer libssh2-php php-soap php5-snmp php5-rrd
pecl install pecl_http
echo "extension=http.so" >/etc/php5/conf.d/pecl_http.ini
pecl install rrd
echo "extension=rrd.so" >/etc/php5/conf.d/rrd.ini

Time to get Scalr code:

cd /tmp
wget http://bit.ly/scalr35
tar zxvf scalr35
cd scalr-3.5.r7704
cp -r app /var/www/
chown -R www-data:www-data /var/www/app

Create new database and import sql from sql/scalr:

mysql -p
mysql> CREATE DATABASE scalr CHARACTER SET latin1 COLLATE latin1_swedish_ci;
mysql> grant all privileges on scalr.* to scalr@localhost identified by 'password';
mysql> flush privileges;
mysql> quit
mysql -p scalr <sql/scalr.sql

While doing that import I’ve got a nice error:
ERROR 1054 (42S22) at line 2222: Unknown column ‘architecture’ in ‘field list’
Fix:
1) Drop database
2) Search sql/scalr.sql for “CREATE TABLE IF NOT EXISTS `role_images`” and add after platform:

`architecture` varchar(25) DEFAULT NULL,
`os_family` varchar(25) DEFAULT NULL,
`os_name` varchar(25) DEFAULT NULL,
`os_version` varchar(25) DEFAULT NULL,
`agent_version` varchar(25) DEFAULT NULL,

Reference:https://groups.google.com/forum/?fromgroups=#!topic/scalr-discuss/ApCGfrNXqz8

Configuration of Scalr is quite simple:

cd /var/www/app/etc
cp config.ini-sample config.ini
edit config.ini

Cron jobs required by Scalr? Just type crontab -e and add the following lines:

*/2 * * * * /usr/bin/php -q /var/www/app/cron-ng/cron.php --Poller
* * * * * /usr/bin/php -q /var/www/app/cron/cron.php --Scheduler2
*/10 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --MySQLMaintenance
* * * * * /usr/bin/php -q /var/www/app/cron/cron.php --DNSManagerPoll
17 5 * * * /usr/bin/php -q /var/www/app/cron/cron.php --RotateLogs
*/2 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --EBSManager
*/20 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --RolesQueue
*/5 * * * * /usr/bin/php -q /var/www/app/cron-ng/cron.php --DbMsrMaintenance
*/2 * * * * /usr/bin/php -q /var/www/app/cron-ng/cron.php --Scaling
*/5 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --DBQueueEvent
*/2 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --SzrMessaging
*/4 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --RDSMaintenance
*/2 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --BundleTasksManager
* * * * * /usr/bin/php -q /var/www/app/cron-ng/cron.php --ScalarizrMessaging
* * * * * /usr/bin/php -q /var/www/app/cron-ng/cron.php --MessagingQueue
*/2 * * * * /usr/bin/php -q /var/www/app/cron-ng/cron.php --DeployManager
*/2 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --UsageStatsPoller
* * * * * root /usr/bin/php -q /var/www/app/cron-ng/cron.php --SNMPStatsPoller

Time to add a Virtual Host:

cat <<EOF> /etc/apache2/sites-available/scalr
<VirtualHost *:80>
ServerName scalr.example.com
ServerAdmin webmaster@example.com
DocumentRoot "/var/www/app/www"

<Directory "/var/www/app/www">
Options Indexes FollowSymLinks
AllowOverride All
Order allow,deny
Allow from all
</Directory>
</VirtualHost>
EOF

Enable required Apache modules and site and restart everything:

a2ensite scalr
a2enmod rewrite
service apache2 restart

DNS managed by bind9:

chmod g+w /etc/bind/named.conf
echo 'include "/var/named/etc/namedb/client_zones/zones.include";' >> /etc/bind/named.conf
mkdir -p /var/named/etc/namedb/client_zones
chown root.bind /var/named/etc/namedb/client_zones
chmod 2775 /var/named/etc/namedb/client_zones
echo ' ' > /var/named/etc/namedb/client_zones/zones.include
chown root.bind /var/named/etc/namedb/client_zones/zones.include
chmod g+w /var/named/etc/namedb/client_zones/zones.include

To get rid of nasty AppArmor warnings and errors edit /etc/apparmor.d/usr.sbin.named and add:

/var/named/etc/namedb/client_zones/zones.include rw,

And finish it by restarting AppArmor and bind9:

service apparmor restart
service bind9 restart

Open your browser and go to http://scalr.example.com. Default username/password: admin/admin.

If you have issues or you need more info please feel free to comment 🙂

Apache2 worker vs prefork for ISPConfig benchmark

I’ve been running ISPConfig latest version(3.0.4) on Amazon cloud t1.micro instance for some time to host several small sites, mostly WordPress. I’m quite happy with the performance of the instance. The OS is Ubuntu 10.04 LTS. Until recently I’ve used the default mpm which is prefork, but I decided to test out worker also. If you are wondering I use mod_fcgid for all the sites. That being said I performed several tests with ab (apache benchmark) to see which mpm can server most requests per second.

While I do not claim this is the best setup, I think worker is better suited for me. Some people said they had problems because of mpm worker. So far so good, but will update this post if there are any issues.

Test results:

preforkworker
Concurrency Level: 32
Time taken for tests: 7.834 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 4972
Total transferred: 84831033 bytes
HTML transferred: 83206915 bytes
Requests per second: 638.27 [#/sec] (mean)
Time per request: 50.136 [ms] (mean)
Time per request: 1.567 [ms] (mean, across all concurrent requests)
Transfer rate: 10575.21 [Kbytes/sec] received
Concurrency Level: 32
Time taken for tests: 7.096 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 4968
Total transferred: 84877824 bytes
HTML transferred: 83247322 bytes
Requests per second: 704.63 [#/sec] (mean)
Time per request: 45.414 [ms] (mean)
Time per request: 1.419 [ms] (mean, across all concurrent requests)
Transfer rate: 11681.17 [Kbytes/sec] received

Creating consistent backups for EBS with EXT4 and quota

What’s this about?
Data security and backups are very important aspects when you work with servers, especially if you are using cloud infrastructure. I am using AWS(Amazon Web Services) as my preferred IaaS, so the following how-to is tailored for Amazon EC2 instances using EBS as storage for the web sites files. On my instance I have Ubuntu 10.04 LTS installed and on top of it I run ISPConfig 3.0.4(latest version at the moment I write this article). Some of the programs required to run this setup were already installed, but it should be pretty obvious if you miss anything. If you need help you can either leave a comment or contact me via email.

The following setup will allow you to create an EBS using EXT4 as file system, with quota enabled on it(for ISPConfig) and weekly backups of the EBS. In case of instance failure you should be able to launch a new instance and attach the EBS, without losing any web sites files. In case of EBS failure you can recreate one from the most recent snapshot.

Create an EBS in the same zone as your instance and attach it to your instance as /dev/sdf. This can be easily done from AWS Management Console.

Install xfsprogs

sudo apt-get install xfsprogs

Create EXT4 filesystem on /dev/sdf

sudo mkfs.ext4 /dev/sdf

Now mount it temporarily

sudo mkdir /mnt/ebs
sudo mount /dev/sdf /mnt/ebs

Stop the apache2 web server and copy the files to /mnt/ebs

sudo service apache2 stop
cd /mnt/ebs
sudo cp -rp /var/www/* .

Prepare quota

touch quota.user quota.group
sudo chmod 600 quota.*

Add the entry to /etc/fstab

/dev/sdf /var/www ext4 noatime,nobootwait,usrjquota=quota.user,grpjquota=quota.group,jqfmt=vfsv0 0 0

Unmount the EBS and remount it to /var/www

sudo umount /dev/sdf
sudo mount /dev/sdf /var/www -o noatime,usrjquota=quota.user,grpjquota=quota.group,jqfmt=vfsv0

Enable quota

sudo quotacheck -avugm
sudo quotaon -avug

Start the apache2 web server and check that the web sites are working properly

sudo service apache2 start

Install ec2-consistent-snapshot script for weekly backups of EBS

sudo add-apt-repository ppa:alestic
sudo apt-get update
sudo apt-get install -y ec2-consistent-snapshot

Prepare first snapshot(I assume the cron will run as root user, hence I create the awssecret file in /root directory)

sudo touch /root/.awssecret
sudo chmod 600 /root/.awssecret

Edit .awssecret and add following lines, in this order, replacing ACCESS_KEY_ID and SECRET_ACCESS_KEY with your own, both can be found under Account->Security Credentials:

ACCESS_KEY_ID
SECRET_ACCESS_KEY

Test the snapshot creation with debug mode activated, replace VOLUME_ID with the right volume ID:

sudo ec2-consistent-snapshot --debug --description "snapshot $(date +%Y-%m-%d-%H:%M:%S)" --freeze-filesystem /var/www vol-VOLUME_ID

If everything went well you should be able to see your new snapshot in the AWS Management Console.

Finally add this to your root crontab (by running sudo crontab -e):

@weekly /usr/bin/ec2-consistent-snapshot --debug --description "snapshot $(date +'%Y-%m-%d %H:%M:%S')" --freeze-filesystem /var/www vol-VOLUME_ID>>/var/log/backup.log 2>&1

Make sure you put the correct VOLUME_ID!

This should be all, you now have all your web sites on EBS, quota is enabled and weekly backups enabled. I think I pretty much nailed everything you need in order to perform this setup, but if there are any issues feel free to leave a comment. Also I love getting feedback so if you found this article useful leave a comment also 🙂

Amazon RDS SUPER privileges

#1419 – You do not have the SUPER privilege and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable

This error occurs sometimes on RDS instances when you try to use procedures. You will soon find out that grant super privilege for a user won’t work. So the only way to make things work is to set log_bin_trust_function_creators to 1.

RDS console available at https://console.aws.amazon.com/rds/ allows you to create a new group and modify its parameters. Log in to RDS console, go to “DB Parameters Groups” and click the “Create DB Parameter Group”. Set the following

  • DB Parameter Group Family: mysql5.1
  • DB Parameter Group Name: mygroup
  • Description: mygroup

Confirm by clicking “Yes, create” button.

Here comes the ugly part, since you cannot edit from the console the parameters (for the moment, I hope they are going to change that). You will need to log to your instance using SSH and download RDS cli from here: http://aws.amazon.com/developertools/2928?_encoding=UTF8&jiveRedirect=1

To do so right click on “Download” button and copy link location. In the SSH window use wget to download and unzip it:

wget "http://s3.amazonaws.com/rds-downloads/RDSCli.zip"
unzip RDSCli.zip

If you don’t have unzip you can quickly get it using “apt-get install unzip”(for ubuntu) or “yum install unzip”(for centos). Of course you will need root privileges.

After successfully unpacking the RDSCli cd to that directory and set a few variables. Following is an example on Ubuntu 10.04:

cd RDSCli-1.4.006
export AWS_RDS_HOME="/home/ubuntu/RDSCli-1.4.006"
export JAVA_HOME="/usr/lib/jvm/java-6-sun"
cd bin
./rds --help

If rds –help outputs no errors then you have set it correctly. Congrats. One more command:

./rds-modify-db-parameter-group mygroup --parameters="name=log_bin_trust_function_creators, value=on, method=immediate" --I="YOUR_AWS_ACCESS_KEY_ID" --S="YOUR_AWS_SECRET_ACCESS_KEY"

The AWS keys can be obtain from your AWS account Security Credentials->Access Credentials->Access Keys.

Go to AWS RDS console, “DB Instances”, select your instance and right click “Modify”. Set “DB Parameter group” to “mygroup” and check “Apply Immediately”. Confirm with “Yes, modify”.

You are done 🙂

Mysql benchmark: RDS vs EC2 performance

the setup: 1 m1.small ec2 instance vs 1 db.m1.small rds instance, tests are being run from the m1.small instance. The goal is to determine how the site will perform when moving the database from localhost to a remote instance.

I used sysbench for mysql benchmarks. On a linux server running ubuntu 10.04 you can simply install it with the following command(it’s obvious but just in case):

sudo apt-get install sysbench

The first tests performed were m1.small EC2 instance running mysql-server 5.1.41-3ubuntu12.8 VS RDS instance type db.m1.small running mysql server 5.1.50. The test database had been set to 10 000 records, number of threads = 1, test oltp.

sysbench --test=oltp --mysql-host=smalltest.us-east-1.rds.amazonaws.com --mysql-user=root --mysql-password=password --max-time=180 --max-requests=0 prepare
sysbench --test=oltp --mysql-host=smalltest.us-east-1.rds.amazonaws.com --mysql-user=root --mysql-password=password --max-time=180 --max-requests=0 run

The results

m1.small EC2 instancedb.m1.small RDS instance
OLTP test statistics:
queries performed:
read: 263354
write: 94055
other: 37622
total: 395031
transactions: 18811 (104.50 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 357409 (1985.56 per sec.)
other operations: 37622 (209.01 per sec.)
Test execution summary:
total time: 180.0044s
total number of events: 18811
total time taken by event execution: 179.7827
per-request statistics:
min: 4.04ms
avg: 9.56ms
max: 616.04ms
approx. 95 percentile: 38.42ms
OLTP test statistics:
queries performed:
read: 188230
write: 67225
other: 26890
total: 282345
transactions: 13445 (74.67 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 255455 (1418.74 per sec.)
other operations: 26890 (149.34 per sec.)
Test execution summary:
total time: 180.0573s
total number of events: 13445
total time taken by event execution: 179.9174
per-request statistics:
min: 9.08ms
avg: 13.38ms
max: 904.58ms
approx. 95 percentile: 20.99ms

As you can see the EC2 can perform 40% more transactions than the RDS instance. Nothing unexpected so far.

Time to move on and increase the number of threads to 10

m1.small EC2 instancedb.m1.small RDS instance
OLTP test statistics:
queries performed:
read: 264866
write: 94545
other: 37818
total: 397229
transactions: 18899 (104.97 per sec.)
deadlocks: 20 (0.11 per sec.)
read/write requests: 359411 (1996.22 per sec.)
other operations: 37818 (210.05 per sec.)

Test execution summary:
total time: 180.0462s
total number of events: 18899
total time taken by event execution: 1799.9289
per-request statistics:
min: 4.08ms
avg: 95.24ms
max: 2620.70ms
approx. 95 percentile: 445.91ms

OLTP test statistics:
queries performed:
read: 343812
write: 122772
other: 49109
total: 515693
transactions: 24551 (136.18 per sec.)
deadlocks: 7 (0.04 per sec.)
read/write requests: 466584 (2588.13 per sec.)
other operations: 49109 (272.41 per sec.)

Test execution summary:
total time: 180.2788s
total number of events: 24551
total time taken by event execution: 1801.8298
per-request statistics:
min: 13.41ms
avg: 73.39ms
max: 1126.02ms
approx. 95 percentile: 143.83ms

In this test the small RDS instance is faster than the EC2, 136 vs 105 transactions per second. I’ve also benchmarked a large RDS instance (the next one available after db.m1.small) and it got 185 transactions per second. Quite good, but the price is 4x higher.

The next test was performed vs a 10 million records, 16 threads. This time I only benchmarked a small and a large RDS instance. The large instance managed to do 228 transactions per second while the small one got a decent score of 127 transactions. One thing I noticed during this test is that the small instance started to use it’s swap, while the large one did not have this issue. This is probably due to the fact that 10M records db is aprox 2.5GB and the small RDS only has 1.7GB of RAM.

So if you are planing to grow and want an easy way to do it, switching your database to its own RDS is one of the first things you should consider. One of the immediate effects you will notice is that the CPU usage on the EC2 instance will be greatly reduced, leaving more power for the web server. You can easily increase the size and capacity of the RDS instance with just a few clicks. The backups are done automatically, which is great considering how many times I had to recover databases.

Mysql max_allowed_packet error

You are probably here because you tried to import a big database (several GB) and got the following error:

ERROR 1153 (08S01) at line 2533: Got a packet bigger than 'max_allowed_packet' bytes

If you have access to your mysql server and SUPER privileges things are easy, you just need to log in as superuser to mysql and type this:

mysql>set global max_allowed_packet=64*1024*1024;

and then import the database normally, just adding “–max_allowed_packet=64M” to the parameter list. Example:

$mysql --max_allowed_packet=64M database < database.sql

Everything is so easy. But if you are using Amazon RDS you are out of luck. You setup a user when you create the instance but of course it doesn’t have the SUPER privilege so if you try to execute the above command it will fail. Not even “grant super on *.* to myuberuser” will help you, no no. So after some googling and reading a lot of crap I found this blog which had the same error as mine. Yuppy! Thanks Henry!

The solution is to use DB Parameter Groups. Grab your mouse and start copy pasting fast.

Download Amazon RDS Command Line Toolkit
The latest version can be found here

wget http://s3.amazonaws.com/rds-downloads/RDSCli.zip
unzip RDSCli.zip
cd RDSCli-1.3.003 (this will surely change so make sure you cd to the right directory)
export AWS_RDS_HOME=`pwd`
export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk (this may vary depending on your java location and may not have to set it)
cp credential-file-path.template credential-file
vi credential-file (set your aws credentials there, use whatever text editor you like)
export AWS_CREDENTIAL_FILE=${AWS_RDS_HOME}/credential-file
cd bin
./rds --help

If everything went well you should get some output. On Henry blog he says he suggests that you create a parameter group. Well the reality is you have to create it since Amazon won’t let you modify parameters inside the default group.

./rds-create-db-parameter-group mygroup -f MySQL5.1 -d "My group"
./rds-modify-db-parameter-group mygroup --parameters "name=max_allowed_packet,value=67108864,method=immediate"
./rds-modify-db-instance YOURINSTANCENAMEHERE --db-parameter-group-name mygroup

Go to Amazon management console and check that the new parameter group is created and applied to your instance. You can begin now the import as you would do normally just add “–max_allowed_packet=64M” to the list of your options.

Hope it helps!

Amazon announces Free Usage Tier

Beginning from 1st of November you will be able to run a micro instance on Amazon cloud for free. Micro instance has 613 MB memory, up to 2 ECU(for short period bursts),32/64 bit platform and only has access to a 10 GB EBS(provided also for free). Also you get 5 GB to use with S3(simple storage) which is great for saving images (for example) and 30 GB data transfer(15 GB in and 15 GB out).

I believe this is a great opportunity to see what AWS is and also you may start to develop applications for cloud. You will probably want to use the EBS for MySQL storage and the S3 for media files.

Make sure you read all about this on their site before you use it and note that if you exceed the micro instance limits you are going to pay per use. Although the rates are low it never hurts to double check if you want to stay at 0$/month.

Once this offer is available (starting from 1st November) I am going to start testing a lot of stuff like how WordPress performs on this instance and how to install it so make sure you visit my blog from time to time! If you are interested in certain software installation on the amazon cloud platform feel free to send me an email and I will see if I can help.

Zend Framework provides cloud support

In a recent release announcement (for ZF 1.11.0 beta 1) I noticed that they now have Zend_Cloud which is, based on their description:

Zend Framework 1.11.0 markes the first official, stable release of Zend_Cloud, Zend Framework’s PHP version of the SimpleCloud API.

It offers support for some of the Amazon and Windows cloud solutions like S3/SQS and BLOB(spelled wrong Windows Azure’s blog storage)/Queue Service. While I didn’t work with Microsoft’s Azure, the S3 from Amazon is commonly used for all kinds of file storage. I believe it is great to be able to save files on S3, never having to fear about hard disk limitations and to some degree about their safety. While for small applications it might not be an obvious better solution, for applications that deal with a lot of files(like image storage for example) it is a big plus.

Although at the moment only S3 is interesting for me they stated they will be adding new features in the near future. Can’t wait that!

mEgo is sky high

Yes ladies and gents, it is true! mEgo Inc, my former company where I have worked for the last 2 years as sysadmin, made the move to Amazon Elastic Compute Cloud (aka EC2) today!!!11oneone It’s a moment of great joy to see this thing happening now at the beginning of 2010.

Best wishes to the mEgo team!

LE: Proof

PING mego.com (67.202.39.53) 56(84) bytes of data.
64 bytes from ec2-67-202-39-53.compute-1.amazonaws.com (67.202.39.53): icmp_seq=1 ttl=48 time=10.6 ms