Posts

Rackspace vs Amazon impressions

I’ve been a customer of Amazon Web Services since they have been in beta. I’ve also worked with Rackspace now and then. Lately I’ve been working extensively with Rackspace infrastructure so I have a better understanding of their products. Bellow there are a few things that bother me at Rackspace:

Instances

The thing you are going to use 99% of the time with any of the providers. Amazon offers normal instances, high cpu instances, high memory instances, huge memory and cpu instances. On Rackspace you will have a very limited offer of instances. No high cpu or high memory. They do have plans to add them in nearby future. Until then, if you have a CPU bottleneck then it’s tough luck: scale up the instance and pay double.

And since we are talking about pricing for instances, Amazon is cheaper on the long run due to Reserved Instances (you pay an upfront fee and lower price/hour for the instance after). Amazon has been cutting down prices several times. Rackspace? Never.

Block storage

Called EBS on Amazon or CBS on Rackspace, it’s the preferred way of adding extra space to running instances without upgrading them. Beware that on Rackspace it takes an horrible amount of time to do a snapshot. I had to contact RS support several times because of this.

MySQL instances

Called RDS on Amazon or Cloud Database on Rackspace, they are basically optimized instances that run MySQL, but you are not given SSH access to them. Here Rackspace has the worst offer:
no scheduled backups (RS recommends mysqldump … lol try to do mysqldump for 10GB+ of data)
no replication (really? I couldn’t replicate my database)
no hotspare (Amazon MultiA-Z)

The good thing about Cloud Database is that they are fast. Really fast. Also RS promised to address all the above things in the nearby future (yeah).

Load Balancing

Rackspace does not support LB inside private network. Come on RackSpace, do your homework. This thing has been working for AWS since ancient history.

Conclusion:

There are many other things that need to be discovered, probably not pleasant ones. So far I am disappointed with Rackspace and what it has to offer and I would recommend to any customer to use Amazon instead.

Installing memcached with repcached patch for HA memcache cluster

About

Repcached is an interesting patch for memcached which allows replication between 2 memcached nodes (servers). The purpose of this article is to setup 2 memcached servers that replicate each others.

Note: This article is specifically written for Ubuntu 12.04 and memcached version 1.4.13. It may or it may not work for other versions.

Setup

2x 512MB RackSpace instances called node01(IP 10.0.0.1) and node02(IP 10.0.0.2)

Patch for memcached v1.4.13:
https://github.com/usecide/repcached/blob/master/repcached-2.3.1-1.4.13.patch

Build

Prepare to build the package:

apt-get build-dep memcached
apt-get source memcached
cd memcached-1.4.13
wget https://github.com/usecide/repcached/blob/master/repcached-2.3.1-1.4.13.patch
patch -p1 -i repcached-2.3.1-1.4.13.patch

Now edit the file debian/rules and look for config.status and add –enable-replication like this:

config.status: configure
        dh_testdir
        CFLAGS="$(CFLAGS)" ./configure --host=$(DEB_HOST_GNU_TYPE) 
                                       --build=$(DEB_BUILD_GNU_TYPE) 
                                       --prefix=/usr 
                                       --mandir=$${prefix}/share/man 
                                       --enable-replication 
                                       --infodir=$${prefix}/share/info

Now build the package:

dpkg-buildpackage -us -uc -nc
cd ..

You should see a package named memcached_1.4.13-0ubuntu2_amd64.deb. Copy this file on both of your memcached servers and install it using this command:

dpkg -i memcached_1.4.13-0ubuntu2_amd64.deb

Configuration

You are almost done now. Kill all running memcached processes on both nodes:

killall -9 memcached
ps aux | grep memcached

On node01 do the following things:

cp /etc/memcached.conf /etc/memcached_server1.conf

Edit /etc/memcached_server1.conf and replace the line:

-l 127.0.0.1 

with

-l 10.0.0.1

and add at the end:

-x 10.0.0.2

Start memcached:

service memcached start

On node02 do something similar:

cp /etc/memcached.conf /etc/memcached_server2.conf

Edit /etc/memcached_server2.conf and replace the line:

-l 127.0.0.1 

with

-l 10.0.0.2

and add at the end:

-x 10.0.0.1

Start memcached:

service memcached start

Testing

From node01:

telnet 10.0.0.1 11211:
Escape character is '^]'.
get hello
set hello 0 0 4
abcd
get hello
VALUE hello 0 4
abcd

From node02:

telnet 10.0.0.2 11211:
Escape character is '^]'.
get hello
VALUE hello 0 4
abcd

How to setup Galera 3 node cluster on Ubuntu 12.04

Galera is a multi-master replication solution for MySQL, which provides an interesting alternative to the standard master-master MySQL replication we are all so used with. One main advantage of Galera is the ability of doing sync replication, thus reducing the risk of data inconsistency between masters.

Setup on RackSpace Cloud

3x 512MB RAM instances, with 20GB storage space
1x Load Balancer for MySQL, RoundRobin algorithm, Health check enabled
1x 512MB RAM instance for testing
OS: Ubuntu 12.04 LTS 64bit

Goal:

Quickly setup a Galera cluster and run some benchmarks using sysbench.

Note: For the sake of simplicity I will refer to the Galera instances as node01, node02 and node03. The test instance will be referred as test01.

Common settings on all nodes

On every node execute:

  1. An apt-get update and upgrade to bring the instances up to date.
  2. Install required packages
    apt-get install libaio1 libssl0.9.8 mysql-client libdbd-mysql-perl libdbi-perl
  3. Download Galera wsrep provider
    wget https://launchpad.net/galera/2.x/23.2.4/+download/galera-23.2.4-amd64.deb
    dpkg -i galera-23.2.4-amd64.deb
  4. Download MySQL server with wsrep patch
    wget https://launchpad.net/codership-mysql/5.5/5.5.28-23.7/+download/mysql-server-wsrep-5.5.28-23.7-amd64.deb
    dpkg -i mysql-server-wsrep-5.5.28-23.7-amd64.deb
  5. I had some issues and I had to create /var/log/mysql
    mkdir -pv /var/log/mysql
    chown mysql:mysql -R /var/log/mysql
  6. Secure the mysql installation and assign a good password to root user:
    service mysql restart
    mysql_secure_installation
  7. Create an user for galera nodes to use for connect/replication
    mysql -p
    mysql> grant all privileges on *.* to galera@'%' identified by 'password';
    Query OK, 0 rows affected (0.00 sec)
    
    mysql> flush privileges;
    Query OK, 0 rows affected (0.00 sec)
    
    mysql> set global max_connect_errors = 10000;
    Query OK, 0 rows affected (0.01 sec)
  8. Edit /etc/hosts and make sure you add all the nodes and their corresponding IPs

Galera setup for each node

Edit the /etc/mysql/conf.d/wsrep.cnf and change the values for the following variables:

Configuration for node01:

wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="galera"
wsrep_cluster_address="gcomm://"
wsrep_sst_method=mysqldump
wsrep_sst_auth=galera:password

Configuration for node02:

wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="galera"
wsrep_cluster_address="gcomm://node01:4567"
wsrep_sst_method=mysqldump
wsrep_sst_auth=galera:password

Configuration for node03:

wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="galera"
wsrep_cluster_address="gcomm://node02:4567"
wsrep_sst_method=mysqldump
wsrep_sst_auth=galera:password

Testing the setup

Now restart mysql on all the nodes and check out if cluster is working:

service mysql restart
mysql -p
mysql> show status like 'wsrep%';
+----------------------------+-------------------------------------------------------------+
| Variable_name | Value |
+----------------------------+-------------------------------------------------------------+
| wsrep_cluster_size | 3 |
| wsrep_ready | ON |
+----------------------------+-------------------------------------------------------------+

One more thing before you are done:
Edit node01 wsrep_cluster_address=”gcomm://node3:4567″ and restart mysql server.

Benchmarks were performed from test01 instance using sysbench 0.5 OLTP read-only complex test:

sysbench OLTP (ro) Galera cluster transactions vs threads
ThreadsTransactions/s
115
225
449
8103
16205
32390
64506
128653

galera-transactions-threads

sysbench OLTP (ro) Galera cluster avg response time
ThreadsAvg response timeMin response timeAprox 95%
16642131
27953135
48042153
87742136
167743143
328142142
6412548322
12819445427

galera-response-times

Benchmark Galera cluster vs MySQL master-master on RackSpace

Setup:

Before starting this I would like to point out that I have compared 2 instances(master-master) vs 3 instances(galera cluster) so the test is not correct/accurate. It’s more of a “what if I switch from master-master replication to 3 nodes galera”.

MySQL Master-Master replication:

2x 512 MB instances with 20GB of storage, Ubuntu 12.04 64bit, mysql-server 5.5 was used with no optimization changes to my.cnf, except the required changes for master-master replication.
1x LoadBalancer, RoundRobin algorithm

Galera 3 nodes cluster:

3x 512 MB instances with 20GB of storage, Ubuntu 12.04 64bit, mysql-server 5.5 from galera was used, with no changes to my.cnf, only required node changes were made wsrep.cnf.
1x LoadBalancer, RoundRobin algorithm

Test instance:

1x 512MB instance with 20GB of storage, Ubuntu 12.04 64bit running sysbench

sysbench --test=oltp --mysql-host=loadbalancer_ip --mysql-user=root --mysql-password=password--oltp-table-size=1000000 prepare

The tests were performed on a database of about 256MB size, InnoDB table(s). No optimization changes were made to default my.cnf files, except the required to setup replication.

sysbench OLTP transactions per second
TestMaster-MasterSingle nodeGalera cluster
1 thread,3m10.9717.1112
16 threads,1m, rw1541400
16 threads,1m, r only217158.7206
32 threads,1m, r only325160.79375

galera-cluster-vs-master-master

As you can see from the table and graph I had some issues performing sysbench for Galera cluster in rw mode for 16 threads. From what I have found on Internet it’s an issue with sysbench 0.4.12 so I will attempt to rerun the tests with a newer version.

Installing Scalr 3.5 Open Source on Ubuntu 12.04

This is an update on an older post of mine, one of my first articles regarding cloud computing. Much has changed since since 2008 when I have wrote this article “How to install Scalr on Ubuntu 8.10 EC2 Instance“.

For example the Ubuntu has evolved to 12.04 LTS (I am using LTS 64bit for this howto) and Scalr is now version 3.5. One thing didn’t change: it’s still a royal PITA to get Scalr open source working. Hopefully this howto will help you to install Scalr on your server. It doesn’t cover operating Scalr and other things, which I will address in future posts, if there is enough interest.

After you have installed Ubuntu 12.04 64bit server edition to your server or virtual machine the way you like it it’s time to start the update process:

apt-get update && apt-get upgrade

Now you are ready to run tasksel and select the following roles for your server: OpenSSH, DNS server, LAMP server

You will need to install some dev packages before going anything else:

apt-get install libcurl4-gnutls-dev make librrd-dev

Now it’s time for PHP5 related extensions:

apt-get install php5-curl php-gettext php-net-socket php5-mcrypt php-xml-serializer libssh2-php php-soap php5-snmp php5-rrd
pecl install pecl_http
echo "extension=http.so" >/etc/php5/conf.d/pecl_http.ini
pecl install rrd
echo "extension=rrd.so" >/etc/php5/conf.d/rrd.ini

Time to get Scalr code:

cd /tmp
wget http://bit.ly/scalr35
tar zxvf scalr35
cd scalr-3.5.r7704
cp -r app /var/www/
chown -R www-data:www-data /var/www/app

Create new database and import sql from sql/scalr:

mysql -p
mysql> CREATE DATABASE scalr CHARACTER SET latin1 COLLATE latin1_swedish_ci;
mysql> grant all privileges on scalr.* to scalr@localhost identified by 'password';
mysql> flush privileges;
mysql> quit
mysql -p scalr <sql/scalr.sql

While doing that import I’ve got a nice error:
ERROR 1054 (42S22) at line 2222: Unknown column ‘architecture’ in ‘field list’
Fix:
1) Drop database
2) Search sql/scalr.sql for “CREATE TABLE IF NOT EXISTS `role_images`” and add after platform:

`architecture` varchar(25) DEFAULT NULL,
`os_family` varchar(25) DEFAULT NULL,
`os_name` varchar(25) DEFAULT NULL,
`os_version` varchar(25) DEFAULT NULL,
`agent_version` varchar(25) DEFAULT NULL,

Reference:https://groups.google.com/forum/?fromgroups=#!topic/scalr-discuss/ApCGfrNXqz8

Configuration of Scalr is quite simple:

cd /var/www/app/etc
cp config.ini-sample config.ini
edit config.ini

Cron jobs required by Scalr? Just type crontab -e and add the following lines:

*/2 * * * * /usr/bin/php -q /var/www/app/cron-ng/cron.php --Poller
* * * * * /usr/bin/php -q /var/www/app/cron/cron.php --Scheduler2
*/10 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --MySQLMaintenance
* * * * * /usr/bin/php -q /var/www/app/cron/cron.php --DNSManagerPoll
17 5 * * * /usr/bin/php -q /var/www/app/cron/cron.php --RotateLogs
*/2 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --EBSManager
*/20 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --RolesQueue
*/5 * * * * /usr/bin/php -q /var/www/app/cron-ng/cron.php --DbMsrMaintenance
*/2 * * * * /usr/bin/php -q /var/www/app/cron-ng/cron.php --Scaling
*/5 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --DBQueueEvent
*/2 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --SzrMessaging
*/4 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --RDSMaintenance
*/2 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --BundleTasksManager
* * * * * /usr/bin/php -q /var/www/app/cron-ng/cron.php --ScalarizrMessaging
* * * * * /usr/bin/php -q /var/www/app/cron-ng/cron.php --MessagingQueue
*/2 * * * * /usr/bin/php -q /var/www/app/cron-ng/cron.php --DeployManager
*/2 * * * * /usr/bin/php -q /var/www/app/cron/cron.php --UsageStatsPoller
* * * * * root /usr/bin/php -q /var/www/app/cron-ng/cron.php --SNMPStatsPoller

Time to add a Virtual Host:

cat <<EOF> /etc/apache2/sites-available/scalr
<VirtualHost *:80>
ServerName scalr.example.com
ServerAdmin webmaster@example.com
DocumentRoot "/var/www/app/www"

<Directory "/var/www/app/www">
Options Indexes FollowSymLinks
AllowOverride All
Order allow,deny
Allow from all
</Directory>
</VirtualHost>
EOF

Enable required Apache modules and site and restart everything:

a2ensite scalr
a2enmod rewrite
service apache2 restart

DNS managed by bind9:

chmod g+w /etc/bind/named.conf
echo 'include "/var/named/etc/namedb/client_zones/zones.include";' >> /etc/bind/named.conf
mkdir -p /var/named/etc/namedb/client_zones
chown root.bind /var/named/etc/namedb/client_zones
chmod 2775 /var/named/etc/namedb/client_zones
echo ' ' > /var/named/etc/namedb/client_zones/zones.include
chown root.bind /var/named/etc/namedb/client_zones/zones.include
chmod g+w /var/named/etc/namedb/client_zones/zones.include

To get rid of nasty AppArmor warnings and errors edit /etc/apparmor.d/usr.sbin.named and add:

/var/named/etc/namedb/client_zones/zones.include rw,

And finish it by restarting AppArmor and bind9:

service apparmor restart
service bind9 restart

Open your browser and go to http://scalr.example.com. Default username/password: admin/admin.

If you have issues or you need more info please feel free to comment 🙂