HA cluster implementation using LAMP and Ultramonkey

We’ll now go into the specifics of this HA cluster implementation in detail in the section below. The gaming website receives millions of hits from all over the world. The HA cluster implementation is aimed at balancing the http traffic over the clustered web-servers as well as providing high availability.

Load Balancing :

> The gaming website is hosted on a LAMP clustered setup, and the heavy HTTP traffic to the site is distributed over a six-node Apache cluster with the aid of a load balancer setup in front of the Apache clusters. To ensure that the load balancer does not become a “Single Point of Failure’, the load balancer too is made up of two nodes forming an Active/Active configuration( LB-1 and LB-2), that monitored each other using a heartbeat, therefore ensuring that if one load balancer fails, the other would take over silently.Both these load balancer nodes have a public IP address and homogeneous configuration.

> DNS Round Robin is used to route the website to the public IP addresses of the two load balancers.The dns is configured in such a way that the domain name has multiple A records pointing to the public IP addresses of the load balancers, and the RR data returned in a dns query is rotated in cyclic manner to balance requests between servers. In this manner, the dns requests are equally distributed to the two load balancers – LB-1 and LB-2.

> The load balancers are configured in such a way that it would send requests to the Apache node with less load, as well as take care of connections and sessions to ensure smooth delivery of the website. The HA and load balancing solution is actually achieved by Ultramonkey installed on the load balancers. We will now come to how Ultramonkey(UM) handles this effectively:

– HTTP requests for the website reaches either of the load balancers based on the dns round robin mechanism. Ultramonkey installed on the load balancer recieves this request and sends it to one of the real http servers thereby making the clustered web server appear as a single web server to end-users.

– UM does this load balancing seamlessly by making use of the Linux Virtual Server or LVS. LVS enables TCP/IP and UDP connections to be load balanced with the help of Layer 4 Switching mechanism(ie. with the help of IP address and port information) to make load balancing decisions. The host that LVS runs on is refered to as the Linux-Director which is our actual load balancer.

– On the Linux Director, we have virtual services running , in our case a virtual HTTP service defined by an IP Address, port and protocol. The IP here is the global IP of the load balancer and port/protocol is specified based on the service we need(Port 80/tcp). By doing this, the linux director acts as an interface that receives the http requests from end users and passing it to the real servers.

– When an end-user sends a packet to the above virtual service, linux-director with the help of a scheduling algorithm decides which real-http-server to send the packet to and forwards the packet. When the server replies, linux-director forwards this reply back to the end-user.

– Since we have two load balancers to ensure high availability, UM uses heartbeat installed on linux-directors to ensure availability of both load balancers, and it does this by sending messages to the linux directors at regular intervals. If a response message is not received from a particular load balancer ,then the machine is assumed to have failed and heartbeat on the active load balancer takes up the IP address of the failed load balancer. In such a scenario, requests to both public Ips setup using dns round robin will be handled by the active load balancer node.

– A service called Ldirectord running on the linux directors handles the monitoring of the real servers and their insertion and removal from the pool of servers available. It monitors the health of the real-servers by periodically making a request and checking for an expected response. For HTTP servers this means requesting a known URL and checking that the response contains an expected string. This is called a “negotiate” check.If a real-server fails then that server is made quiescent and is reinserted once it comes back on line.

– Ganglia Cluster monitoring software is installed on one of the load balancers to monitor the availability of the cluster nodes and for maintaining a history of the node-availability statistics.And the ganglia monitoring daemon (gmond) is installed on all the six clustered webserver nodes. This daemon uses a simple listen/announce protocol via XDR to collect monitoring state and then shares this information via XML over TCP.

Data Management

We now come to how the data management is handled in our cluster configuration.

All the six real servers have a LAMP installation, and the web data is mainly replicated on all the six clustered servers via rsync to reduce I/O throughput, while variable data on the website is mounted via NFS from a single machine. One of the six nodes is tagged as a master node for making changes to the website and all the other nodes rsync to this master node.This storage approach has the benefit of never requiring any software mirroring of data between cluster nodes, thus saving precious CPU, I/O and network resources. There is also a script setup for rsyncing to the other nodes, that is executed only when file changes are made to the website data on the master node.Running this script ensures that the website content is synchronized on all the nodes.

The mysql databases used by this website is maintained on a single high end dedicated dual quad core machine capable of handling high I/O, with all the clustered nodes connecting to this single machine for database access.This server is maintained solely for mysql access and it also had its mysql data folder synchronized with a backup folder on one of the clustered nodes in the event of a server failure.

The part of the website that hosts variable data such as user modifiable folders that permit write access to users is maintained only on the master node with its backup on one of the other nodes, and all the other nodes mounted this data via NFS. The NFS option did pose a performance bottleneck due to which other options for data integrity is being considered for future scalability.

The figure below explains the HA cluster implementation :

HA Cluster Implementation

These high-availability(HA) or Failover clusters also helped to eliminate any single points of failure by having multiple nodes and load balancers that take over in case of a service disruption or fault in any of the active nodes. This solution helped the company grow by leaps and bounds over the past few months.

* Name of the client cannot be revealed as per the NDA (Non-disclosure Agreement). References and testimonials are available.

This client* is basically a provider of e-commerce based hosting solutions. They deal with plenty of sensitive and important data. Hence, becoming PCI Compliant was mandatory for them. Recently they were getting too many potential customer queries whether they are PCI Compliant.

What is PCI Compliance ?

A PCI Scan tells you what could be potentially insecure about your server. This is particularly important where storage of sensitive data occurs. Therefore, PCI Compliance is something which is preferred by most credit-card companies these days.

The PCI Security Standards Council talks about 12 basic requirements broadly divided into 6 categories. This is called the PCI-DSS (Payment Card Industry Data Security Standard)

This is required in order to avoid data frauds where card information is stored.

You can find these details here

The approach

Following are the important steps that we took in order to ensure that their cPanel servers pass the PCI Scan :-

- Installed a firewall : A server is not likely to pass the PCI Scan if there are unnecessary open ports. We installed CSF on the server. Alternatively, APF can also be used. We closed all the ports except for the ones required for the essential services. Certain standard ports like 2082, 2086 and 2095 could produce a negative result. So, we configured WHM to use the secure ports only.

- Updated the packages : Just run /scripts/upcp to update all the packages. Also, we had to make sure that Apache , PHP and MySQL were running the latest version.

The suggested versions are :

MySQL 4.1.22 or above
PHP 5.2.5 or above
Apache 1.3.39 or above ( Certain scans might require Apache 2.0.x )
OpenSSL 0.9.7j or above

cPanel suggests that you should keep cPAddons up to date as well. We did all the above on their servers.

- Disabled mod_userdir : If a site on the server can be accessed as http://serverip/~username then it means that mod_userdir is ‘enabled’. For PCI Compliance, this should be disabled and that is exactly what we did. It can be done via WHM > Security Center > Apache mod_userdir Tweak

- Installed SSL : At least, one SSL certificate from a recognized certificate authority is required. We installed SSL for Apache. SSL can be installed for other services as well.

- Apache Setup should not be revealed: We all have seen the ‘404 Error’ page at some point. Information about the Apache Setup should not be available on that page. We did this by adding the following lines to the ‘httpd.conf’ file :

ServerSignature Off

ServerTokens Prod

FileETag None

- Disabled SSLv2 and other weak encryption methods : Some services doesn’t allow you to
choose between SSL protocols but most PCI Scan overlook it.
The Weak SSL cipher issue has been an headache for people who want to pass the scan. We disabled SSLv2 on the servers.

- mod FrontPage – It is likely to cause a scan failure. Therefore, we kept it disabled.

- 2 factor authentication – This is another suggestion by cPanel that we adopted. A 2-factor
authentication procedure which requires a key and a pass-phrase.

- Besides all these, another important measure that we took was running the Nessus Scan.

It is a wonderful freely available tool to find any vulnerabilities on your server. You can find the details on the official Nessus website – http://nessus.org

Nessus basically consists of two parts, the server and the client. Once you are done with the two installations you need to add an user for the scanner and then you can start a scan on any remote server. The scan might take a while. It will give you a detailed report about all the package related vulnerabilities and any security loopholes. The best thing about Nessus is that it will also give you suggestions on how to fix those.

Thus, Nessus will tell you almost everything that needs to be done in order to achieve PCI Compliance.

We got a list of vulnerabilities as the result of the scan. We then went ahead and fixed them one by one. This was of immense help in our goal to get the servers PCI Compliant.

The result

What else? The client achieved PCI Compliance. Achieving this has helped them grow their business by almost 50% in last couple of quarters. The trust that the customers had in the company increased. Word of mouth is the best mode of publicity. That is exactly what happened here resulting in quick and sustained growth.
Note that different scan companies have a different approach . Hence, the requirements vary and they might have many more than the ones mentioned above. But these are the very basic ones that need to implemented for sure. We hope that this write-up would be of some help to those looking forward to achieve PCI Compliance.

* Name of the client cannot be revealed as per the NDA (Non-disclosure Agreement). References and testimonials are available.

The Challenge :

The biggest challenge was that most people were new to rails. Majority of the account holders were people who had just started learning Rails. Providing support to them was not an easy task. Moreover, since the clients are developers, dealing with them needs lot of knowledge. Many Rails hosts fail because support is not very good and developers take their business to someone who knows Rails well. Talent availability in Rails hosting is low as it is a relatively new field. It was not easy to manage servers which have the latest software installed on them. Moreover, dealing with support requests based on a relatively new framework and maintaining the guaranteed uptime was a daunting task.

Rails is designed to serve a dedicated environment. Lots of stuff are cached in memory which makes the sites work faster. The server RAM is utilized more in order to fulfill the browser requests quickly. Therefore,it is very difficult to manage a huge shared environment where hundreds of processes are cached in the memory.
Shared hosting companies usually rely on the fact that users would only use a fraction of the resources guaranteed to them. This does not hold good when it comes to Rails hosting. It is not possible to pack as many users on a server as you can using Perl/PHP. Loads would frantically go up resulting in crashed servers and unhappy customers.

The Solution :

This company teamed up with us for Outsourced Technical Support and Server Administration Services . Everything was ready and the hosting company was fully functional in a couple of weeks time. After signing on the dotted line, a pool of techs were drafted into the team, organized some technical corporate sessions and worked overtime to make sure everything starts off smoothly. We set up special Rails training programs and offered developer level training.
At the same time, we also had a stand-by team who did a great job studying Rails. Soon, we started doing deployments for developers, thus creating a very positive feedback in the developer community. We also ensured robust servers that could handle resource heavy Rails applications. As time passed by, the number of techs working in the team was increased according to the requirements and also to make sure that each and every aspect of the hosting company is working perfectly.

The Result :

Today, three years down the line, they have grown to some 70+ servers. Besides consistent service , the company is also known for adding its own brand of personal touch to further support the requirements of its customers. Satisfied clients constantly appreciate the company’s effort in providing a complete hosting solution with a personalized flavor.

* Name of the client cannot be revealed as per the NDA (Non-disclosure Agreement). References and testimonials are available.

Most of the servers run stable without giving any problems at all. But a handful of them used to freak out regularly. The servers are monitored using Nagios. It were the same names , most of the times , which came up with critical alerts. Though most of the issues were resolved within minutes , the customers started getting irate over a period of time. More and more users were getting dissatisfied. This was evident from the number of support tickets and chat sessions by the same people over and over again. At this point, we decided to take this matter more seriously as the client could easily loose out on potential business if this continued.

We hand-picked few people from the dedicated support team particularly to deal with this problem. They were assigned the task of exclusively monitoring these problematic servers 24×7 . We have always believed in the theory that the best way to deal with a problem is to eliminate the root cause itself. That is exactly what we did in this case. We observed trends in load spike and identified the causes. We also identified a few accounts which were causing most of the problems. We realized that these accounted for 90% of the downtime causing inconvenience to other users on the server.

Now that the problems were identified, the bigger challenge was how to counter them. We had a discussion with the client about this and decided to migrate such accounts to less populated servers. This measure showed immediate results. But we realized that we need to have a strategy in place to deal with this in the long run. This was necessary due to the fact that just few accounts were eating up most of the resources. We sent out notifications to such users. Some of them were persuaded to stop the scripts which caused heavy load. The company identified a huge business opportunity in this situation. They offered VPS and dedicated hosting options to the high load customers at discounted prices. While some of them agreed, few of them moved away as well. But still the company was in profit.

Thats not all, this streamlined usage also allowed them to keep the light accounts together on servers and such servers could hold 50% more accounts without any problems. That meant more business, higher profit ! At the end of the day, it’s not all about money. Customer satisfaction increased substantially. They were happy as there were no more downtimes. Their sites were loading faster and applications never hung up. Within the next quarter, there was a 300% increase in new sign-ups. All this happened by word of mouth.

Thus, the hosting company got a serious profitability boost by making servers lighter and freeing up capacity to add many more accounts. Also, through selling larger packages, VPS and dedicated hosting for several of the heavy users. Along with this came , increased customer satisfaction and decrease in the number of support requests This is a perfect example of the saying – There’s an opportunity in every crisis !

* Name of the client cannot be revealed as per the NDA (Non-disclosure Agreement). References and testimonials are available.

When the client came to us , they already had a fairly large team. But they were unhappy with its functioning which was far from being productive. This was the result of an highly disjoint team which did not work in co-ordination. Their techs were working remotely from different geographical locations. There was a complete communication gap between the supervisors and the staff. This made things worse. On top of it, they did not have any processes to check, measure and ensure high quality and efficiency.

This is when they approached us and we put forward a fully managed solution for them in which we asked them to outsource an entire process to us. Though they were a little circumspect initially , we convinced them to opt for a completely risk free solution. We offered them a free trial period of 15 days during which they can see what exactly our service is like and make out the difference. They readily accepted the offer as they had nothing to loose!

At InstaCarma, YOU define quality and WE assure it !

We asked them to define what quality means to them in terms of service parameters, policy adherence, technical ,language and customer service skills. We then documented all this and used various Quality matrices to formulate a Quality
Index . This QI was promised to the client via an SLA ( Service Level Agreement) .Once we deployed the techs to the client, we constantly kept checking their work for adherence to these quality parameters. Our QA managers were coached on the client’s expectation. They in turn checked tickets and monitored the techs’ work to make sure that the Quality Index matched the client’s expectations.

The Quality Index is of utmost importance to us because it helps us measure quality , which is really important and something which was lacking in the client’s existing support structure.
InstaCarma believes that unless something can be measured, it cannot be controlled.

The client was extremely satisfied by the end of the trial period . Instead of the one process which was ticketing , they also asked us to handle their Live chat support as well. They signed a long term contract with us and we provided them an end to end managed solution relieving them of all management headaches. We also provided them with special facilities like Basecamp and video conferencing so that they can be in touch with the team and have direct control over them when required. We also assigned a dedicated account manager for them who was the first point of contact regarding any issue.

The co-ordination between the techs improved and the overall efficiency increased by leaps and bounds. This saved them lot of costs and relieved them of the management problems as well. They were able to invest that time and money in further business development. Its been three years since and they have grown with us throughout. A company which had a couple of dozen servers when they started of with us have hundreds now. Also, they were able to venture into wider areas like VPS.

* Name of the client cannot be revealed as per the NDA (Non-disclosure Agreement). References and testimonials are available.