2 Servers - Load Balancing

[Daz]

Hi,

I currently have a dedicated server, with cpanel, and recently had a network outage, resulting in the ecommerce site being down for several hours.

Not good for business...... So my question is, if i bought a second server with another host, how can i implement a fail safe system, where by

1. The primary server serves the shop 24/7
2. In the event of a server crash or network failure, i can switch to the secondary server
3. What software would you use to make the switch?
4. Are there any other better ways to do what i need?


Thanks in advance!

 

[penguin_comel]

i suggest u using linux and BalanceNG ..u can get free for that .

 

[Daz]

So your suggestion is to actually setup a load balancing solution, rather than have one server as the primary, and only have the secondary kick in if that fails or dies?

 

[penguin_comel]

So your suggestion is to actually setup a load balancing solution, rather than have one server as the primary, and only have the secondary kick in if that fails or dies?

it is the backup server and not balancing ..

if using the balancing , u will make the traffic and download to you site more faster . and if one down , the other is still up. and u not need to come to office at down time ..

eg if you server down at 3 am , are you like to come at that time ??

or use linux firewall or buy the firewall that support load balancing ..

sorry my English is pure

 

[Cromulent]

So your suggestion is to actually setup a load balancing solution, rather than have one server as the primary, and only have the secondary kick in if that fails or dies?

Load balancing makes more sense in the long run. What if your site gets much busier in the future and it requires 2 or 3 servers to keep it running fast? With load balancing you just add as many servers as you need and keep going.

With the other option you would need to reconfigure your entire internal network and most likely switch to load balancing anyway.

 

[FireFleur]

If you want failsafe, then setup a monitor, so it could be accessing a page, and then when it fails to access the page you go through various levels of redirect.

In the event that the server has completely toppled, then your redirect will be DNS or IP allocation, DNS could have a time constraint, and IP allocation is probably the best way but do inform the ISP

There are problems of session, but it would keep you up longer.

Load balancing has an overhead, normally it is done on 3+ plus hosts, but can be done on two if finely tuned, again session can be a problem.

 

[Daz]

Excellent advice thanks.

So just to confirm, load balancing can be done using 2 different hosts, in 2 different locations?

Otherwise if the load balancing is done on 2 servers in the same location, if the network connection goes down at that location, the load balancing becomes obsolete?

So does that mean the load balancing software would have to be installed on a 3rd host? ...Is that what you are referring to by 3+ hosts ?

Damn that sounds expensive?

 

[mke]

Essentially, I'm in agreement with FireFleur. You could also use database replication to ensure maximum consistency. That's not a trivial matter to set up robustly, though.

You can use session-aware software. What platform/stack are you on with the main site?

If stock levels are not critical, you could set up a VM image of the site and, redirecting DNS, fire it up when the primary fails.

Adding, having just seen your post done while I was typing, yes you can do that. A different power and connectivity supply source to the primary is the constraint. It doesn't have to be in another location although it usually would be more sensible. Yes, it ain't cheap.

 

[Dominic Taylor]

How far was the network issue from your server? If it was just a switch failure then get a new host because that shouldn't take hours to fix. If it was a router failure, get a new host because they should have a fully redundant network.

If it was something more complex than that then unfortunately such things do happen. But perhaps I'm biased because the most network disruption we've had is 20 minutes!

To do proper load balancing depends on how far away from your first server's network you want to balance....at minimum you'd need 3 servers with the MySQL server connected to either server.

You could use two servers and rsync the data but you can't copy a live MySQL instance in that way.

What would your budget be?

 

[Dwebs-Ltd]

Not going to be cheap or possible really.

You could have two servers with two providers and use round robin DNS but the problem is most clients DNS caches will last longer than the outage.

Every provider has outages even the famous rackspace.

If you were offline for several hours you need to find a different host.

 

[Dwebs-Ltd]

If it was something more complex than that then unfortunately such things do happen. But perhaps I'm biased because the most network disruption we've had is 20 minutes!

Over what period of time?

 

[FireFleur]

For the IP move you would have to be in the same physical location attached to the same router, or you will have a harder time than DNS reallocation.

It is admirable to look into keeping your servers, but it is the more awkward area of IT administration, and you have to go right through the software, hardware and network stack.

What you could do, is get three servers, one to load balance, that you spec. for stability to the nines, that one you don't want to go down. Then you use that one to monitor and balance to the other servers, which can then be anywhere. But, still you would be using things like a proxy thru, and trying to maintain state centrally, notifying the other nodes of a failure. It is interesting stuff, but it comes at a premium.

There are many variants, having multiple domains might not be a bad move, let the user auto balance. And often you can get the server up to deliver information that just says go to another server, by a redirect. There an ISP could offer that service, jump in quick with another box and redirect.

I think the latter advice is quite good, identify where your problems are first, most computer equipment just chugs on, and if there is a problem it can normally be quickly rectified.

 

[Dwebs-Ltd]

No matter what you do your going to end up with a SPOF unless you start doing geo-load-balancing which unless you have a spare £25-100k you are not going to be able to do.

 

[Daz]

We have used Xcalibre for the past 4 years, and have never had any downtime until now, i mentioned several hours, but it may have been a lot less.

Im happy with the company, but having downtime made me realise we need a better failsafe in place. I have emailed them and asked what they can offer.

We have the budget for a second server, but dont quite know the best way to configure the setup. (load balancing would be nice, but depends on cost).

Thanks for your feedback guys... much appreciated.

 

[Dwebs-Ltd]

We have used Xcalibre for the past 4 years, and have never had any downtime until now, i mentioned several hours, but it may have been a lot less.

What time of day was the outage and what did they give as the reason?

Does some research.......
http://www.xcalibre.co.uk/content/view/138/266/
1hr 45mins downtime

Bad time of day for downtime but it happens.

 

[mke]

Xcalibre are an established, known, quality, reliable host, Chris. We used to use them several years ago. Few points to be scored there.

 

[Daz]

In everyones experience, do server failures (software or hardware) occur more frequently than network outages?

Im wondering whether 2 servers, software load balanced, would be the cheapest & best option?

That way, its only the cost of the additional server? Although the software load balancer would need to be stored on a 3rd party server anyway? hmmmm, it's not very easy figuring this out...

 

[FireFleur]

I think you have to realise that load balancing is more for performance than fault tolerance.

Sure, there is an element of fault tolerance to load balancing, but it also introduces more potential points of failure. Load balancing is designed to manage the load of very popular servers.

So, you go there after sorting out the real problem, which is how to maintain availability of your site to customers.

For that a cheap way would be two domains, two servers, two ISPs, different data centres and talk to both ISPs about failure response. If they can put in a quick redirect in event of the server going down then you just ask to point it to the other server. If their entire network goes down, you are still sort of available, if people are aware of the other domain at the other server.

You could merge the servers every night or at some time increment that works for you.

 

[Erlen]

Before going deep into load-balancing stuff, you should really check your numbers. This is rather simple, you take 3 or 4 cases, like connectivity loss for 2h, 10h, data center destroyed by flooding, hard drive failure. For each case, you evaluate what it means for the number of hours out of business and what it means for work on your side.

So you have for example :

case 1: 2h out of business, no work
case 2: 4h out of business, 30 min work to answer email questions
case 3: 10h out of business, 8h to rebuild the server from backup
case 4: you name it

Now, you can evaluate the cost of each outage, each hour out of business is for example GBP100 of missing sales + 20% of "lost of quality image". Each of your working hour is GBP80.

For each case, you evaluate the probability to get that in the year. Now, you can easily calculate the yearly loss of the downtimes.

The trick is now to setup a solution that will cost you less than this downtime loss both in terms of work and money. You do not need to remove all the cases, you can consider that 2h without connection because of work on the line is ok. But you can also setup a system just to ensure that case 3 will be handled in 2h with just 1h of work for you.

At the end, you provide the best value for you and your customers. This is what I am doing for indefero . net.

 

[stugster]

You could also add two servers (one as the main, one as the backup), use rsync between the two. And then a third server for your DNS - which cPanel provide licences for free of charge.

 

[Dominic Taylor]

Over what period of time?

3 years or so? Plus a couple of minor <5min outages. None in daytime hours.

Don't come back at me with stats for Vidahost.com though, that's hosted separately in Manchester and, ironically, the network there is less reliable than our customer services....

Only other issues have been power but let's not talk about redbus unplugging the wrong rack....

 

[Dwebs-Ltd]

3 years or so? Plus a couple of minor <5min outages. None in daytime hours.

Don't come back at me with stats for Vidahost.com though, that's hosted separately in Manchester and, ironically, the network there is less reliable than our customer services....

Only other issues have been power but let's not talk about redbus unplugging the wrong rack....

So in fact more downtime than you said.

Downtime is downtime no matter if its power, human error or network providers issue.

We don't tell our customers we have had zero downtime but we have had downtime that wasn't our fault blah blah.

As a host we are totally honest with our customers if we balls up we admit fault, that's what the industry lacks honest providers who aren't scared of the truth.

 

[Dominic Taylor]

So in fact more downtime than you said.

Downtime is downtime no matter if its power, human error or network providers issue.

We don't tell our customers we have had zero downtime but we have had downtime that wasn't our fault blah blah.

As a host we are totally honest with our customers if we balls up we admit fault, that's what the industry lacks honest providers who aren't scared of the truth.

Sorry? The OP said he'd had 2hrs downtime due to a network issue. I said that's ridiculous as have others. There's zero reason why you should have such a prolonged downtime due to network issue - I said the longest network disruption we've had was 20 minutes, ie "there have been network problems but because of a well-designed network, they didn't last long at all".

When the wrong rack was unplugged the disruption in that rack was, again, only about 20 minutes...at around 7am on a Sunday.

There's all sorts of things that can cause extended downtime which are acceptable, such as a RAID array eating itself, but that's not what the OP had, so what he suffered wasn't acceptable

You can't say I'm not honest, the statistics are all in the footer of our site

 

[kapow]

It's quite possible to run Linux High Availability with Load Balancing on just 2 boxes.

Take a look at Ultra Monkey. We've used this for larger clusters of dozens of machines, but they do have a nice network layout which works with just 2 servers, Here; http://www.ultramonkey.org/3/topologies/sl-ha-lb-overview.html

This provides 1 Active Load Balancer and 1 Passive (waiting for a fault on the 1st box). Both machines are also working as ACTIVE HTTP/SMTP/Whatever servers... so that load is being balanced and if either one is too busy or has problems with it's client facing processes then the other is automatically used.

It wouldn't avoid network problems upstream (which are rare if your bandwidth is multi-homed), but maybe it'll help you avoid server issues?

Hope so.

A.

 

[leemason]

The solution depends on whether you need to share data between your two servers. Or will they be able to run independently without the need to synchronised their data.

Do you have a database backend, and if so what is it?

You are correct in saying that the load balancer software would sit on a front-end server (or 2). If you are doing true load balancing you are probably looking at using 4 servers. Two for load balancing and two for web serving.

If you just want primary/secondary servers you can implement this using Linux and Heartbeat software running on the two server. Each server has it's own IP address but then shares another (or more than one) to failover resources (i.e. httpd, mysql, etc) from one server to the other.

If you need data to be consistent and failed over between the two servers have a look ar DRBD. This allows a disk on one server to be mirrored to a disk on a second (failover) server, so when heartbeat fails over the mirror disk device is failed over as well and the resource continues to run on the second server.

If you want anymore detailed information then please PM me.

 

[PayVector]

No matter how much redundancy you have at one location data centers have issues from time to time. I cant remember which data center it was before christmas that had to shut down all power, including backup generators because an early fire warning detector went off. That meant all power to an entire floor gone until fire department let them turn things back on. I also remember a case in the states where a data center's main power box blew up and the data center was down for days.

The 100% uptime holly grail costs a fortune. That is the simplest answer. My advise is to put a second server in a separate data center. Your issue if your main site goes down is swapping DNS to your backup site. Set a short Time To Live TTL on your DNS but some customers may not see the backup site naughty ISPs like to ignore TTL records.

There are some services that monitor your site and if it can not be reached or does not respond properly will automatically swap your DNS so you do not have to do it at 3am.

Big incidents at data centers are not all that rare. Your most likely failure is going to be on your actual server. Make sure you spend money and get one that is fully redundant internally, 2 x nics, raided hot swap drives drives, redundant power supplies etc. Drives go the most often as they have moving parts so get some decent quality ones.

If you need data synched between two locations use the new version of mysql. Its free and is very fast.

Hope that helps.