123-REG down once again

[SimplyMyLove]

How many major downtime I had in the past quarter? Three or four?

Before I were aware of the issue and paused all PPC campaigns, £££ has been wasted on Google AdWords and other CPC sites.

Once the current contract expires, I would never use 123-reg again!!!

VPS connectivity issues
Created: 16 April 2016, 09:43
Last Updated: 16 April 2016, 12:40

Some customers on our VPS platform may currently experience connectivity issues. Our engineers are aware of this issue and are currently in the process of resolving this.

We apologise for any inconvenience this may cause.

We will provide a further update as soon as this has been resolved.

UPDATE 1:
Sys Admins are still investigating and we have all hands on deck to get this fixed.

UPDATE 2:
This morning, our teams were made aware of issues affecting the performance of our VPS product. Our teams are continuing to investigate which has led to some customers experiencing denigrated service levels.

Our tech teams are working hard to rectify for all customers and we thank you for your patience.
123-reg would like to apologise to customers affected and will continue to provide further updates.

 

[arnydnxluk]

Looks like there's another UKBF member hit by these issues too: http://www.ukbusinessforums.co.uk/threads/best-uk-vps-service.357363/

 

[ryedale]

How long have you got to go on on the contract now?

 

[Dan_HiHosting]

Sorry to read about your trouble with 123-Reg.

They really are poor and a bad idea for hosting. They get recommended a lot for domain registration as they're the biggest name in the UK, but it's not hard to get that right (and there are better and better value alternatives even for domains).

Hosting is a very different business, and larger accounts and a VPS requires care and attention, not their mass volume approach.

When does your contract end?

If it's not got much time left, you may find another host would absorb some of the cost for you, it's certainly something we've done in the past to help those that are suffering with a bad host to move over.

I hope this gets resolved soon for you.

Do make sure you're taking regular backups of your data and website, so download a copy of your database and website files. Then if you get more issues you can use that if necessary, and aren't relying on their backups (which may not be available, or requests for which they may respond to slowly).

I'd take a daily offsite database backup if you aren't already.

Downtime is far too costly for a business to put up with.

Best of luck,

Dan

 

[SimplyMyLove]

Many thanks Mark and Dan. I'll review my contract term with 123-reg as soon as the VPS becomes back on again. So far it is still down. What is more frustrating is the attitude of 123-reg customer service, no update on the issue, no answer on the phone, no reply to emails or support tickets. They literally blocked all ways of communication. 1000s of their VPS customers like myself are now left in darkness, wondering what happened and how much longer we still have to wait.

 

[SimplyMyLove]

UPDATE 3 (published at 4:19pm, after the server was down for nearly 8 hours):
We are still working on finding appropriate solutions for the affected VPS customers. We have allocated extra resources to expedite this. Thank you so much for the patience.

 

[SimplyMyLove]

Out of interest, what kind of connectivity issue would take so long to solve?

I assume it is not as easy as replacing a broken data cable.

The server is still down at the moment. But look at the brighter side, it's good to take a day off work and watch some girly DVDs with my daughters on a sunny Saturday.

 

[mzone]

123 they are offering a very cheap service, so many users placed at server can create a potential problem like high load, DDOS attack or many more.
But to be honest 8hrs downtime must be something serious.

 

[SimplyMyLove]

24 hour downtime now.

 

[Voicebooth]

You should look at soyoustart.co.uk which is the slightly cheaper arm of OVH.

 

[mzone]

Soyoustart is managed by OVH support team, that's very cheap service, in case of any problems with server he will stack again same place with long time downtime.

SimplyMyLove@ Is server location is important to you? What is your monthly budget?

 

[Dan_HiHosting]

OVH themselves would be a bad choice for a UK business, considering the location (France) and support you'll get, choosing their budget option's just asking for trouble.

The OP wants to solve their reliability issues. Offshore budget and unmanaged options known for their poor support and slow response aren't the way to solve that.

 

[Dan_HiHosting]

24 hour downtime now.

Sorry to read that. It looks like it could take them a long time to resolve this.

Have you got a local copy of your website and database?

If you want your website back up as soon a possible, you may be better off quickly getting a basic web hosting or eCommerce package and uploading a backup.

If you've got access to your domain you can then update the nameserver entries to point to that new, temporary hosting. If you were then to move to a new VPS I'd imagine a decent provider would credit anything you'd paid toward that, we certainly would.

Then at least your website is live as soon as possible.

 

[SimplyMyLove]

Thanks everyone. The VPS is down for more than 27 hours now. Just had a live chat with their customer service, and there is still no update as yet when they would possibly bring the service back on again.

 

[SimplyMyLove]

Thanks for your suggestion Dan. Yes, I have a backup of all files and images on my hard drive, and the MySQL database was backed up two days before. I am now seriously looking for a good hosting company to switch, and I don't care what 123-reg would say about the rest of my contract terms with them.

 

[mzone]

If you have backup you can have website restored in 2hrs, if you interested please send me an PM.

 

[ryedale]

The 123 issue is looking really serious, they're not putting out any updates at all since 11 last night and everything is still down.

 

[Dan_HiHosting]

Thanks for your suggestion Dan. Yes, I have a backup of all files and images on my hard drive, and the MySQL database was backed up two days before. I am now seriously looking for a good hosting company to switch, and I don't care what 123-reg would say about the rest of my contract terms with them.

No problem. That's good news that you have recent backups.
I'd seriously consider using your backups to bring your site back up with a decent host then, if you want to avoid more downtime.

You can get a new VPS up and running in a few hours easily, and a normal web hosting account can be instantly deployed.

It looks like your domain's with 123-Reg, so all you need to do is login there to change the nameserver entries to switch it to your new provider.

If you can any questions or we can be of any help then just let us know.

Best of luck,

Dan

 

[SimplyMyLove]

Finally, the latest update from one of the company directors.

Dear Customer.


I am writing to you to explain what happened to some VPS services on 16.04.16. This email is to detail what our steps have been. I am committed to open communication with all customers and would like to take this opportunity to explain in detail.


So what happened to some services? As part of a clean-up process on the 123-reg VPS platform, a script was run at 7am on 16.04.16. This script is run to show us the number of machines active against the master database. An error on the script showed 'zero-records' response from the database for some live VPS. For those customers, this created a 'failure' scenario - showing no VM's and effectively deleting what was on the host. As a result of our team's investigations, we can conclude that the issues faced having resulted in some data loss for some customers. Our teams have been and continue to work to restore. What have we done? We have been working with an extended team of experts and have left no stone unturned. Our teams have been working long into the night to restore as much as we possibly can. We have also invested in external consultants to recover, in the best way possible.


We have recovery running on the VPS servers and some are restoring to new disks. We have also begun copying recovered VPS images to new hosts and we expect some VPS to be back up and running throughout the night and in to tomorrow.


Our teams have worked for more than 24 hours and will continue to do so. No stone is being left unturned.


As the technical teams come back with updates for individual VPS we will communicate updates to customers.


For those customers with their own backup of their settings and data, if you wish to restore services yourself you can do this by issuing a reimage command through your 123 Reg control panel, this will give you a freshly installed VPS on a new cluster, where you can restore your service.


I understand that some customers may have lost some confidence in the service that we offer. So, I want to explain what we have done to prevent this happening again. We have started an audit on all cron-jobs and scripts controlling the platform, and associated architecture, so that no script will have ability to delete images, only suspend. For image deletion for those suspended over 28 days we will have a human eye to double check. A new platform will be available by the end of the year for customers which we will provide self-managed and automated snapshot backups, in addition to architecture technology to backup the whole platform, something that is not available on the current platform. I hope this goes some way to win back your confidence.


Richard Winslow,

123 Reg Brand Director

 

[GraemeL]

Not a hint of an apology or sense of being sorry!

I never have this problem with @Dan_HiHosting who looks after my VPS. He gives personal service and cares about customers.

 

[SimplyMyLove]

Their facebook page is flooded with complaints. Some customers even paid for the backup service at an extra monthly fee, and only found the backup files were also deleted on their VPS. So 123-reg has indeed deleted all data on the server, putting the blame on a little poor script.

 

[ryedale]

It's beyond comprehension that this could happen and the ramifications are horrendous for a lot of businesses.

I think it will make a lot of people think twice about using unmanaged solutions again.

 

[Dan_HiHosting]

Or about using 123-Reg for anything other than domains.

We knew they were bad and have migrated many clients from their services because of overcrowding, poor performance and service - but to write such a script in the first place, and then have no contingency or protection in place is staggering. It's a really amateur mistake. While overcrowding and so forth are expected from the mass hosts, mismanagement on this scale isn't I think.

@SimplyMyLove given you've thankfully got a recent copy of your data and there's no guarantee 123-Reg will even be able to restore your VPS, let alone soon, uploading your backups to a new host is what I'd do in your case.
If you complain to 123-Reg, given the scale of the problem you may be able to have your contract ended early. It's certainly worth trying.

 

[UKSBD]

123 aren't using this guy are they
http://www.independent.co.uk/life-s...mpany-with-one-line-of-bad-code-a6984256.html

 

[Dan_HiHosting]

I was just going to mention that story, although it's actually a hoax.
It's staggeringly similar!

 

[webhostuk]

Thats pretty bad , Its second major downtime in couple of years time.

 

[webhostuk]

123 aren't using this guy are they
http://www.independent.co.uk/life-s...mpany-with-one-line-of-bad-code-a6984256.html

It was fake story to promote his new startup ,his trick worked. He got lots of attention he was looking for.

 

[UKSBD]

It was fake story to promote his new startup ,his trick worked. He got lots of attention he was looking for.

I think the clue was in the fact he said he had approx. 1535 customers

 

[zigojacko]

I think it will make a lot of people think twice about using unmanaged solutions again.

Unfortunately, there is a lot of business owners that won't as well though as they're drawn in by the cheap monthly fees for hosting.

===

It's been said time and time again, and it should be evident in business, you get what you pay for. Pay for cheap hosting, that is exactly what you'll get, poor services, limitations, the bare minimum etc.

Too few actually think about how much their website uptime and a reliable hosting service is worth to their business. Many will learn the hard way, hopefully, most of these will think twice going forward.

Frustratingly, all the cheap providers are those massive companies with huge catchment ands is exactly those kind of services that hurt businesses. 123 Reg, Go Daddy, 1&1 etc...

 

[zigojacko]

I was just going to mention that story, although it's actually a hoax.
It's staggeringly similar!

Was it proven to be a hoax in the end? I didn't hear of that since the story went viral.

 

[zigojacko]

It was fake story to promote his new startup ,his trick worked. He got lots of attention he was looking for.

Wow, decent marketing!

Let's hope his start-up never wants to host clients though, haha.

 

[ryedale]

Unfortunately ZigJacko. following the events on Twitter, people doesn't seem to be learning

They are all jumping ship to Heart, Go Daddy, 1+1 etc

 

[zigojacko]

Unfortunately ZigJacko. following the events on Twitter, people doesn't seem to be learning

They are all jumping ship to Heart, Go Daddy, 1+1 etc

Crazy! All of which have the same major issues in their hosting business model... :/

 

[it-comms]

Why don't the big companies use VMware like we do? its not cheap on licences etc but is by far the most reliable technology and if used with veeam backups. there will be no data loss

 

[Dan_HiHosting]

Yeah, and it's not as bad in the UK as the US, where Endurance International Group dominate, but people can end up switching from one brand to another owned by the same company!

123-Reg is owned by Host Europe Group, who are like a smaller UK/German equivalent of EID in the US.

In fact, Heart Internet is owned by HEG, so there you go.
People will switch to Heart from 123-Reg, as they've also seen their mass marketing, thinking they'll get better service - when it's owned by the same company, and typically runs on the same infrastructure.

The fact that the division these brands fall under is called "Mass Hosting" says it all really.

 

[zigojacko]

Why don't the big companies use VMware like we do? its not cheap on licences etc but is by far the most reliable technology and if used with veeam backups. there will be no data loss

Because their entire business models are based on automation and 'do as little as possible for as cheap as possible'. They all focus on quantity and not quality.

 

[Dan_HiHosting]

Not a hint of an apology or sense of being sorry!

I never have this problem with @Dan_HiHosting who looks after my VPS. He gives personal service and cares about customers.

Thanks Graeme.

That's the the key difference really, we try to help our clients as best as we can, and we often then get recommended.
It's the traditional model of winning business by doing what you do well, and looking after your clients.

The mass hosts depend on marketing, and it's very difficult to maintain good service when you have that volume of customers, even if they cared.
They pretty much rely on people ignoring reviews and recommendations, and using an advert to make their decision instead.

Unfortunately plenty enough people aren't aware of that practise so it to works for them.

I'm glad you're pleased with our service .

 

[SimplyMyLove]

I've learnt my lesson, and will never ever use any unmanaged hosting service in my life.

A big THANK YOU to all guys here who offered me advice and help, especially to Mark @ VTSHosting, who worked on Sunday to get my new hosting account set up.

A lot of recovery work to do in the week ahead, but at least we are now on the right route.

 

[SeanDigitalSave]

Yep we're in the same boat. Website down all weekend again and still not back. Was in the same position only a few months back!!!!

Interesting post on CMS Critic called Avoid 123 Reg which was posted a few days before all this happened. I've also read about another guy who pointed out a security flaw to them 10 years ago which is STILL present!!

I rang them last week asking to increase my PHP limits, the guy just kept going on about how much RAM my server had, didn't understand at all.....

 

[Dan_HiHosting]

@SimplyMyLove Glad you're getting sorted.

Your domain is still pointing to 123-Reg though, and it can take some time for nameserver changes to propagate and go live across the Internet, so I'd change that as soon as possible to prevent further downtime for your site.