Voipfone Outages Getting Silly

[jakew009]

Our phones have now been off all morning... no one is able to dial in not even get through to voicemail.. for the second time in less than 10 days?

Voipfone's website doesn't work
Voipfone status page doesn't work

Anyone know what is going on? I think we are going to have to move away costing us a fortune in lost business now.

 

[UKSBD]

Follow them on twitter where they are keeping people updated - https://twitter.com/Voipfone or https://twitter.com/voipfonestatus

 

[Faevilangel]

Same here, just checked their twitter and got major issues.

I don't get that many calls but it's probably costing me a job or 2

 

[british steve]

Have you tried them on Twitter? They may have some updates!

 

[jakew009]

Have you tried them on Twitter? They may have some updates!

It's not really a matter of waiting for updates, outages on this scale just should not be able to happen - I thought they had a totally redundant network anyway? Going to be interesting to see what the explanation is...

 

[Faevilangel]

It's not really a matter of waiting for updates, outages on this scale just should not be able to happen - I thought they had a totally redundant network anyway? Going to be interesting to see what the explanation is...

Amazon has gone down a few times recently, it's one of those things, systems break for whatever reason and will go offline for a short period.

 

[Clouvider]

Amazon has gone down a few times recently, it's one of those things, systems break for whatever reason and will go offline for a short period.

The point is, that it is affecting their customers so they should keep it to the minimum. Yes, systems may brake, but if something brakes that frequently (as @jakew009 mentioned 2 times in 10 days) they need to act to prevent that happening in the future.

If you value your customers you need to learn a lesson on every failure and ensure it will not happen again.

 

[Faevilangel]

The point is, that it is affecting their customers so they should keep it to the minimum. Yes, systems may brake, but if something brakes that frequently (as @jakew009 mentioned 2 times in 10 days) they need to act to prevent that happening in the future.

If you value your customers you need to learn a lesson on every failure and ensure it will not happen again.

I have been with VF for over 4 years and this is the first outage I have known, and it's not actually VF that has gone down, it's their datacentre provider which has.

Servers break, nothing we can do about it and just need to ride the rough with the smooth, VF will get it sorted and be back running asap.

 

[DigitalDaz]

Good look with fixing it but why is it taking so long?

 

[Clouvider]

@Faevilangel I am not using them, my post was entirely based on the thread author, whose message was different.

 

[Clouvider]

Voipfone has posted in IT &Internet, sorry, can't link directly.

 

[cjd]

Sorry guys, I've been out all morning at our accountants and didn't know about this outage.

At the moment all we know is that one of our two data centres lost all connectivity and went totally dark. This affected hundreds of company - including banks.

We have our network duplicated in a second data centre in London and traffic went to there automatically. Sadly that started failing too, we don't yet know why because it's designed to cope with an extreme outages like this.

I'll get a full report out as soon as I can.

 

[Employment Law Clinic]

I often find it odd to read posts about Voipfone (today) or Vidahost (previously) - both active users on this forum - being down, while nothing is mentioned when providers like BT don't get a mention when they have outages.

It happens. It's frustrating (I was in the middle of a call today when the line suddenly went dead), but if we don't have any contingency plans in place, we can only blame ourselves.


And this is only the first time I've known of Voipfone to be down, so can't agree it's getting silly. If it happened previously recently, I missed it.


Karl Limpert

 

[danstrutt]

It affected us also. We have been with them for a year and this is the first outage I can remember...

 

[russ74]

Well it's not as if outages are an uncommon thing. Look at Natwest, Lloyd's, Blackberry, O2 etc all have had outages of some description in the last couple of years. Whilst I don't consider VoIP to be quite as resilient as ISDN you still have Comms outages or phone system errors etc no matter what you go for.

 

[SaltireTelecom]

Our VoIP Servers rarely go down as we know how important it is to biz for phone systems to be working.

We support several multi national companies which we have to spread there traffic across 2 data centres.

 

[cjd]

We've had only two total outages in the last seven years both lasting less than 2 hours. For the last 3 years our up time has been 100%

We're still unclear what happened yesterday - we're waiting for a final incident report from the The bunker Data Centre in Kent that had their two fibre failures and a report from our own engineers on why the failover to our London Data Centre didn't operate as it should have.

The fact remains that we recovered from an entire Data Centre failure - this data centre houses hundreds of blue chip companies and all lost connectivity, an almost unheard of failure.

But we're still very unhappy that our enormous investment in redundancy didn't fully work - it's all very disappointing and frustrating but we'll get to the bottom of it.

 

[Feldon]

I wasn't even aware of it been down which is a bit worrying as my potential customers would just phone someone else if they could not get through to me, friends and family just use my mobile number so again i would be none the wiser.

Is it back up and running?

 

[cjd]

Is it back up and running?

Yes, the outage lasted 2 hours.

 

[KM-Tiger]

But we're still very unhappy that our enormous investment in redundancy didn't fully work - it's all very disappointing and frustrating but we'll get to the bottom of it.

I know from experience of regular DR simulations for a client just how difficult that is. It's all a lot easier said than done, and when you have a 24/7/365 service there is no opportunity to do a real test.

I'd say you did well to be back up in 2 hours.

 

[kulture]

I know from experience of regular DR simulations for a client just how difficult that is. It's all a lot easier said than done, and when you have a 24/7/365 service there is no opportunity to do a real test.

I'd say you did well to be back up in 2 hours.

You are exactly right. I remember years ago working for the banks, and their DR practices to cover when their main clearing house data centre went down were a joke. They needed to plan at least a week ahead of the SCHEDULED practice to book vans etc. It never really occurred to them that you do not get to schedule a disaster.

 

[ianm10]

As advised previously, failures do happen and it is the lessons learnt that is important to ensure no reoccurrence (this may be so slightly harder due to a 3rd party service provider failing). Although many people are not aware, similar incidents do regularly occur also to FTSE100 companies, however usually on a smaller scale and are not visible in the public domain.

From reading the regular twitter updates at the time, and the response from Colin, I believe Voipfone acted commendably in this situation (especially the two hour resolution).


Sent from my iPad using Tapatalk

 

[Nuno]

I think we are going to have to move away costing us a fortune in lost business now.

If your VOIP phone lines go down for a couple of hours and this costs you "a fortune" shouldn't you have at least one backup system of your own? Landlines, mobile phones etc.?
If phone contact is that critical should you have all your eggs in one basket, a VOIP system?

 

[russ74]

If your VOIP phone lines go down for a couple of hours and this costs you "a fortune" shouldn't you have at least one backup system of your own? Landlines, mobile phones etc.?
If phone contact is that critical should you have all your eggs in one basket, a VOIP system?

Don't be ridiculous. Most small businesses want to run all their mission critical applications over the cheapest residential broadband possible and refuse to pay for enhanced care and Level 4 care on the PSTN line.

And then scream blue murder if it goes down for 2 minutes

 

[webgeek]

If your telco is mission critical then prudence would dictate that you've put a primary solution in place with a failover to a backup solution, with end to end testing of a failure scenario and a harsh SLA penalty for a total outage.

Most businesses accept a single point of failure, no SLA performence penalty clauses, and more, because they can't afford to pay for that level of service. If you spend more at Starbucks each month than you do on your business telco, then you pay your money and take your chances.

BT Wholesale / OpenReach have their fair share of outages, and the public outcry is off the charts when it happens. To this day we're subject to the occasional crash and burn of a router in Edinburgh which takes out thousands of broadband customers in the north of Glasgow. Why aren't things alt-routed around the problem? Good luck getting an answer to that one!

I must admit that I see voipfone getting a lot of praises from people on the forums and ever so rarely a bad word. Inevitably, at some point stuff will go wrong.

 

[cjd]

Fyi

Incident Report 27th February 2014

At approximately 8:30 am on the 27th February our monitoring system alerted us to an issue with one of the primary links between The Bunker, our data centre in Kent, and our data centre in London, Telehouse East.

Our automated failover kicked in and the traffic was re-routed via our back up link as designed. This link takes traffic via our Goswell Road, London data centre and is our secondary route used for just such occasions.

All of the above had no impact on customers and services were unaffected.

Later that morning our monitoring system detected an issue with some services in our primary data centre at Sovereign House in London due to unusual latency on the secondary inter-data centre link that had become active earlier. Although the issue was not big enough for our automated failover to kick in, in order to provide a high quality service to all our customers, our engineers decided that it would be safer to move some of the main services to our secondary data centre in Kent.

At 10.24 and just before our engineers finished making these changes, our back up link (which is from a third completely independent supplier using separate routing) also went down which resulted in our data centre in Kent going completely off line. At that point some of the main functions of our platform had become completely unavailable as they had been moved to the data centre that was now down. All we could do was rebuild the network so that wherever connectivity returned, we’d be ready for it. Connectivity was re-established just as our engineers were bringing the main functions (registrations and call routing) on line.

This final outage did not only affect us but also the thousands of other companies in the same data centre including many high-street shops, banks and financial institutions. The outage lasted from 10.25am to 12.09pm, 1 hour 44 minutes.

All our data centres and the connectivity to them is ISO 27000 certified and we have been told that the odds of what happened could be as low as a million to one.

We have invested a considerable amount of money in the best data centres in the UK, along with tier one connectivity and multiple backups and for us to lose all connectivity with multiple redundancy in place is hard to believe – and extremely disappointing.

Of course as a company we will learn from this and you can be assured that we are working with our data centres and providers and doing everything we can to ensure that this doesn’t happen again.

 

[garyk]

Agreed you can't have a cheaper service and expect 100% uptime and bulletproof resiliency. As others have said if the phone is critical to your business have a pstn backup.