Fragile Fibre

[fisicx]

I've got a dedicated server with HostingUK (who brought the excellent OpenMind Hosting a while back). It's all been good so far but last night everything went kablooey.

It turns out someone has dug a micro-trench across a field and severed the fibre 2m down! This has knocked out the whole of their network. They are understandably miffed as they pay the fibre provider to provide resilience.

But it does show how fragile a network can be.

 

[johndon68]

Thanks for that as my sites and email are with them as well and now I know what I can’t access anything this morning...

John

 

[Awinner2]

Just shows that no matter how good our suppliers are, someone, somewhere can cock it all up. Hope they get it sorted ASAP for you both

 

[arnydnxluk]

Just shows that no matter how good our suppliers are, someone, somewhere can cock it all up.

Well let's be honest, if their facility was "good" it would have multiple feeds from separate physical locations. Fibre cuts aren't that rare!

 

[fisicx]

The facility is good and the fibre service they pay for should have multiple feeds. But it seems Zayo have been a bit lax in delivering the expected service.

 

[LiveNetworks Ltd]

I know their St Asaph building, they do have multiple diverse fibre routes into it, (and I know how much time and effort they put in to making sure it worked!!!)

I know how the North Wales side of the network operated (with the diversity) but what happened over the border in England, was down to Geo and then Zayo.

 

[Ian J]

I know how the North Wales side of the network operated (with the diversity) but what happened over the border in England, was down to Geo and then Zayo.

That's as maybe but my contract is with Hosting UK and my site was down for over 16 hours

 

[LiveNetworks Ltd]

That's as maybe but my contract is with Hosting UK and my site was down for over 16 hours

What SLA had you arranged with HostingUK? What provision had you made to provide your own resilient backup should your primary service provider fail?

 

[Ian J]

What SLA had you arranged with HostingUK? What provision had you made to provide your own resilient backup should your primary service provider fail?

I'm a factoring broker not an IT techie. When I buy a car I don't buy a second one as a backup

 

[ffox]

I'm a factoring broker not an IT techie. When I buy a car I don't buy a second one as a backup

Very true. But, what happens when your car fails and you have to travel for an important meeting?

Either your garage supplies another car, or you obtain another car, or you postpone the meeting. Which of these depends on the SLA with your garage.

 

[fisicx]

The problem wasn’t a backup server it was a failure of the fibre provider to provide the service they had been paid for.

 

[LiveNetworks Ltd]

The problem wasn’t a backup server it was a failure of the fibre provider to provide the service they had been paid for.

Exactly! I know the guys in that office and I know how much time, effort and money they put into ensuring everything was as reliable as it could be. The fault was 70 miles away from their DC, that's way outside of anything that they have under their control.

Either it's a routing/fibre decision at the top of their parent company, or it's an issue within their fibre provider.

Sadly. s**t happens and either you roll with it, or you make your own provision to mitigate the issues.

I've spent 15 years in the industry dealing with customers who wanted to pay £15 a year for their hosting, expected 5 hours a month of technical support, 50% of the resources available on a server and five nines availability.

 

[ffox]

Exactly! I know the guys in that office and I know how much time, effort and money they put into ensuring everything was as reliable as it could be. The fault was 70 miles away from their DC, that's way outside of anything that they have under their control.

But, did they test the fail over provision?

The facility is good and the fibre service they pay for should have multiple feeds. But it seems Zayo have been a bit lax in delivering the expected service.

If the service being sold featured resilience the fail over should have been tested by the seller of the service.

As Ian said -

That's as maybe but my contract is with Hosting UK and my site was down for over 16 hours

Someone is responsible somewhere up the chain.

Having said that, it may be very expensive to get compensation. When risks are assessed and questions are asked, and the answers tick the boxes, it's sometimes still safest to build in your own work around resilience.

 

[fisicx]

There were apparently two bundles so that if one fell over the other would take up the slack.

But they both went across the same field and both got chopped.

So any resilience testing would have shown the backup worked. What you can’t test for is the installer cutting costs and putting both bundles in the same trench or a farmer cutting a new trench without checking for existing cables.

 

[ffox]

But they both went across the same field and both got chopped.

touché

 

[fisicx]

touché

Like I said, this wasn’t the service they paid for. Unless you are watching them lay the hundreds of miles of fibre you can never be 100% sure they have done what you asked. So it’s not really touché.

 

[LiveNetworks Ltd]

But, did they test the fail over provision?

When risks are assessed and questions are asked, and the answers tick the boxes, it's sometimes still safest to build in your own work around resilience.

Without giving too much away in technical details, you buy a fibre that goes east to one POP and a fibre that goes West to another POP. You know that there's a resilient route from the East Pop that goes 'the long way round'.

You can plug and unplug SFP's all day long and the routes fail over just great.

Sadly it doesn't mean that somewhere beyond the horizon someone hasn't made the decision to dump both bundles into one duct, or even put the bundles in two ducts 3m apart. (Yes, in the fibre world they claim that 3m is sufficient to be classed as diverse and resilient)

The build your own resilience, fine, approach the companies that offer fibre routes across say North Wales, Fibrespeed, Zayo, Cogent, CeltixConnecx, now ask them all where they site their active equipment and where the ducts are?

 

[LiveNetworks Ltd]

Like I said, this wasn’t the service they paid for. Unless you are watching them lay the hundreds of miles of fibre you can never be 100% sure they have done what you asked. So it’s not really touché.

A note of caution, that we don't know what they paid for, what they thought they were buying, or what was delivered!

As I've said, 3m between ducts is accepted as diverse. No one would ever cut two cables 3m apart with a JCB would they! (A trencher in a field however...)

 

[fisicx]

A note of caution, that we don't know what they paid for, what they thought they were buying, or what was delivered!

They said in a tweet that this is what they paid for.

 

[Ian J]

I don't follow any of the techie arguments but what saddens me is that they haven't had the decency to apologize to me

 

[estwig]

If it was still OpenMind, instead of this new crowd, Phil would've been out there in his PJ's, down a trench with a roll of duct tape, if that's what it took!!


In fairness this new crowd have played fair with me and I believe Phil works there now, so sh*t like this isn't his problem anymore, which is fair enough.

 

[Ian J]

In fairness this new crowd have played fair with me .

If your site was down last week have they contacted you and apologized as they certainly haven't to me which is why tomorrow I will be changing hosts to someone who does take customer service seriously

 

[estwig]

Yes my site was down, along with email, very annoying.

In fairness I've been with them and Phil before them for well over 10 years, in all that time any problems I have had have always been of my own making, I do fiddle with things and they have always been great about it. Downtime has been almost zero in ten years.

I can forgive them this, although if they did want to pop round, to cook me a roast dinner as way of an apology, I would want proper Yorkshires!

 

[Ian J]

In fairness I've been with them and Phil before them for well over 10 years, in all that time any problems I have had have always been of my own making, I do fiddle with things and they have always been great about it.

I can forgive them this,

It isn't the downtime that I find unforgiveable but the lack of acknowledgement that it actually happened and subsequent apology.

 

[fisicx]

You have the option to receive status updates and they have made a public apology. Not sure what else you were expecting Ian.

 

[Ian J]

You have the option to receive status updates and they have made a public apology. Not sure what else you were expecting Ian.

I'm not sure what you mean about having the option to receive status updates and they may well have made a public apology but that is only available to people that know where to look and I certainly don't follow them on Twitter and even if I did I only look at that every few weeks.

I have a contract with Hosting UK and if they don't fulfil that contract for whatever reason I expect them to contact me and not to expect me to spot it somewhere online.

I didn't originally decide to go with Hosting UK as after problems with my original host I transferred to a forum member Open Mind Hosting and only ended up with Hosting UK as they bought out Phil's company.

 

[estwig]

That is a bit harsh Ian, I quickly found an article by, I think it was the register, and stuff on twitter explaining the problem and apologising, I don't see what difference a generic email to me and everyone else would make.

Beyond that it's them coming round to cook me that roaster dinner, complete with Yorkshires.

 

[Ian J]

That is a bit harsh Ian, I quickly found an article by, I think it was the register, and stuff on twitter explaining the problem and apologising, I don't see what difference a generic email to me and everyone else would make.

I looked at my stats package and wondered why my website had no visitor that day and it never occurred to me that the site had been down and if I hadn't been told by someone else that the site had been down all day I would never have been any the wiser

 

[estwig]

They couldn't tell anyone at the time, cause their interwebs had been dug up by a digger happy, numpty farmer!

Saying sorry now, in person, maybe a phone call from their Mum if you don't fancy the roast dinner, won't help.

 

[fisicx]

It isn't the downtime that I find unforgiveable but the lack of acknowledgement that it actually happened and subsequent apology.

I've now had an email and a much fuller explanation about what happened. It links to this blog post which is quite an eye opener (if you read the whole thing):

https://hostinguk.net/blog/seriously/

 

[Ian J]

I've now had an email and a much fuller explanation about what happened.

I received that email too and it read as if they assumed that everyone knew about their problems which may not be true

 

[LiveNetworks Ltd]

Just had information that a customer of another service provider on the same network suffered from the cut. What's interesting is that none of the main resellers of the services in North Wales have any information on the outage on their web sites, news pages, twitter feeds etc despite being impacted by the issue.

This map makes an interesting illustration, 'resilient route'
http://www.fibrespeed.co.uk/wp-content/dl/MKT_Network Map2_FLAT_v1.0.pdf

 

[Toby Willows]

I'm not sure what you mean about having the option to receive status updates and they may well have made a public apology but that is only available to people that know where to look and I certainly don't follow them on Twitter and even if I did I only look at that every few weeks.

I have a contract with Hosting UK and if they don't fulfil that contract for whatever reason I expect them to contact me and not to expect me to spot it somewhere online.

I didn't originally decide to go with Hosting UK as after problems with my original host I transferred to a forum member Open Mind Hosting and only ended up with Hosting UK as they bought out Phil's company.

Well unless they guarantee you 100% up time (which they won’t) they haven’t unfulfilled the contract. I bet there’s also clauses in the contract about “acts of god” and “circumstances beyond their control” (which would obviously cover digger happy farmers).

You’re getting your knickers in a twist over a incident a week ago that has been long resolved. Life’s to short to still be fretting over this, move on.