SagePay Down - Again

[cjd]

SagePay repeat payments has been broken since 4pm yesterday.

I'm usually fairly forgiving about technical glitches but this is happening too often and it seems they don't work Sundays. We're having to deal with hundreds of p1ssed off customers here.

29 March 2010 08:32:02 - Please be advised this issue is on going. We will provide updates as soon as these are available.


28 March 2010 22:19:08 - Please be advised this issue on going.

Customers will be unable to view or perform actions such as Refund, Void, Repeat, RepeatDeferred, Release, Abort, Cancel and Authorise against archived transactions until further notice.

Thank you for your patience in this matter, we will provide updates as soon as these are available.


28 March 2010 15:40:09 - Please be advised there is currently an issue accessing archived transaction data.

Customers will be unable to view or perform actions such as Refund, Void, Repeat, RepeatDeferred, Release, Abort, Cancel and Authorise against archived transactions until further notice.

Thank you for your patience in this matter, we will provide updates as soon as these are available.

http://www.sagepay.com/system_monitor.asp

 

[sysops]

On the plus side, the live transaction processing was uninterrupted - they managed to switch over from their main servers to the backup without any noticeable downtime (certainly no more than a few minutes). I'm quite impressed!

 

[cjd]

On the plus side, the live transaction processing was uninterrupted - they managed to switch over from their main servers to the backup without any noticeable downtime (certainly no more than a few minutes). I'm quite impressed!

That's how any service dealing with payment transactions should work. They don't get any points from me for being able to do that - it's like congratulating a builder because he put a roof on the house anticipating that it might rain some day.

Very annoyed now - 'sometime today' is the latest time to fix. Seems likely to be a 24 hour outage.

We wanted to spend the day trying to resolve a flood of Tesco users problems (after Tesco announced that they're shutting their VoIP service last week), now we have to deal with this instead.

My mum said there'd be days like this.

 

If payment processing is so critical to your business, shouldn't you have a backup payment processor?

 

[cjd]

Beginning to think so but it may be impossible without keeping credit card data on our system which I do not wish to do.

It's also not service affecting as our systems work without payment for 37 days, but we have to deal with hundreds of confused customers who have received payment failure notices.

 

[Amy_Sagepay]

Hi All,

As you are aware we have encountered some issues with the system. ALL transactions are processing as normal as we were able to switch to our backup server within minutes. However, historical data is not yet available. We realise how inconvenient this is for you and our Technical team is working on this as we speak. As soon as the historical data becomes available we will let you know.

Thank you for bearing with us during this time.

Amy

 

[cjd]

Hi All,
ALL transactions are processing as normal as we were able to switch to our backup server within minutes.
Amy

All new transactions you mean. Any repeat billed - which is 95% of our transactions with you - have been broken for 20 hours now.

Can you please request - demand - an equivalent back-up server for historic data.

 

[Searcher]

It's also not service affecting as our systems work without payment for 37 days, but we have to deal with hundreds of confused customers who have received payment failure notices.

Out of interest, if your customers get payment failure notices are they informed why the payment failed ? Or could they assume that it's because you have lack of funds, which could be quite a serious issue.

 

[cjd]

Out of interest, if your customers get payment failure notices are they informed why the payment failed ? Or could they assume that it's because you have lack of funds, which could be quite a serious issue.

They get a pleasantly worded generic payment failure message because it could be for many reasons and we say that it's usually because of an expired card (even though we email prior warnings of card expiry).

 

[sysops]

That's how any service dealing with payment transactions should work. They don't get any points from me for being able to do that - it's like congratulating a builder because he put a roof on the house anticipating that it might rain some day.

No, it's like congratulating the builder because you know that the last 20 houses he built had no roofs ;-)

 

[Amy_Sagepay]

Hi CJD,

Thanks for your reponse. Yes, repeat billing is part of this historical data as the original transaction that you are repeating was put through previously. Unfortunatley this historical data is still unavailable but we will keep you informed when we know more. In regards to your second query, we do have a back up server however as all transactions were being put through our backup server during the issue, it takes some time to transfer them all back over to our primary server which is why data is unavailable. The refactoring project we are working on at the moment will mean that all our backup servers will be sent a copy of the information when transactions are being processed so that if this happens in the future, historical data will be available.

Again thank you for bearing with us,

Amy

 

[sysops]

Hi Amy,

Since you are in a particularly informative mood today, would you be so kind as to give us a clue as to why SagePay's systems are so vulnerable? What actually failed? Was it a hardware failure? We never get to find out after the outages.

 

[Amy_Sagepay]

Hi,

Our Technical team are currently working on getting all archived transactions back onto the system for you so once everythings back in place we will have a full debrief. We will then provide our customers with a full account.

Amy

 

[sysops]

Our Technical team are currently working on getting all archived transactions back onto the system for you so once everythings back in place we will have a full debrief. We will then provide our customers with a full account.

Never happened in the past, despite many similar promises.

 

[cjd]

So now we have a full 24 hours.

If this goes to midnight a few thousand more failed payment emails go out (unless I can stop them) and the phones get clogged up again.

Really crap day here at Voipfone Towers.

 

[Amy_Sagepay]

Hi All,

Thanks for your patience. I understand your frustration and annoyance at the lack of information being supplied by us. Unfortunately, we were unable to provide a rough ETA earlier due to the volume of archived transactions affected and the issue experienced.

As transactions were being processed through our backup server when we first identified the issue, it is taking some time to transfer them all back to our primary server, which is why this issue is affecting all historical transactions.

Our operations team have been working continuously to ensure transactions are fully restored as quickly as possible and most historical transaction data is now available. We are now verifying some remaining data to ensure all information is complete and expect everything to be fully operational by 8pm this evening.

I am sincerely sorry for the inconvenience this has caused you today and I hope I can re-assure you that we have been doing everything we resolve this.

Amy

 

[cjd]

Fair enough and thanks for the update.

Can you please ask your management what they plan to do to prevent a third occurrence; I know everyone will be doing their best now and under a lot of pressure, but I need some reassurance that SagePay is able to resolve these problems permanently.

I don't need to know now and the solution doesn't need to be next week, but I'd like to know that you have a credible plan for recovering from this situation in future.

Some of my customers are being less reasonable with me and I need to be able to reassure them as well as myself or find another solution.

If you don't feel able to discuss it here, please PM me.

 

[Amy_Sagepay]

Hi,

Simon Black, Sage Pay's managing director will be sending out a communication regarding these issues to all customers and partners first thing in the morning which should cover off all of the points you have raised. However, if you do have any further queries, please do not hesitate to contact me directly or respond to Simon's email

Amy

 

[cjd]

We are now verifying some remaining data to ensure all information is complete and expect everything to be fully operational by 8pm this evening.

10pm, still down and no updates for over 3 hours.

Anyone home?

 

[sysops]

Our Technical team are currently working on getting all archived transactions back onto the system for you so once everythings back in place we will have a full debrief. We will then provide our customers with a full account.

Amy

It's funny how that bit never actually happens.

 

[sysops]

Simon Black, Sage Pay's managing director will be sending out a communication regarding these issues to all customers and partners first thing in the morning which should cover off all of the points you have raised. However, if you do have any further queries, please do not hesitate to contact me directly or respond to Simon's email

Amy

Which morning Amy, which morning?

 

[cjd]

I can confirm that repeat billing was fixed before midnight, thankfully, but the status monitor wasn't updated until the following morning.

We've had no email from Simon yet either.

 

[SFD]

Which morning Amy, which morning?

I got one yesterday:

Hello

As you may be aware we’ve experienced some issues with the Sage Pay system in the past 24 hours. These are now resolved, but it’s a huge disappointment for me to be writing this, when in my last email in January, I promised that you would see improvements to every aspect of the Sage Pay offering this year.


This commitment still stands. We are just weeks before the first phase of our system upgrade is due to be implemented and in the coming months, you will benefit from a world-class system with a far higher resilience.

We’re working extremely hard to bring many improvements to you, especially in processing and reporting, and will do this as quickly as possible. I’m sure you are finding these issues frustrating, so please accept my sincerest apologies. If you’d like to discuss anything related to this directly with me, don’t hesitate to email [email protected]

Thank you for bearing with us — we really do appreciate it.

Kind regards

Simon Black
MD

 

[sysops]

I can confirm that repeat billing was fixed before midnight, thankfully, but the status monitor wasn't updated until the following morning.

We've had no email from Simon yet either.

We always get promises of a full update after the fix, but this rarely happens. I guess it's difficult writing an email which goes something like "we're sorry our service keeps going down so much and for such extended periods of time, it's because it's poorly designed, and the servers are held together with bits of duct tape and string".

 

[cjd]

That uses a lot of words that say nothing.

 

[SFD]

That uses a lot of words that say nothing.

Send him an email with a link to the thread, he may post.

 

[Clinton]

That uses a lot of words that say nothing.

The bigger the company the better they are with platitudes.

 

[sysops]

And we're down again.

 

[NuBlue]

Only with certain banks. We're with Barclays and everything is processing fine.

 

[cjd]

For once we got lucky too (so far)

 

[NuBlue]

Just had this from Sage Pay, thought I'd post in case any customers have not received it. Looks like this might affect quite a few people:

Hello,

We'd like to update you on an external issue which is effecting nationwide telecomms/datacomms.

British Telecom has confirmed a fire at their Exchange in the Paddington area of London. This has impacted a huge number of UK businesses, including Sage Pay and the major banks that route transactions through the Exchange. Sage Pay has taken action on your behalf by quickly moving transaction processing to our backup data centre. However, the issue has affected connectivity to some banks who rely on the exchange and we are currently unable to process via those banks at this time.

Please note, as banks and payment providers look for alternative transaction routing solutions away from the Paddington Exchange, this could create some general capacity issues, resulting in transaction 'time-outs'.

We'll be regularly updating our system monitor page, however if you would prefer to speak to one of our advisors, we remain open 24/7 throughout the Easter period.

Kind regards,

Colin Mould
Head of Customer Services

Bad week at the office for Sagepay!

 

[stugster]

That fire is a biggie, and it's not just Sagepay affected by it.

 

[SFD]

http://www.theregister.co.uk/2010/03/31/burne_house_burns/

More Info on the fire

 

[cjd]

spoke too f****** soon

31 March 2010 14:30:08 - We have received reports that this may be affecting other acquirers and customers.

We are closely monitoring the transaction flows and will provide updates as soon as these are available.

 

[cycloneuk]

All payments are failing here now, think i'll pack the rest of the orders and call it a day.

 

[SFD]

Sagepay back processing for us now.

I doesn't seem right that sagepay updated twitter 25 minutes ago to say it was working yet their website status monitor still hasn't been updated.

I've been checking the status monitor closely waiting to put a pretty important transaction through.

 

[sysops]

They must like you, because we're still down.

 

[SFD]

31 March 2010 15:34:55 - We have received an update from the respective third parties who have provided us with temporary communication routes.

All acquirers are now processing, however intermittent authorisations are still frequent which we believe is capacity related.

We are closely monitoring the transaction flows and will provide updates as necessary.

Just updated their site

 

[RBS]

"While fire was the problem earlier down in Paddington, flooding is the issue now apparently."

 

[Mustaka]

Just found these :

http://www.finextra.com/news/fullstory.aspx?newsitemid=21244

&

http://sagepayoutages.blogspot.com/

I dont think the most recent outage is listed there. I wonder what the total time will be as I notice the system monitor is showing red so maybe they are waiting for the the entire issue to be resolved. Still 40 hours (ish) downtime in a year is terrible. In one week... well that is just crazy.