Sage Pay Down

cycloneuk · Sep 7, 2009

Just heads up incase you havn't noticed already, been down since around 8:30pm and i have lost quite a few orders so far.

http://www.sagepay.com/system_monitor.asp

Planck · Sep 7, 2009

cycloneuk said:
Just heads up incase you havn't noticed already, been down since around 8:30pm and i have lost quite a few orders so far.
http://www.sagepay.com/system_monitor.asp

7.51pm we had the first failure.

Jay08 · Sep 7, 2009

Same here, lost few orders. Phil from OMC sent me this link for those wanting to keep up to date.

http://www.sagepay.com/system_monitor.asp

business123 · Sep 7, 2009

I had a sale at 18.40 go thorugh OK, but it's not showing on the admin screen?

Jay08 · Sep 7, 2009

business123 said:
I had a sale at 18.40 go thorugh OK, but it's not showing on the admin screen?

This may have something to do with it.

07 September 2009 22:19:26 - The processing of transactions has now been resumed however, historical transactions and reports will be unavailable for several hours due to ongoing work on the system.

It is therefore not currently possible to refund / void / repeat / authorise or release any historical transactions.

We will ensure that this page is updated regularly to let you know when these functions are available again.

Jay08 · Sep 7, 2009

They say The processing of transactions has now been resumed. But I am trying to buy something from the store and it aint going through. Anyone had any luck?

Planck · Sep 7, 2009

Some payments are going through, some are failing as before.

quikshop · Sep 7, 2009

Got to admit, since it became SagePay the level of service has been a bit inconsistent. We've had several random 'issues' with the integration, all apparently caused by the authorising banks - never had any issues like that when it was Protx.

Lets hope they pick their game up, we have been recommending them because of the historical performance as Protx, would like to keep promoting them as a payment solution.

Blagger · Sep 7, 2009

Looks like it is still down, have done a few test orders on ours but it keeps saying the cards have been declined, probably a time out issue.

Maybe a lesson to get a secondary payment installed such as paypal now

conquer · Sep 7, 2009

yea we found out via customers contacting us with payment problems to start with

. Agree on the customer service since the changeover, I have had much slower responses to my support questions.

Fill-Your-World · Sep 7, 2009

Urgh I am extremely annoyed by this, we had our first failure at 19:06 and are now down £000's worth, of course it would have to have happened at the busiest time! Im surprised the issues have lasted this long, not impressed at all...

Planck · Sep 8, 2009

Fill-Your-Pants said:
we had our first failure at 19:06 and are now down £000's worth, of course it would have to have happened at the busiest time!

Odd, our first failure wasn't until 19.51, with 32 successful transactions between 19.00 and 19.50.

cjd · Sep 8, 2009

Still broken it seems - this is going to be a rough day

Planck · Sep 8, 2009

cjd said:
Still broken it seems - this is going to be a rough day

I know the admin system isn't up, but are you saying you're seeing payments failing? We're not (so far!)

Online Trader · Sep 8, 2009

Admin page is working fine here

cjd · Sep 8, 2009

It seems that new payments are working but repeat payments aren't - for us anyway.

Planck · Sep 8, 2009

cjd said:
It seems that new payments are working but repeat payments aren't - for us anyway.

Ah - that's probably linked to the fact that historical data isn't available, I'm guessing they're rebuilding some databases this morning.

We haven't seen any failures this morning so far, but we don't use repeats.

business123 · Sep 8, 2009

Funds from yesterdays sales don't seem to have been credited to my bank this morning.

PayVector · Sep 8, 2009

We've had several random 'issues' with the integration, all apparently caused by the authorising banks

Strange - If the banks were having issues then we should have had the same. Besides a published Barclays scheduled maintenance last month I have not heard of any bank outage in recent memory.

Iridium is offering to give free service fees equal to the amount of time you have left on your Sagepay contract. So if you have 6 months left on your service we will give you six month free so your not out of pocket if you switch.

We do not have these outage issues at all. However don't believe me. I am sure some of the merchants and service providers that lurk on these forums will vouch for my statement.

PM me or email

sales at iridiumcorp dot net

Planck · Sep 8, 2009

IridiumCorp said:
Iridium is offering to give free service fees equal to the amount of time you have left on your Sagepay contract. So if you have 6 months left on your service we will give you six month free so your not out of pocket if you switch.

Want to know the secret of getting us (and a whole bunch of other people no doubt) to migrate to you? Build an interface that is a drop-in replacement for the SagePay Direct interface. I know this means compromises, but trust me, it will increase your signup rate significantly.

There's nothing I'd like more than to add Iridium as a second gateway, so that we can switch between the two as required. But the integration is a barrier.

Optegris · Sep 8, 2009

The integration is actually simpler than Sagepay as Iridium uses XML to return the response rather than name value pairs. NVM can cause problems as the PaRES for 3D secure often contains the = symbol which can cause scripts to break.

To put this into context it took me less than an hour to add Iridium as a payment gateway to our eCommerce software.

Planck · Sep 8, 2009

openmind said:
The integration is actually simpler than Sagepay as Iridium uses XML to return the response rather than name value pairs. NVM can cause problems as the PaRES for 3D secure often contains the = symbol which can cause scripts to break.

I'm sure it is, but that isn't the point - if it was an identical interface, all I have to do is change the url. With it being xml, I have to pay to have the new interface integrated with our system, go through testing etc.

Optegris · Sep 8, 2009

I see your point but all gateways are different unfortunately. What I'm getting at is the API for Iridium is simpler than the Sagepay method so a seasoned developer would not have a problem working with it...

Planck · Sep 8, 2009

openmind said:
I see your point but all gateways are different unfortunately. What I'm getting at is the API for Iridium is simpler than the Sagepay method so a seasoned developer would not have a problem working with it...

But surely it would not be complicated for Iridium to build a second, http api that is a direct replacement (uses the same fields etc) as SagePay?

Surely you can see that this would get them more signups - big banner "Our gateway is directly compatible with your SagePay code".

Optegris · Sep 8, 2009

Yep I can see that would generate more signups but it's not for me to say if it was possible or not

DuaneJackson · Sep 8, 2009

Planck said:
Want to know the secret of getting us (and a whole bunch of other people no doubt) to migrate to you? Build an interface that is a drop-in replacement for the SagePay Direct interface. I know this means compromises, but trust me, it will increase your signup rate significantly.

That's actually a very very good idea.

PayVector · Sep 8, 2009

If the issue for switching is the cost then I would ask you to consider the following.

During the outage your consumers would have attempted to pay for goods and not been able to. Not only does this cost you hard sales it damages your reputation as the consumers definitely did not have a successful experience.

The question is what is that cost in monetary terms. Is it greater or less than a days worth of development.

We have talked about writing emulators in the past but it would only ever be a band-aid solution. The most common occurrence for outages is not actually when the processor goes down. At least a couple of times a year the major Internet backbone providers have issues which causes whole portion of the internet to not be able to see each other. So you get the case where your website is up and the gateway is up but someone in between the two sites is having a service issue. BT are the most common culprit.

As far as you the consumer of the gateway services care is that your consumers are not able to buy from you. Whenever you rely on a third party service which you do not have a contract with to deliver a service you are exposed. You cannot 100% mitigate the risk but you sure as heck can limit it.

We produce well written and documented code packs for all the common web languages. Within each of these code packs we have written in what we call merchant side fail over detection. If for whatever reason your website cannot get a transaction to the main gateway it will instantly on the fly use one of the backup locations. We even have merchant side load balancing. If we are experiencing a spike we can tell your website where to send it next transaction.

We don't have maintenance downtime either because we have these ultra high availability processes built in and the way we designed the gateway.

It would be unethical to say come to Iridium and you will never have down time ever again. That is not the case. We had one incident this year from a DNS issue. The vast majority of our merchants did not even notice it. It turned out that the ones that did experienced an issue because they had not used our code back which had the fail over aspect built in.

The incident however was a textbook case of third parties potentially effecting your ability to process. The problem was ISPs ignoring the Time To Live (TTL) message in our DNS records which they often do to decrease load. We had intentionally failed over our main site to our redundant ISP as our primary was going to do some maintenance.

If you did not understand any of that basically everybody's systems were up but some could not each other. And if you had done the integration in the recommended way you would not have even noticed.

And that is really why a SagePay wrapper would not work. It may provide a quick and easy band-aid solution but you really need to do it correctly to get the benefit of how our systems operate.

I am going to speak to our lead developer and see if I can get some programmers tasked to do some integration work. If anything we ourselves should be able to do integrations into our systems the quickest.

Will post shortly.

cjd · Sep 8, 2009

Everything breaks at some point; it's frustrating but inevitable.

A second payment gateway would not solve the current problem where historical transactions used for repeat billing are still absent.

We're getting very concerned.

PayVector · Sep 8, 2009

cjd said:
Everything breaks at some point; it's frustrating but inevitable.

A second payment gateway would not solve the current problem where historical transactions used for repeat billing are still absent.

We're getting very concerned.

Wow. I did not pick up on that aspect. Not good at all.

Online Trader · Sep 8, 2009

This does look a bit serious for historical data

cjd · Sep 8, 2009

Historical transaction data

08 September 2009 13:08:27 - We are continuing to process normally however historical transaction data is currently unavailable.

It is not possible to view or perform actions such as Refund, Void, Repeat, RepeatDeferred, Release, Abort, Cancel and Authorise against any archived transaction.

Our engineers are currently restoring historical transaction data and we will provide an update when this has been fully restored.

http://www.sagepay.com/system_monitor.asp

Disaster recovery

[FONT=Arial,Helvetica,sans-serif]Sage Pay operates on twin data centres to ensure optimal system security and up-time and has a full disaster recovery and business continuation policy.[/FONT]

Passionate, agile, approachable, engaging, trustworthy and smart.

Ok, time to walk the talk.

quikshop · Sep 8, 2009

IridiumCorp said:
Strange - If the banks were having issues then we should have had the same. Besides a published Barclays scheduled maintenance last month I have not heard of any bank outage in recent memory.

I used that explanation with caution, I'm sure there is more to it than that. Over the past few months we've had failed orders being sent to our software as complete, complete orders not sent to our software at all and recently we had 3 order confirmation automated messages from SagePay about the same order!!

Their explanation was that the communication between their server and the bank must have timed out 3 times, that didn't sound plausible then and it doesn't now I'm typing it out.

I am hoping that the former Protx staff are now SagePay staff, a loss of expertise built up over a period of years would explain some of the issues and raise significant concerns.

cjd · Sep 8, 2009

It looks like they've lost their historical database and are having to restore it from back-up. Let's hope they have a good system in place. It's not something I ever want to have to do.

Historical transaction data
08 September 2009 15:09:46 - Our engineers are currently restoring historical transaction data and we will provide an update when this has been fully restored.

Joe_SagePay · Sep 8, 2009

Hi folks,

Firstly I am very sorry for the outages experienced last night. We appreciate the impact and disruption this causes and expect there to be a full debrief and investigation of what happened, how it happened and how we can ensure it never happens again. I also know that an apology only goes so far and doesnt make up for a loss of processing.

The processing failure was resolved last night as fast as our engineers could resolve it however since that point there have been problems relating to the availability of historical transactions.

It's too early for me to say, and if I'm totally honest I'm not technically minded enough to give you specific reasons for this failure at the moment but I expect to be given this information after the historical transactions are restored. I appreciate that being informed of the reasons for failure is no substitute for being able to process uninterrupted but all the same, if you want more details then I'll do what I can to get them to you.

As an update for you right now, we are currently working to address the restoration of historical transactions so our customers will then be able to perform the repeats, refunds and releases that they need to. This is obviously our key focus right now and this work continues and our engineers will work as long as it takes to restore full functionality.

Once again there is little else I can say at this time apart from offer an apology for the failure to process transactions during the period of outage and the disruption and inconvenience caused by not being able to perform key functions on historical transactions for the period since the outage.

I will be available on the forum if I am needed on an account by account basis and to give updates. I would ask that you also refer to the monitor page which I will ensure is updated.

http://www.sagepay.com/system_monitor.asp

Sincere regards,

Joe

PayVector · Sep 8, 2009

Joe,

Although I feel for your merchants more I do feel for you guys. Best of luck in getting your systems fully restored. I would imagine your techies need some sleep. I know our sales and connections teams have had a real long day today over this issue. I am going to be at work for some time yet working through all this.

cjd · Sep 8, 2009

This is now a 24 outage and if we go past midnight we'll have a real problem re-synchronising billing dates over two lost billing cycles.

I am also worried about data loss now as it seems clear that SagePay has trashed its live one and is having to reconstruct from back-ups.

Joe, thanks for coming in here and I do feel sorry for you and yours - there but for the grace of god etc but all you've essentially said is 'sorry we hope to have it fixed soon and we'll try not to do it again' but in a lot more words.

What I actually need is some idea what the problem is and when it will be fixed so that I can have my guys briefed and available for when we have to start our own clean up process.

Christiane · Sep 8, 2009

Just had an order come through as paid on my website but it's not showing on Sagepay.

I don't recall the last outage to take so long to fix.

Planck · Sep 8, 2009

just.b said:
I don't recall the last outage to take so long to fix.

Hmm, let's see. We had a really bad one in Nov 2007. 2008 wasn't too bad, with only a few minor (2-3 hour) outages.

When our web server falls over (never has, but one day it will), we will switch over to our backup. We have one. It's synchronised with the main server. We're the little guys. Why can't the big guys (SagePay) manage this?

business123 · Sep 8, 2009

just.b said:
Just had an order come through as paid on my website but it's not showing on Sagepay.

I don't recall the last outage to take so long to fix.

Had an order come through just before the problems started yesterday, not showing on the Admin screen and wasn't credited to the bank today?

Did anyone receive funds into their bank accounts today from Sage Pay?

PayVector · Sep 8, 2009

Peps,

Still at work dealing with this. The demand from merchants moving has been staggering. I have set up a quickfire procedure with all but one UK acquirer to get people hit by this sorted ASAP. We already have a whack of MOTO only merchants live. First ECOM/MOTO merchants before the morning.

I also quite funnily enough been contacted by senior peps in all the acquiring banks <--- but one. Offering to give crazy low rates to anyone that wants it. To give you an example SagePay on their site offer 2.5% credit and 40p debit.

I am authorised to offer 1.8% domestic credit and 30p debit for every merchant regardless of your volume. Anyone above 7 figures turnover pcy I can go a whole heck of a lot lower.

Cookie Consent

Essential

Analytics

Marketing

Sage Pay Down

cycloneuk

Jay08

business123

Jay08

Jay08

Online Trader

business123

PayVector

PayVector

PayVector

Online Trader

PayVector

Christiane

business123

PayVector