Sage Pay Down

I had a sale at 18.40 go thorugh OK, but it's not showing on the admin screen?

This may have something to do with it.

07 September 2009 22:19:26 - The processing of transactions has now been resumed however, historical transactions and reports will be unavailable for several hours due to ongoing work on the system.

It is therefore not currently possible to refund / void / repeat / authorise or release any historical transactions.

We will ensure that this page is updated regularly to let you know when these functions are available again.
 
Upvote 0

quikshop

Free Member
Oct 11, 2006
3,644
714
54
Wolves
Got to admit, since it became SagePay the level of service has been a bit inconsistent. We've had several random 'issues' with the integration, all apparently caused by the authorising banks - never had any issues like that when it was Protx.

Lets hope they pick their game up, we have been recommending them because of the historical performance as Protx, would like to keep promoting them as a payment solution.
 
Upvote 0

cjd

Business Member
  • Nov 23, 2005
    15,996
    3,432
    www.voipfone.co.uk
    It seems that new payments are working but repeat payments aren't - for us anyway.
     
    Upvote 0
    We've had several random 'issues' with the integration, all apparently caused by the authorising banks

    Strange - If the banks were having issues then we should have had the same. Besides a published Barclays scheduled maintenance last month I have not heard of any bank outage in recent memory.

    Iridium is offering to give free service fees equal to the amount of time you have left on your Sagepay contract. So if you have 6 months left on your service we will give you six month free so your not out of pocket if you switch.

    We do not have these outage issues at all. However don't believe me. I am sure some of the merchants and service providers that lurk on these forums will vouch for my statement.

    PM me or email

    sales at iridiumcorp dot net
     
    Upvote 0

    Planck

    Free Member
    Jul 1, 2009
    164
    27
    Iridium is offering to give free service fees equal to the amount of time you have left on your Sagepay contract. So if you have 6 months left on your service we will give you six month free so your not out of pocket if you switch.

    Want to know the secret of getting us (and a whole bunch of other people no doubt) to migrate to you? Build an interface that is a drop-in replacement for the SagePay Direct interface. I know this means compromises, but trust me, it will increase your signup rate significantly.

    There's nothing I'd like more than to add Iridium as a second gateway, so that we can switch between the two as required. But the integration is a barrier.
     
    Upvote 0

    Optegris

    Free Member
  • Business Listing
    The integration is actually simpler than Sagepay as Iridium uses XML to return the response rather than name value pairs. NVM can cause problems as the PaRES for 3D secure often contains the = symbol which can cause scripts to break.


    To put this into context it took me less than an hour to add Iridium as a payment gateway to our eCommerce software.
     
    Upvote 0

    Planck

    Free Member
    Jul 1, 2009
    164
    27
    The integration is actually simpler than Sagepay as Iridium uses XML to return the response rather than name value pairs. NVM can cause problems as the PaRES for 3D secure often contains the = symbol which can cause scripts to break.

    I'm sure it is, but that isn't the point - if it was an identical interface, all I have to do is change the url. With it being xml, I have to pay to have the new interface integrated with our system, go through testing etc.
     
    Upvote 0

    Planck

    Free Member
    Jul 1, 2009
    164
    27
    I see your point but all gateways are different unfortunately. What I'm getting at is the API for Iridium is simpler than the Sagepay method so a seasoned developer would not have a problem working with it...

    But surely it would not be complicated for Iridium to build a second, http api that is a direct replacement (uses the same fields etc) as SagePay?

    Surely you can see that this would get them more signups - big banner "Our gateway is directly compatible with your SagePay code".
     
    Upvote 0

    DuaneJackson

    Free Member
    Jul 14, 2005
    8,642
    1,100
    Brighton / London
    Want to know the secret of getting us (and a whole bunch of other people no doubt) to migrate to you? Build an interface that is a drop-in replacement for the SagePay Direct interface. I know this means compromises, but trust me, it will increase your signup rate significantly.

    That's actually a very very good idea.
     
    Upvote 0
    If the issue for switching is the cost then I would ask you to consider the following.

    During the outage your consumers would have attempted to pay for goods and not been able to. Not only does this cost you hard sales it damages your reputation as the consumers definitely did not have a successful experience.

    The question is what is that cost in monetary terms. Is it greater or less than a days worth of development.

    We have talked about writing emulators in the past but it would only ever be a band-aid solution. The most common occurrence for outages is not actually when the processor goes down. At least a couple of times a year the major Internet backbone providers have issues which causes whole portion of the internet to not be able to see each other. So you get the case where your website is up and the gateway is up but someone in between the two sites is having a service issue. BT are the most common culprit.

    As far as you the consumer of the gateway services care is that your consumers are not able to buy from you. Whenever you rely on a third party service which you do not have a contract with to deliver a service you are exposed. You cannot 100% mitigate the risk but you sure as heck can limit it.

    We produce well written and documented code packs for all the common web languages. Within each of these code packs we have written in what we call merchant side fail over detection. If for whatever reason your website cannot get a transaction to the main gateway it will instantly on the fly use one of the backup locations. We even have merchant side load balancing. If we are experiencing a spike we can tell your website where to send it next transaction.

    We don't have maintenance downtime either because we have these ultra high availability processes built in and the way we designed the gateway.

    It would be unethical to say come to Iridium and you will never have down time ever again. That is not the case. We had one incident this year from a DNS issue. The vast majority of our merchants did not even notice it. It turned out that the ones that did experienced an issue because they had not used our code back which had the fail over aspect built in.

    The incident however was a textbook case of third parties potentially effecting your ability to process. The problem was ISPs ignoring the Time To Live (TTL) message in our DNS records which they often do to decrease load. We had intentionally failed over our main site to our redundant ISP as our primary was going to do some maintenance.

    If you did not understand any of that basically everybody's systems were up but some could not each other. And if you had done the integration in the recommended way you would not have even noticed.

    And that is really why a SagePay wrapper would not work. It may provide a quick and easy band-aid solution but you really need to do it correctly to get the benefit of how our systems operate.

    I am going to speak to our lead developer and see if I can get some programmers tasked to do some integration work. If anything we ourselves should be able to do integrations into our systems the quickest.

    Will post shortly.
     
    Upvote 0

    cjd

    Business Member
  • Nov 23, 2005
    15,996
    3,432
    www.voipfone.co.uk
    Everything breaks at some point; it's frustrating but inevitable.

    A second payment gateway would not solve the current problem where historical transactions used for repeat billing are still absent.

    We're getting very concerned.
     
    Upvote 0

    cjd

    Business Member
  • Nov 23, 2005
    15,996
    3,432
    www.voipfone.co.uk
    Historical transaction data

    08 September 2009 13:08:27 - We are continuing to process normally however historical transaction data is currently unavailable.

    It is not possible to view or perform actions such as Refund, Void, Repeat, RepeatDeferred, Release, Abort, Cancel and Authorise against any archived transaction.

    Our engineers are currently restoring historical transaction data and we will provide an update when this has been fully restored.

    http://www.sagepay.com/system_monitor.asp

    Disaster recovery
    [FONT=Arial,Helvetica,sans-serif]Sage Pay operates on twin data centres to ensure optimal system security and up-time and has a full disaster recovery and business continuation policy.[/FONT]
    Passionate, agile, approachable, engaging, trustworthy and smart.



    Ok, time to walk the talk.
     
    Last edited:
    Upvote 0

    quikshop

    Free Member
    Oct 11, 2006
    3,644
    714
    54
    Wolves
    Strange - If the banks were having issues then we should have had the same. Besides a published Barclays scheduled maintenance last month I have not heard of any bank outage in recent memory.

    I used that explanation with caution, I'm sure there is more to it than that. Over the past few months we've had failed orders being sent to our software as complete, complete orders not sent to our software at all and recently we had 3 order confirmation automated messages from SagePay about the same order!!

    Their explanation was that the communication between their server and the bank must have timed out 3 times, that didn't sound plausible then and it doesn't now I'm typing it out.

    I am hoping that the former Protx staff are now SagePay staff, a loss of expertise built up over a period of years would explain some of the issues and raise significant concerns.
     
    Last edited:
    Upvote 0

    cjd

    Business Member
  • Nov 23, 2005
    15,996
    3,432
    www.voipfone.co.uk
    It looks like they've lost their historical database and are having to restore it from back-up. Let's hope they have a good system in place. It's not something I ever want to have to do.

    Historical transaction data
    08 September 2009 15:09:46 - Our engineers are currently restoring historical transaction data and we will provide an update when this has been fully restored.



     
    Upvote 0

    Joe_SagePay

    Free Member
    Apr 23, 2009
    29
    6
    Hi folks,

    Firstly I am very sorry for the outages experienced last night. We appreciate the impact and disruption this causes and expect there to be a full debrief and investigation of what happened, how it happened and how we can ensure it never happens again. I also know that an apology only goes so far and doesn’t make up for a loss of processing.

    The processing failure was resolved last night as fast as our engineers could resolve it however since that point there have been problems relating to the availability of historical transactions.

    It's too early for me to say, and if I'm totally honest I'm not technically minded enough to give you specific reasons for this failure at the moment but I expect to be given this information after the historical transactions are restored. I appreciate that being informed of the reasons for failure is no substitute for being able to process uninterrupted but all the same, if you want more details then I'll do what I can to get them to you.

    As an update for you right now, we are currently working to address the restoration of historical transactions so our customers will then be able to perform the repeats, refunds and releases that they need to. This is obviously our key focus right now and this work continues and our engineers will work as long as it takes to restore full functionality.

    Once again there is little else I can say at this time apart from offer an apology for the failure to process transactions during the period of outage and the disruption and inconvenience caused by not being able to perform key functions on historical transactions for the period since the outage.

    I will be available on the forum if I am needed on an account by account basis and to give updates. I would ask that you also refer to the monitor page which I will ensure is updated.

    http://www.sagepay.com/system_monitor.asp

    Sincere regards,

    Joe
     
    Upvote 0
    Joe,

    Although I feel for your merchants more I do feel for you guys. Best of luck in getting your systems fully restored. I would imagine your techies need some sleep. I know our sales and connections teams have had a real long day today over this issue. I am going to be at work for some time yet working through all this.
     
    Upvote 0

    cjd

    Business Member
  • Nov 23, 2005
    15,996
    3,432
    www.voipfone.co.uk
    This is now a 24 outage and if we go past midnight we'll have a real problem re-synchronising billing dates over two lost billing cycles.

    I am also worried about data loss now as it seems clear that SagePay has trashed its live one and is having to reconstruct from back-ups.

    Joe, thanks for coming in here and I do feel sorry for you and yours - there but for the grace of god etc but all you've essentially said is 'sorry we hope to have it fixed soon and we'll try not to do it again' but in a lot more words.

    What I actually need is some idea what the problem is and when it will be fixed so that I can have my guys briefed and available for when we have to start our own clean up process.
     
    Upvote 0

    Planck

    Free Member
    Jul 1, 2009
    164
    27
    I don't recall the last outage to take so long to fix.

    Hmm, let's see. We had a really bad one in Nov 2007. 2008 wasn't too bad, with only a few minor (2-3 hour) outages.

    When our web server falls over (never has, but one day it will), we will switch over to our backup. We have one. It's synchronised with the main server. We're the little guys. Why can't the big guys (SagePay) manage this?
     
    Upvote 0
    B

    business123

    Just had an order come through as paid on my website but it's not showing on Sagepay. :mad:

    I don't recall the last outage to take so long to fix.
    Had an order come through just before the problems started yesterday, not showing on the Admin screen and wasn't credited to the bank today?

    Did anyone receive funds into their bank accounts today from Sage Pay?
     
    Upvote 0
    Peps,

    Still at work dealing with this. The demand from merchants moving has been staggering. I have set up a quickfire procedure with all but one UK acquirer to get people hit by this sorted ASAP. We already have a whack of MOTO only merchants live. First ECOM/MOTO merchants before the morning.

    I also quite funnily enough been contacted by senior peps in all the acquiring banks <--- but one. Offering to give crazy low rates to anyone that wants it. To give you an example SagePay on their site offer 2.5% credit and 40p debit.

    I am authorised to offer 1.8% domestic credit and 30p debit for every merchant regardless of your volume. Anyone above 7 figures turnover pcy I can go a whole heck of a lot lower.
     
    Upvote 0

    Latest Articles