HSBS Secure Epayments failure

  • Thread starter Deleted member 21013
  • Start date
D

Deleted member 21013

Hi
On Saturday 05/01/08 at 9am the HSBC secure epayments system crashed and wasn't restored until the following day at 1 am. This is obviously a major incident for HSBC but an email I have received informs me that all the transactions we took on the Friday 4th have been lost and should be manually re-imput. Short of contactcting all the customers who placed orders on the Friday and asking for their card details again I am not sure how to do this. In addition our website was unable to accept orders on Saturday which is usually a busy day for us causing us to lose a days trading. Has anyone else on the forum been affected by this?

Regards

Howard
 
Yep, me too. Was terrible customer service, I wasn't told about the problem and the only thing they've done is put a little note on the website after you log in saying 'sorry for our recent problems'. I also have to contact everyone manually - sounds great doesn't it, sorry we've lost all your card details, can we have them again? I have to do it though, as the orders have already been despatched. Pah.
 
  • Like
Reactions: HowardJB
Upvote 0
The HSBC platform is the old Clear Commerce software, a company that is no longer in existence. So your problem as a merchant of them is that when something goes wrong there is no one around with any great knowledge on how to fix it.

Not a great place to be for HSBC. Unfortunately its you the Merchants who suffer.
 
Upvote 0
D

Deleted member 21013

Hi Pelenna
Further to my first posting I have been contacted by HSBC who have been able to re-instate all of my transactions for the missing day except a couple which had been referred to ‘Fraud Review’ but they say I will not be able to view any of them on their site. I am waiting to hear from them to see if these transactions (The biggest of the day of course) can be recovered as well.
As is life wasn’t busy enough!
Howard
 
  • Like
Reactions: Pelenna
Upvote 0
Both myself and my father in law use HSBC CCP and my hubby (who works for his dad) noticed on Saturday that orders were being rejected.

At 10am they told us that it would be fixed in three hours but orders were still being rejected on Sunday morning.

I only know that Friday's transactions are missing because my father in law's business partner noticed on Monday by accident.

Monday was spent ringing customers to ask for their card details again ... not the most fun I've ever had "Thank you for spending £1000 on furniture ... we've lost your payment". Inspires a lot of customer confidence I'm sure. I didn't manage to catch everyone until 8pm last night (Tuesday) hampered by conveniently getting the cold that's going round so my voice now sounds like an axe murderer making the explanation sound even less believable.

I'm concerned that HowardJB's transactions have been reinstated and he can't see them on his CCP gateway as I don't fancy having to face customers to explain their payment has now been taken twice!

Great time for it to happen ... January sale time ... it's bad enough for me but my father in law's company lost 70+ orders on Saturday and hubby had a similar number to phone on Monday to retake payments.

The thing that really annoys me is the lack of communication. There's an email address on the gateway page but the email I sent was returned with a stock reply that didn't answer my question at all ... brill!
 
Upvote 0
The thing that really annoys me is the lack of communication.

Completely - I only had 3 orders disappear (for some reason most people were using Paypal that day) and it's easy to miss something like that. They didn't let anyone know - a simple email sent out to all customers explaining the situation wouldn't have been that hard would it?
 
Upvote 0

Dwebs-Ltd

Free Member
Nov 29, 2007
2,019
264
Blackpool
Well Protx had a major balls up last year, so nothings bullet proof so is it not wise to use two solutions i.e. hsbc epayments and something like protx then if one fails you can switch to the other.

Our payment system is built in such a way that we can disable payment methods instantly and can plug any payment gateway into it :)

I know shopping carts like cubecart, zencart offer multiple payment gateway support.

It's worth while having multiple options if you would loose a lot of business if it wasn't working.
 
Upvote 0
Good point, but the thing is that we weren't told the system wasn't working - I do have Paypal also on my website, had I known I could have just put up a temporary notice saying 'please use Paypal' or whatever. Also, it's the payments that were made using HSBC when it was working that have disappeared - another gateway wouldn't have solved that!
 
Upvote 0

Dwebs-Ltd

Free Member
Nov 29, 2007
2,019
264
Blackpool
Good point, but the thing is that we weren't told the system wasn't working - I do have Paypal also on my website, had I known I could have just put up a temporary notice saying 'please use Paypal' or whatever. Also, it's the payments that were made using HSBC when it was working that have disappeared - another gateway wouldn't have solved that!

Indeed it wouldn't have solved the vanishing transactions but if your clients were getting bounces which i believe they were i.e. the systems stuffed you could then switch to the second gateway.

Again nothing is bullet proof :(

We use Verisign’s system and touch wood we haven't had a failure yet but its always possible. Currently we don't push enough credit card transactions to warrant a second gateway but if the need was there we could have one up and running very quickly :)
 
Upvote 0
Yep friend of mine had their busiest day of sales on Saturday and then bump go the orders as no one could put their card details through, they managed to rescue their regular customers but still lost orders from the new customers who were trying. Interesting that the issue is one of an old unsupported platform, I'll have to let them know that!
 
Upvote 0
This whole HSBC failure is a nightmare, and not at all helped by the frighteningly poor customer service from HSBC.

I spoke to them a short while ago today, Wednesday 9th, and they were able to tell me the number of transactions that have successfully gone through on Friday, and the monetary total of these transactions. If you contact them and have your Client ID handy, they should be able to give you this info.

One thing I was told, which I think it's best to mention, and I don't think anyone previously has mentioned in this thread:
If you're transactions were put through as status 'pre-auth' then all these transactions are now lost. If you try and capture them now you won't be able to.

Also, a question regarding a further issue I've just encountered: how can I refund a transaction that was successfully captured on Friday? Normally I would process it through our API which would give a full credit against the original transaction [that way we don't need the card details - just the transaction ID].
I can't do it through the API [it just fails], and I can't do it manually through secure e-payments as you can't view any of the transactions!

It's a nightmare! Without the original card details we can't give the man his money back. And of course [like everyone else, right?] we don't store CC details as we are PCI compliant.

It will be very interesting to see what the result of the HSBC "Major Incident Review" are!!
 
Upvote 0
Is it just me or is it still not working one hundred percent? I've had a couple of customers say it crashes when they try to return to the website after making payment, and it just crashed on me while trying to ship orders on the virtual terminal... could be user error though I suppose, as it seems to working fine now most of the time :|
 
Upvote 0
D

Deleted member 21013

Is it just me or is it still not working one hundred percent? I've had a couple of customers say it crashes when they try to return to the website after making payment, and it just crashed on me while trying to ship orders on the virtual terminal... could be user error though I suppose, as it seems to working fine now most of the time :|
I haven't been at work today but orders seem to have been coming in at a normal rate. Do you get many transactions that do not return the customer to your site, i.e. you get a payment but no confirmed order?

Howard
 
Upvote 0
I've only had 2 in the past 2 days that have done this, both rang me to make sure their orders had gone through and said the system crashed when they tried to return to my website - however there have been quite a few other orders that have gone through fine, so it's probably not the HSBC system. It's probably just I'm a bit over-cautious now after this whole debacle!
 
Upvote 0
Just to let you know that I have had conversations with some very high up (so they tell me) people at HSBC over the last week regarding the repeated failures of EPayments.

To quote one of them "Our service has been well below standard. I apologise".

He then went on to treat me like a naughty child (he'd listened to a recording of one of my calls to the so-called help desk - in India of course - during which I lost it with them), at which point I decided enough was enough.

So now, folks, I am trialling Protx. So far so good. It's easy to sign up - especially as all HSBC EPayments users have a merchant number which they just transfer across - and feedback from customers is excellent. It took 48 hours from web sign up to going live.

We shall see how it goes. It will cost more than HSBC, marginally, but the freedom from hassle more than makes up for it.

I still have World Pay as a stand-by just in case. I have no plans to retain HSBC.
 
  • Like
Reactions: Sue F
Upvote 0
Just to let you know that I have had conversations with some very high up (so they tell me) people at HSBC over the last week regarding the repeated failures of EPayments.

To quote one of them "Our service has been well below standard. I apologise".

...

So now, folks, I am trialling Protx. So far so good. It's easy to sign up - especially as all HSBC EPayments users have a merchant number which they just transfer across - and feedback from customers is excellent. It took 48 hours from web sign up to going live.

Thanks for posting this.

I'm getting increasingly frustrated with the HSBC epayments system. Since the massive failure in early January our transactions have been littered with "No processor response" entries. We started off contacting the customers who had experienced problems but they had already moved on and bought else where ... and who can blame them ... I do the same thing myself.

Please would you keep us updated on your Protx progress as I'm seriously thinking of moving on too.

The highlight of the whole episode was a call from a market research company just over a week a go asking me to rate the HSBC card processing service. The lady asking the questions was clueless to the problems that we are having at the moment and couldn't write fast enough to keep up with my ranting ... bless her.
 
Upvote 0

CLSLTD

Free Member
Mar 29, 2008
18
3
Portsmouth
Its looking like legal action is required. I am still trying to get compensation (£3+) for the Jan incident and now theres been a second complete failure of ePayments. Went down 12.22 on Fri 28/3/08.
We switched our WorldPay gateway on as soon as we detected the failure...
My branch manager rang head of ePayments and had no luck with info. ePayments have decided to not repeat the black propaganda they tried to use last time - they said it was OUR websites that were the problem!!. They later told me "...you are the only customer that is complaining. You are very rude".
My regional Merchant services manager was informed at 12.22 that a 'level 1' incident had occurred - but seemed to not actually know what this meant!
I have been in lengthy discussion with HSBC to try and get them to understand that a massive failure of management... or fraud has been occurring. We have written scripts to constantly monitor ePayments and have kept data on the continued poor responses coming from their server(s) since the Jan incident.
Anyone that is interested in a class action we would be happy to co-ordinate with our lawyers.
Jonathan
 
  • Like
Reactions: HowardJB
Upvote 0
D

Deleted member 21013

Once again we have been left without any service or information and missed a busy weekend of trade. The first we knew of the latest incident was on Friday morning when we had a couple of telephone calls from disgruntled customers who could not pay for their orders, the rest of them presumably just went elsewhere. As soon as we realised the system was down we put Paypal back in place but orders have been a fraction of what we would expected. It is now 10 p.m. on Saturday evening and there is no sign of life from the HSBC. I have had no luck getting information from the bank about when the service is likely to be restored just grovelling apologies frm the call centre.
Howard
 
Upvote 0
Hi

I share everyones concern, and frustration bordering on anger at this latest failure, and the continuing appalling lack of customer service from HSBC.

I would urge everyone to send an email to epayments so that you have something on record. I am copying one of my emails below, and hope you might find it helpful.

I took a similar course of action in January, and when I submitted my compensation claim I based it on sales lost for the period (calculated with reference to average HSBC account turnover), net of vat. I added to this a charge of £30 per hour for my time expended in sorting out the total mess created by the failure, and a charge of £10 per hour for additional staffing costs in dealing with increased volumes of telephone calls etc.

The claim was agreed in two days, and paid into my account 14 days later.
I am sure that having documented my views in emails during the time of the system failure helped in expediting this, together with the fact that my claim was an accurate reflection of my losses, that were easily reconcileable by them. I realise that I probably lost first time customers that would never return, but such losses are of course not quantifiable. Whilst I am sure it is in all our instincts to try to claim something extra in respect of this, I would rather claim for my actual losses and receive the money quickly than enter into a long drawn out dispute regarding it.

Anyway, my email is copied below, and I hope that people may find my contribution useful.

Merchant Number xxxxxxxxx

I refer to my email of shortly before 8am this morning, in which I complained regarding the complete lack of communication from HSBC in regard to the ongoing gateway failure, with particular reference to the identical lack of communication in the failure early this year.

The gateway has now been down for over 25 hours.

Not a single communication has been received from HSBC regarding this.

I stated in my earlier email that I expected to receive a reply by noon. Disappointingly, but unsurprisingly, this has not happened.

The latest of many telephone calls by myself to your call centre has provided me with the latest update as follows:-

1. You are working on it.
2. You do not have any timeframe for when the gateway will be available again.
3. When it is back up, you will put a notice on the e-payments website.

An acknowledgement of the serious impact it is having on your customers would not go amiss. There are, apparently, no plans to inform your customers proactively as to what is going on.

I will now have to check your website constantly to discover when you have rectified the problem. This is in addition to expending time on updating my own website constantly, as, unlike yourselves, I believe that we should keep our customers fully informed of problems.

Please note that I will be expecting HSBC not only to compensate me for the large amount of orders that I am losing, but also for the significant amount of time that I have expended already on this matter, and, no doubt, will have to continue to expend until such time as you have rectified the problem and dealt with any data problems arising from it.

Many further issues arise from this failure, in particular the stability of the platform you are using, the inadequacy of back-up systems and the apparent total ineffectiveness of the contingency procedures arising from the so-called "Major Incident Review" of earlier this year.

I would normally expect a prompt acknowlegement to an email containing such significant and valid complaints as detailed above. With the track record of e-payments department I will not be holding my breath.

The lack of concern for the needs of your customers continues to astound me.
 
Upvote 0

CLSLTD

Free Member
Mar 29, 2008
18
3
Portsmouth
I think the most worrying thing about ePayments is the complete lack of notification. WorldPay contacted me within an hour the last time they had an issue. They are clearly doing this not because 'we don't know whats going on and are trying to fix it.."... but because if they deny theres a problem they won't have to compensate.

This time I forced them to email me so I had a record of them admitting there was a problem - so like Derek I urge you to email them (which they will ignore) and call them (15mins on hold) until you get something to prove there was a problem.

There policy of lies and denial last time prevented UK HSBC from realising the scope of the problem. Hence how this second collapse was allowed to happen.

Last time my Merchant area manager + branch manager continued to tell me that there hadn't been a problem and it was just me that was complaining - until I showed them the posts on here! ie no one in UK HSBC had been informed. The alternative - that UK HSBC was also involved in a cover-up is very scary!

My real worry is that India was so keen to cover up the problem that because they were hiding something much more sinister than just incompetence...
Jon
 
Upvote 0
D

Deleted member 21013

Thanks Derek for the text of your email which will be most useful in composing my own email to HSBC. It is now 9.30 a.m. on Sunday and there is still no service from them. It is not just their online service which is bad, we have one of their terminals in our shop which is painfully slow at authorizing transactions, often failing to connect altogether and leaving us to manually telephone the call centre while the bemused customer looks on thinking you are making off with their money.
I would be grateful if you and Jon can keep me updated on your progress with HSBC.
Thanks again
Howard
 
Upvote 0
Yes, thanks for the draft. I too have used it to draft my own. I really do find it utterly frustrating to be treated in this way.

This time I think enough is enough. I intend to persue this matter in every way I can. I do not think it is that HSBC has no regard for its customers; it seems simply to be a case of a huge organisation attempting to automate everything. By so doing, when something goes wrong it seems that nobody is ready for it. They certainly have created a system where it is very difficult to talk to anyone and I just wonder if internal communications are as poor. I suspect so.

On the whole, I find that everything works quite well most of the time. On the other hand, I have not experienced a single problem with HSBC that has been dealt with properly and on time or where any communication has been made. The idiocy of telling us that it will be posted on the secure e-payments site when it is up and running defies belief. Why on earth can they not have a status page which e-payment users can log onto which is entirely separate from the e-payments site? Like within the internet banking site? Or is there and I don't know about it?
 
Upvote 0

matty262

Free Member
Mar 30, 2008
8
0
I am absolutely disgusted with HSBC as an organisation - I am in the unfortunate position of banking with them as well as allowing them to process our card payments. The service generally is appalling but the secure e-payments fiasco is beyond a joke. I spoke with them this morning and they told me they ahd fixed it but then many retailers called them to day it wasn;t fixed so they are working on it again and it "should" be up and running by this evening. I will certainly be a claim for compenstaion as our sales have been serioulsy effected by this - as they were in January but I think the best thing we can all do is move to another payment gateway and merchant facility because HSBC obviously don't take this issue seriously. They ahve not ven bothered to call anyone in from the secure e-payments team to take calls because "they don't work at the weekend" to which i explained - "neither do I normally - but here I am" Lets hope they get it sorted soon!
 
Upvote 0

benjo132

Free Member
Mar 30, 2008
10
0
Many thanks Derek for the email information, I will get onto this at once. I, like you guys am amazed at how incompetent HSBC are in resolving this matter when it is apparent how many businesses are affected. I have actually emailed the Times newsdesk to make them aware of this as I feel that some bad press may give HSBC the kick up the backside that they need to sort out their shambolic systems. I urge you guys to also email some national newspapers as HSBC should not be allowed to simply sweep this under the carpet.
 
Upvote 0

jd1977

Free Member
Sep 17, 2005
48
3
Nice to see they are STILL advertising a system that doesnt bloody work
hsbc.co.uk/1/2/business/cards-payments/secure-epayments


3 days of trading lost, and my most busiest days of the week too. Not Happy.

If you went overdrawn on your personal account, HSBC indian or phillepenes call centre would phone you 8am saturday or sunday expecting you do something about it there and then, on that day.

Good thing about this, is we should all stick together and dont let them *shrug their shoulders*

If you get something like this tommorow morning, for example

You: "What compensation will I get and can you give me confidence you will have a better backup system in the future?"
HSBC "Sorry we cannot give compensation and are doing all we can to resolve this"
You:"Ok Goodbye then I'm moving to.." <insert name of new bank here>

This happened in January, I dont know how much trade others have lost I'm only small and have lost around £5000 from both technical cockups.
 
Upvote 0

Dwebs-Ltd

Free Member
Nov 29, 2007
2,019
264
Blackpool
I'm at the beginning of the second year of a two year contract with them - would I be able to get out of it do you think and move to another processor because of this?

If its in the agreement you signed with them that they have to provider a specific level of uptime on their service or provide notification of faults etc you could use this and other incidents as a reason to get out of the contract as they are not providing the service you signed for.
 
Upvote 0

benjo132

Free Member
Mar 30, 2008
10
0
If I decided to switch to another bank because of this I would make sure that HSBC paid all developer costs to implement the new system! We do need to stick together on this to make sure that they realise how serious to us it is.

The last techinical probelm did not cause us too many problems due to the time of day it occured but this could already of cost us £2000 in lost revenue. Not a massive amount as we are only a small business, but as we are a small business it is a big loss to us! Luckliy our website does accept Paypal payment and we have a "Phone Back" option so I have insructed customers to use these other options.

If this isnt resolved by the morning though we could start having serious problems.

I am wondering how many people received compensation last time and was this based on sales revenue or profit?

Thanks,

Ben
 
Upvote 0

Dwebs-Ltd

Free Member
Nov 29, 2007
2,019
264
Blackpool
If I decided to switch to another bank because of this I would make sure that HSBC paid all developer costs to implement the new system! We do need to stick together on this to make sure that they realise how serious to us it is.

The last techinical probelm did not cause us too many problems due to the time of day it occured but this could already of cost us £2000 in lost revenue. Not a massive amount as we are only a small business, but as we are a small business it is a big loss to us! Luckliy our website does accept Paypal payment and we have a "Phone Back" option so I have insructed customers to use these other options.

If this isnt resolved by the morning though we could start having serious problems.

If you would loose a decent amount of payments its probably worth setting up with another processor and using one as a backup option and the other as a primary. All payment processors have their issues even if its a 5 second blip due to network issues etc.

I doubt you will get any money out of HSBC for development costs.

Maybe someone should pass the issues your experiencing to theregister.co.uk as they are aimed at the more technical side of things they will publish it i know they did a recent article on HSBC's business internet banking missing its SSL for the messages section!
 
Upvote 0

jd1977

Free Member
Sep 17, 2005
48
3
I dont think HSBC will pay developer costs for another system, but if you are clear in your intentions to "go elsewhere" they will probably offer an incentive for you to stay with them. I have a feeling most departments will bat you from one post to the next and not take responsibility. You could try your commercial manager? as they are responsible for your business account, them keep taking it higher if you get no action.

The financial ombudsman has been key in claiming back bank charges, so they could be just as important in compensation for lost sales.

The main fact is NO warning was given before the weekend that it would be offline all weekend, you literally have to phone and chase up HSBC which in my book voids any contract with them.
 
Upvote 0
How about this clause:

'We will use reasonable efforts to inform you without undue delay through the secure e-payments service and/or our website(s) if any service under the secure e-payments service is not available'

I'd say they'd breached that, as there has been absolutely no communication regarding this whatsoever!
Will be giving them a call tomorrow I think.
 
Upvote 0

Dwebs-Ltd

Free Member
Nov 29, 2007
2,019
264
Blackpool
How about this clause:

'We will use reasonable efforts to inform you without undue delay through the secure e-payments service and/or our website(s) if any service under the secure e-payments service is not available'

I'd say they'd breached that, as there has been absolutely no communication regarding this whatsoever!
Will be giving them a call tomorrow I think.

Indeed if there is no notice via any communication method including the epayments website itself then that would break their clause.
 
Upvote 0

jd1977

Free Member
Sep 17, 2005
48
3
'We will use reasonable efforts to inform you without undue delay through the secure e-payments service and/or our website(s) if any service under the secure e-payments service is not available'

Only contact from THEM to ME, has been today (3 days later) at 5pm by email from secure-epaymentsAThsbc, I'm not disclosing any email details, but is a similar message to the 0845 service. "sorry for the problems etc"
 
Upvote 0
The e-payments website is entirely down, so a message there wouldn't help! After all the rigmarole last time this is unbelievable - I know I'm only a small business, and I won't be making any monetary claims, but to not even send out a customer service email about this is just disgusting. I've recently joined the FSB, so I think I'm going to be inquiring after their streamline service.
 
Upvote 0

Latest Articles

Join UK Business Forums for free business advice