HSBS Secure Epayments failure

[HowardJB]

Hi
On Saturday 05/01/08 at 9am the HSBC secure epayments system crashed and wasn't restored until the following day at 1 am. This is obviously a major incident for HSBC but an email I have received informs me that all the transactions we took on the Friday 4th have been lost and should be manually re-imput. Short of contactcting all the customers who placed orders on the Friday and asking for their card details again I am not sure how to do this. In addition our website was unable to accept orders on Saturday which is usually a busy day for us causing us to lose a days trading. Has anyone else on the forum been affected by this?

Regards

Howard

[Pelenna]

Yep, me too. Was terrible customer service, I wasn't told about the problem and the only thing they've done is put a little note on the website after you log in saying 'sorry for our recent problems'. I also have to contact everyone manually - sounds great doesn't it, sorry we've lost all your card details, can we have them again? I have to do it though, as the orders have already been despatched. Pah.

[IridiumCorp]

The HSBC platform is the old Clear Commerce software, a company that is no longer in existence. So your problem as a merchant of them is that when something goes wrong there is no one around with any great knowledge on how to fix it.

Not a great place to be for HSBC. Unfortunately its you the Merchants who suffer.

[HowardJB]

Hi Pelenna
Further to my first posting I have been contacted by HSBC who have been able to re-instate all of my transactions for the missing day except a couple which had been referred to ‘Fraud Review’ but they say I will not be able to view any of them on their site. I am waiting to hear from them to see if these transactions (The biggest of the day of course) can be recovered as well.
As is life wasn’t busy enough!
Howard

[Pelenna]

Hi there - that's good news, I'm still waiting to hear back from them myself. Thanks for letting me know!

[Sue F]

Both myself and my father in law use HSBC CCP and my hubby (who works for his dad) noticed on Saturday that orders were being rejected.

At 10am they told us that it would be fixed in three hours but orders were still being rejected on Sunday morning.

I only know that Friday's transactions are missing because my father in law's business partner noticed on Monday by accident.

Monday was spent ringing customers to ask for their card details again ... not the most fun I've ever had "Thank you for spending £1000 on furniture ... we've lost your payment". Inspires a lot of customer confidence I'm sure. I didn't manage to catch everyone until 8pm last night (Tuesday) hampered by conveniently getting the cold that's going round so my voice now sounds like an axe murderer making the explanation sound even less believable.

I'm concerned that HowardJB's transactions have been reinstated and he can't see them on his CCP gateway as I don't fancy having to face customers to explain their payment has now been taken twice!

Great time for it to happen ... January sale time ... it's bad enough for me but my father in law's company lost 70+ orders on Saturday and hubby had a similar number to phone on Monday to retake payments.

The thing that really annoys me is the lack of communication. There's an email address on the gateway page but the email I sent was returned with a stock reply that didn't answer my question at all ... brill!

[Pelenna]

The thing that really annoys me is the lack of communication.

Completely - I only had 3 orders disappear (for some reason most people were using Paypal that day) and it's easy to miss something like that. They didn't let anyone know - a simple email sent out to all customers explaining the situation wouldn't have been that hard would it?

[Dwebs-Ltd]

Well Protx had a major balls up last year, so nothings bullet proof so is it not wise to use two solutions i.e. hsbc epayments and something like protx then if one fails you can switch to the other.

Our payment system is built in such a way that we can disable payment methods instantly and can plug any payment gateway into it :)

I know shopping carts like cubecart, zencart offer multiple payment gateway support.

It’s worth while having multiple options if you would loose a lot of business if it wasn't working.

[Pelenna]

Good point, but the thing is that we weren't told the system wasn't working - I do have Paypal also on my website, had I known I could have just put up a temporary notice saying 'please use Paypal' or whatever. Also, it's the payments that were made using HSBC when it was working that have disappeared - another gateway wouldn't have solved that!

[Dwebs-Ltd]

Good point, but the thing is that we weren't told the system wasn't working - I do have Paypal also on my website, had I known I could have just put up a temporary notice saying 'please use Paypal' or whatever. Also, it's the payments that were made using HSBC when it was working that have disappeared - another gateway wouldn't have solved that!

Indeed it wouldn't have solved the vanishing transactions but if your clients were getting bounces which i believe they were i.e. the systems stuffed you could then switch to the second gateway.

Again nothing is bullet proof :(

We use Verisign’s system and touch wood we haven't had a failure yet but its always possible. Currently we don't push enough credit card transactions to warrant a second gateway but if the need was there we could have one up and running very quickly :)

[boho]

Yep friend of mine had their busiest day of sales on Saturday and then bump go the orders as no one could put their card details through, they managed to rescue their regular customers but still lost orders from the new customers who were trying. Interesting that the issue is one of an old unsupported platform, I'll have to let them know that!

[tall_c]

This whole HSBC failure is a nightmare, and not at all helped by the frighteningly poor customer service from HSBC.

I spoke to them a short while ago today, Wednesday 9th, and they were able to tell me the number of transactions that have successfully gone through on Friday, and the monetary total of these transactions. If you contact them and have your Client ID handy, they should be able to give you this info.

One thing I was told, which I think it's best to mention, and I don't think anyone previously has mentioned in this thread:
If you're transactions were put through as status 'pre-auth' then all these transactions are now lost. If you try and capture them now you won't be able to.

Also, a question regarding a further issue I've just encountered: how can I refund a transaction that was successfully captured on Friday? Normally I would process it through our API which would give a full credit against the original transaction [that way we don't need the card details - just the transaction ID].
I can't do it through the API [it just fails], and I can't do it manually through secure e-payments as you can't view any of the transactions!

It's a nightmare! Without the original card details we can't give the man his money back. And of course [like everyone else, right?] we don't store CC details as we are PCI compliant.

It will be very interesting to see what the result of the HSBC "Major Incident Review" are!!

[Pelenna]

Is it just me or is it still not working one hundred percent? I've had a couple of customers say it crashes when they try to return to the website after making payment, and it just crashed on me while trying to ship orders on the virtual terminal... could be user error though I suppose, as it seems to working fine now most of the time :|

[HowardJB]

Is it just me or is it still not working one hundred percent? I've had a couple of customers say it crashes when they try to return to the website after making payment, and it just crashed on me while trying to ship orders on the virtual terminal... could be user error though I suppose, as it seems to working fine now most of the time :|
I haven't been at work today but orders seem to have been coming in at a normal rate. Do you get many transactions that do not return the customer to your site, i.e. you get a payment but no confirmed order?

Howard

[Pelenna]

I've only had 2 in the past 2 days that have done this, both rang me to make sure their orders had gone through and said the system crashed when they tried to return to my website - however there have been quite a few other orders that have gone through fine, so it's probably not the HSBC system. It's probably just I'm a bit over-cautious now after this whole debacle!

[HowardJB]

I suppose I will have to try and sort it out when I go back in tomorrow, hopefully it will all be back to normal and all the money will be in my bank account!

[snorestore]

Just to let you know that I have had conversations with some very high up (so they tell me) people at HSBC over the last week regarding the repeated failures of EPayments.

To quote one of them "Our service has been well below standard. I apologise".

He then went on to treat me like a naughty child (he'd listened to a recording of one of my calls to the so-called help desk - in India of course - during which I lost it with them), at which point I decided enough was enough.

So now, folks, I am trialling Protx. So far so good. It's easy to sign up - especially as all HSBC EPayments users have a merchant number which they just transfer across - and feedback from customers is excellent. It took 48 hours from web sign up to going live.

We shall see how it goes. It will cost more than HSBC, marginally, but the freedom from hassle more than makes up for it.

I still have World Pay as a stand-by just in case. I have no plans to retain HSBC.

[Sue F]

Just to let you know that I have had conversations with some very high up (so they tell me) people at HSBC over the last week regarding the repeated failures of EPayments.

To quote one of them "Our service has been well below standard. I apologise".

...

So now, folks, I am trialling Protx. So far so good. It's easy to sign up - especially as all HSBC EPayments users have a merchant number which they just transfer across - and feedback from customers is excellent. It took 48 hours from web sign up to going live.

Thanks for posting this.

I'm getting increasingly frustrated with the HSBC epayments system. Since the massive failure in early January our transactions have been littered with "No processor response" entries. We started off contacting the customers who had experienced problems but they had already moved on and bought else where ... and who can blame them ... I do the same thing myself.

Please would you keep us updated on your Protx progress as I'm seriously thinking of moving on too.

The highlight of the whole episode was a call from a market research company just over a week a go asking me to rate the HSBC card processing service. The lady asking the questions was clueless to the problems that we are having at the moment and couldn't write fast enough to keep up with my ranting ... bless her.

[deviltronics]

So thats what all the emails from Protx were regarding HSBC settlements. Didn't know what they were on about!

[CLSLTD]

Its looking like legal action is required. I am still trying to get compensation (£3+) for the Jan incident and now theres been a second complete failure of ePayments. Went down 12.22 on Fri 28/3/08.
We switched our WorldPay gateway on as soon as we detected the failure...
My branch manager rang head of ePayments and had no luck with info. ePayments have decided to not repeat the black propaganda they tried to use last time - they said it was OUR websites that were the problem!!. They later told me "...you are the only customer that is complaining. You are very rude".
My regional Merchant services manager was informed at 12.22 that a 'level 1' incident had occurred - but seemed to not actually know what this meant!
I have been in lengthy discussion with HSBC to try and get them to understand that a massive failure of management... or fraud has been occurring. We have written scripts to constantly monitor ePayments and have kept data on the continued poor responses coming from their server(s) since the Jan incident.
Anyone that is interested in a class action we would be happy to co-ordinate with our lawyers.
Jonathan

[HowardJB]

Once again we have been left without any service or information and missed a busy weekend of trade. The first we knew of the latest incident was on Friday morning when we had a couple of telephone calls from disgruntled customers who could not pay for their orders, the rest of them presumably just went elsewhere. As soon as we realised the system was down we put Paypal back in place but orders have been a fraction of what we would expected. It is now 10 p.m. on Saturday evening and there is no sign of life from the HSBC. I have had no luck getting information from the bank about when the service is likely to be restored just grovelling apologies frm the call centre.
Howard

[DerekB]

Hi

I share everyones concern, and frustration bordering on anger at this latest failure, and the continuing appalling lack of customer service from HSBC.

I would urge everyone to send an email to epayments so that you have something on record. I am copying one of my emails below, and hope you might find it helpful.

I took a similar course of action in January, and when I submitted my compensation claim I based it on sales lost for the period (calculated with reference to average HSBC account turnover), net of vat. I added to this a charge of £30 per hour for my time expended in sorting out the total mess created by the failure, and a charge of £10 per hour for additional staffing costs in dealing with increased volumes of telephone calls etc.

The claim was agreed in two days, and paid into my account 14 days later.
I am sure that having documented my views in emails during the time of the system failure helped in expediting this, together with the fact that my claim was an accurate reflection of my losses, that were easily reconcileable by them. I realise that I probably lost first time customers that would never return, but such losses are of course not quantifiable. Whilst I am sure it is in all our instincts to try to claim something extra in respect of this, I would rather claim for my actual losses and receive the money quickly than enter into a long drawn out dispute regarding it.

Anyway, my email is copied below, and I hope that people may find my contribution useful.

Merchant Number xxxxxxxxx

I refer to my email of shortly before 8am this morning, in which I complained regarding the complete lack of communication from HSBC in regard to the ongoing gateway failure, with particular reference to the identical lack of communication in the failure early this year.

The gateway has now been down for over 25 hours.

Not a single communication has been received from HSBC regarding this.

I stated in my earlier email that I expected to receive a reply by noon. Disappointingly, but unsurprisingly, this has not happened.

The latest of many telephone calls by myself to your call centre has provided me with the latest update as follows:-

1. You are working on it.
2. You do not have any timeframe for when the gateway will be available again.
3. When it is back up, you will put a notice on the e-payments website.

An acknowledgement of the serious impact it is having on your customers would not go amiss. There are, apparently, no plans to inform your customers proactively as to what is going on.

I will now have to check your website constantly to discover when you have rectified the problem. This is in addition to expending time on updating my own website constantly, as, unlike yourselves, I believe that we should keep our customers fully informed of problems.

Please note that I will be expecting HSBC not only to compensate me for the large amount of orders that I am losing, but also for the significant amount of time that I have expended already on this matter, and, no doubt, will have to continue to expend until such time as you have rectified the problem and dealt with any data problems arising from it.

Many further issues arise from this failure, in particular the stability of the platform you are using, the inadequacy of back-up systems and the apparent total ineffectiveness of the contingency procedures arising from the so-called "Major Incident Review" of earlier this year.

I would normally expect a prompt acknowlegement to an email containing such significant and valid complaints as detailed above. With the track record of e-payments department I will not be holding my breath.

The lack of concern for the needs of your customers continues to astound me.

[CLSLTD]

I think the most worrying thing about ePayments is the complete lack of notification. WorldPay contacted me within an hour the last time they had an issue. They are clearly doing this not because 'we don't know whats going on and are trying to fix it.."... but because if they deny theres a problem they won't have to compensate.

This time I forced them to email me so I had a record of them admitting there was a problem - so like Derek I urge you to email them (which they will ignore) and call them (15mins on hold) until you get something to prove there was a problem.

There policy of lies and denial last time prevented UK HSBC from realising the scope of the problem. Hence how this second collapse was allowed to happen.

Last time my Merchant area manager + branch manager continued to tell me that there hadn't been a problem and it was just me that was complaining - until I showed them the posts on here! ie no one in UK HSBC had been informed. The alternative - that UK HSBC was also involved in a cover-up is very scary!

My real worry is that India was so keen to cover up the problem that because they were hiding something much more sinister than just incompetence...
Jon

[HowardJB]

Thanks Derek for the text of your email which will be most useful in composing my own email to HSBC. It is now 9.30 a.m. on Sunday and there is still no service from them. It is not just their online service which is bad, we have one of their terminals in our shop which is painfully slow at authorizing transactions, often failing to connect altogether and leaving us to manually telephone the call centre while the bemused customer looks on thinking you are making off with their money.
I would be grateful if you and Jon can keep me updated on your progress with HSBC.
Thanks again
Howard

[Stuartvb]

Yes, thanks for the draft. I too have used it to draft my own. I really do find it utterly frustrating to be treated in this way.

This time I think enough is enough. I intend to persue this matter in every way I can. I do not think it is that HSBC has no regard for its customers; it seems simply to be a case of a huge organisation attempting to automate everything. By so doing, when something goes wrong it seems that nobody is ready for it. They certainly have created a system where it is very difficult to talk to anyone and I just wonder if internal communications are as poor. I suspect so.

On the whole, I find that everything works quite well most of the time. On the other hand, I have not experienced a single problem with HSBC that has been dealt with properly and on time or where any communication has been made. The idiocy of telling us that it will be posted on the secure e-payments site when it is up and running defies belief. Why on earth can they not have a status page which e-payment users can log onto which is entirely separate from the e-payments site? Like within the internet banking site? Or is there and I don't know about it?

[matty262]

I am absolutely disgusted with HSBC as an organisation - I am in the unfortunate position of banking with them as well as allowing them to process our card payments. The service generally is appalling but the secure e-payments fiasco is beyond a joke. I spoke with them this morning and they told me they ahd fixed it but then many retailers called them to day it wasn;t fixed so they are working on it again and it "should" be up and running by this evening. I will certainly be a claim for compenstaion as our sales have been serioulsy effected by this - as they were in January but I think the best thing we can all do is move to another payment gateway and merchant facility because HSBC obviously don't take this issue seriously. They ahve not ven bothered to call anyone in from the secure e-payments team to take calls because "they don't work at the weekend" to which i explained - "neither do I normally - but here I am" Lets hope they get it sorted soon!

[benjo132]

Many thanks Derek for the email information, I will get onto this at once. I, like you guys am amazed at how incompetent HSBC are in resolving this matter when it is apparent how many businesses are affected. I have actually emailed the Times newsdesk to make them aware of this as I feel that some bad press may give HSBC the kick up the backside that they need to sort out their shambolic systems. I urge you guys to also email some national newspapers as HSBC should not be allowed to simply sweep this under the carpet.

[peterjhale]

I urge you guys to also email some national newspapers as HSBC should not be allowed to simply sweep this under the carpet.

That would be great - normally newspapers aren't interested in how businesses suffer - only consumers...

[Dwebs-Ltd]

Couldn't you contact FOS and make a complaint.

As a business your all still customers of the bank!

[jd1977]

Nice to see they are STILL advertising a system that doesnt bloody work
hsbc.co.uk/1/2/business/cards-payments/secure-epayments


3 days of trading lost, and my most busiest days of the week too. Not Happy.

If you went overdrawn on your personal account, HSBC indian or phillepenes call centre would phone you 8am saturday or sunday expecting you do something about it there and then, on that day.

Good thing about this, is we should all stick together and dont let them *shrug their shoulders*

If you get something like this tommorow morning, for example

You: "What compensation will I get and can you give me confidence you will have a better backup system in the future?"
HSBC "Sorry we cannot give compensation and are doing all we can to resolve this"
You:"Ok Goodbye then I'm moving to.." <insert name of new bank here>

This happened in January, I dont know how much trade others have lost I'm only small and have lost around £5000 from both technical cockups.

[Pelenna]

I'm at the beginning of the second year of a two year contract with them - would I be able to get out of it do you think and move to another processor because of this?

[Dwebs-Ltd]

I'm at the beginning of the second year of a two year contract with them - would I be able to get out of it do you think and move to another processor because of this?

If its in the agreement you signed with them that they have to provider a specific level of uptime on their service or provide notification of faults etc you could use this and other incidents as a reason to get out of the contract as they are not providing the service you signed for.

[benjo132]

If I decided to switch to another bank because of this I would make sure that HSBC paid all developer costs to implement the new system! We do need to stick together on this to make sure that they realise how serious to us it is.

The last techinical probelm did not cause us too many problems due to the time of day it occured but this could already of cost us £2000 in lost revenue. Not a massive amount as we are only a small business, but as we are a small business it is a big loss to us! Luckliy our website does accept Paypal payment and we have a "Phone Back" option so I have insructed customers to use these other options.

If this isnt resolved by the morning though we could start having serious problems.

I am wondering how many people received compensation last time and was this based on sales revenue or profit?

Thanks,

Ben

[Dwebs-Ltd]

If I decided to switch to another bank because of this I would make sure that HSBC paid all developer costs to implement the new system! We do need to stick together on this to make sure that they realise how serious to us it is.

The last techinical probelm did not cause us too many problems due to the time of day it occured but this could already of cost us £2000 in lost revenue. Not a massive amount as we are only a small business, but as we are a small business it is a big loss to us! Luckliy our website does accept Paypal payment and we have a "Phone Back" option so I have insructed customers to use these other options.

If this isnt resolved by the morning though we could start having serious problems.

If you would loose a decent amount of payments its probably worth setting up with another processor and using one as a backup option and the other as a primary. All payment processors have their issues even if its a 5 second blip due to network issues etc.

I doubt you will get any money out of HSBC for development costs.

Maybe someone should pass the issues your experiencing to theregister.co.uk as they are aimed at the more technical side of things they will publish it i know they did a recent article on HSBC's business internet banking missing its SSL for the messages section!

[jd1977]

I dont think HSBC will pay developer costs for another system, but if you are clear in your intentions to "go elsewhere" they will probably offer an incentive for you to stay with them. I have a feeling most departments will bat you from one post to the next and not take responsibility. You could try your commercial manager? as they are responsible for your business account, them keep taking it higher if you get no action.

The financial ombudsman has been key in claiming back bank charges, so they could be just as important in compensation for lost sales.

The main fact is NO warning was given before the weekend that it would be offline all weekend, you literally have to phone and chase up HSBC which in my book voids any contract with them.

[Pelenna]

How about this clause:

'We will use reasonable efforts to inform you without undue delay through the secure e-payments service and/or our website(s) if any service under the secure e-payments service is not available'

I'd say they'd breached that, as there has been absolutely no communication regarding this whatsoever!
Will be giving them a call tomorrow I think.

[Dwebs-Ltd]

How about this clause:

'We will use reasonable efforts to inform you without undue delay through the secure e-payments service and/or our website(s) if any service under the secure e-payments service is not available'

I'd say they'd breached that, as there has been absolutely no communication regarding this whatsoever!
Will be giving them a call tomorrow I think.

Indeed if there is no notice via any communication method including the epayments website itself then that would break their clause.

[jd1977]

'We will use reasonable efforts to inform you without undue delay through the secure e-payments service and/or our website(s) if any service under the secure e-payments service is not available'

Only contact from THEM to ME, has been today (3 days later) at 5pm by email from secure-epaymentsAThsbc, I'm not disclosing any email details, but is a similar message to the 0845 service. "sorry for the problems etc"

[Pelenna]

The e-payments website is entirely down, so a message there wouldn't help! After all the rigmarole last time this is unbelievable - I know I'm only a small business, and I won't be making any monetary claims, but to not even send out a customer service email about this is just disgusting. I've recently joined the FSB, so I think I'm going to be inquiring after their streamline service.

[jd1977]

Does anyone know of any other payment services? I know there is PRotx, do Nat West or Barclays have a secure epayments service?

[HowardJB]

Many thanks Derek for the email information, I will get onto this at once. I, like you guys am amazed at how incompetent HSBC are in resolving this matter when it is apparent how many businesses are affected. I have actually emailed the Times newsdesk to make them aware of this as I feel that some bad press may give HSBC the kick up the backside that they need to sort out their shambolic systems. I urge you guys to also email some national newspapers as HSBC should not be allowed to simply sweep this under the carpet.

I have emailed the Daily Mail a link to this thread
Howard

[jd1977]

If this is seen by the wider media/public, can I recommend keeping any confidential emails details or company details to a min, purely on legal grounds of course, ie. my name iis X and by branch manager is X and my company name is.. etc.

just being safe :)

[NickW]

duplicate post

[NickW]

As a small UK business whose online trade is conducted 100% via HSBC Secure ePayments, we've watching this latest outage with disbelief.

Aside from the thousands in lost sales, it's also costing us massively in out-of-hours developer and administration hours as we started integrating with another payment gateway (Protx) last night. Although it looks like we'll have a working system by tommorrow morning, it transpires that we'll still have to wait 48 hours for Protx to review our application and get us trading. Consequently this is going to be a very expensive issue for us.

We've certainly learned a hard lesson this weekend about having more than one internet payment gateway available to us.
I have the utmost sympathy for any firms who solely use HSBC secure ePayments for online payments, but do not have technical staff at hand to help them get up and running immediately.

As for comment on HSBC Secure ePayments handling of this situation:

+ No communication from them at all in any form, at any time, regarding the outage.
+ No explanation of what caused the issue, what the consequences are likely to be, and when the system will be available again.

This is in keeping with the way they handled to their last outage earlier this year (HSBC 'lost' a whole days trading data across their system in January this year. Customers who contacted them received varying vague explanations over the 'phone. After the event they also received a 'sorry' letter that promised a 'major incident review' and a hard copy of the transactions in question. Customers who didn't bother to contact them received nothing.)

We will be taking every action possible to claim against HSBC Secure ePayments for the consequences of their incompetance.

I'll be posting any information that may be useful to others about our claim against the HSBC here, and will be watching this thread with great interest to see how others get on over the next few weeks..

[jd1977]

I will also be keeping a close eye on the compensation progress, for those who rely 100%, I do have a sinking feeling this may not be resolved. (thousands of retailers x ££££'s each) The contract writers will be working away as we speak to get out of this one.

I'm sure a lot of businesses do however rely on a daily cashflow to keep going, I know clearance times allowing after 3 or 4 days of no sales it would be enough to cripple my business from further trading. At the very least there should be extended an overdraft period to shelter the businesses through this knock back (which they have caused).

We all work very hard as small businesses to survive, and make cutbacks within the business to save £££'s

All the hard work I have done since December in making cutbacks and lowering my overdraft have all been lost in 3 days of lost trading
nearly £3000 worth. Any shrugging of shoulders by HSBC on this one should not be allowed at any costs.

[CLSLTD]

Hi JD, well really i'm happy to share any information! Last time HSBC 'lost' 200+ peoples payments data, that included names, addresses, what they bought etc. No media covered it. Where it went... or who now has that data we don't know.
Their reconciliations team sent me via email a list of all payments they had for the Jan incident, ie the only data they had secured was funds data. I have been asking roughly once a week for them to send me the lost data, and they still won't (for obvious reasons) admit they don't know where the data is.
Therefore either they had a catastrophic hardware failure, or they were victim of a sophisticated hack or server hijack. If that was the case then its the case now... and that means someone now has all of your customers card details and could be charging your customers AS YOU!
This is serious stuff,
Jonathan

[jd1977]

According to an email I received, the system is working, they are just testing it with robots to make sure every part of it is robust and unbreakable for people to use again.

This would lead me to believe it was subject to a sophisticated hack or crap equipment/software.

[CLSLTD]

ha ha I can just see this insane robot wandering around knocking cups of coffee over! I've had no email, and there IP still isn't responding... More lies is my guess. Really, I've spoken to people at all levels many times, they have no idea what they are doing.
They outsourced it to India and management have lost complete control of it. ePayments is not a professionally run organisation.
Jonathan

[jd1977]

You have just raised a very good point there, although the call centre is in India (I have no problems with this, and have received an OK service in terms of other services) but where is the actual payment server, or system/s ? Is this UK based ? or America? or even India. It would be interesting to know where the actual "Nuts and Bolts" are.

[Grassman]

According to an email I received, the system is working, they are just testing it with robots to make sure every part of it is robust and unbreakable for people to use again.

This would lead me to believe it was subject to a sophisticated hack or crap equipment/software.

You've had more information than me then
Oh how I relish being kept in the dark :mad:

[jd1977]

You've had more information than me then
Oh how I relish being kept in the dark :mad:

I have phoned them at least 3 times a day since Friday 11am.

[sculli]

I have had an email - promising regular and honest (!) updates (that will be a first). For two years I have complained about the problems I have experienced with secure e payments and have been fobbed off with every excuse possible. I am 100% reliant on it as a payment method, I have no technical back-up and have lost thousands of ££££ through the incompetence of the site. I had one of my many conversations with Ashish and his boss (I think there are only the two of them) on Thursday and was assured that everything was working perfectly and it was my customers pressing the back cursor that caused that dreaded 'waiting for HSBC message', none of my french customers have any money in their accounts which is why their payments don't go through and at the end of the day all my customers who have experienced problems are liars (in not so many words) ..................
I thought of alerting the press in January, however due to the loss of customers credit card details thought it would be more detrimental to my own business if it went public - however this weekend has been the final straw!
Please everyone make your business managers aware of the situation (very few of them seem to have any information on secure e payments), this is a situation that has got totally out of hand and is terminally damaging to small businesses.

[jd1977]

I received an email from the same person sculli, you have to wonder whether the server is kept in someone's back bedroom in India with just 2 people keeping check, one of those a toy robot lol . Seriously though, dont lose your rag, try stay calm and talk to your commercial manager be firm and make it clear to them you are unhappy and demand to know what they intend to do to compensate you for the loss in trade.

[matty262]

I am finding the whole HSBC experience beyond belief (banking, e-payments - the whole lot!). We have lost thousands this weekend and as others have mentioned, we rely heavily on the regular cashflow we experience through secure e-payments. We have Paypal but as we all know, these funds don't reach your account for 5 - 7 days. I have made several phone calls since Friday but there is no commitment to a timescale to resolve this and then, at 5pm, the update message was gone and they just said these offices are now closed - goodbye!! Unbelieveable. In my opinon, there has not been a single person working on this and if you want to know where the server is - you can get a good idea by the time it comes back on again when the technical team walk in Monday morning at 9am in whichever country they are in and switch the server off and back on again to reset it. I was even told on Saturday that it was up and running but they were resetting the links! Absolute garbage that they were told to say because if you disagree and tell them how ridiculous that is, the person on the other end of the phone can honestly say they know nothing about e-payments (because the e-payments team don't work at weekends) and this is all the information they have. In the end, there is no point even speaking to them because nobody there knows anything about what is going on!

[julie@affordablelighting]

can anyone give me the email address to send my complaints/claim to of hsbc secure payments

thank you

[jd1977]

can anyone give me the email address to send my complaints/claim to of hsbc secure payments

thank you

secure-epayments AT hsbc DOT com.

[boarderblokey]

Hi all,

Just wanted to share our frustrations with you and give support to everybody else in the same boat.

We're a small company running 3 sites all with HSBC epayments. We do have Paypal as back up but many still seem reluctant to use this.

We're down around 1k from this outage, I'd guess.

In the last month we've had 2 of our sites taken off-line due to billing errors (our hosting companies fault not ours). This lost us around £1000 and took a while to re-appear in Google and also wasted Google Adwords click as we did not realise for 24 hours.

BTBroadband have kindly charged us £25 a month extra (on top of our £19) since November 2007 but only informed us in March with a total surcharge bill of £100 for downloading more than our allowance.

Three mobile kindly ended our 18 month 1/2 price line rental a whole year early and it took 45 minutes on the phone to sort their error out.

It seems that all the big boys will just use and abuse us and take our money and then expect us to sort out their errors.

Phew, end of rant-it's been a bad month!

Let's hope HSBC get it sorted.

Cheers

Nick

[divorce-online]

According to an email I received, the system is working, they are just testing it with robots to make sure every part of it is robust and unbreakable for people to use again.

This would lead me to believe it was subject to a sophisticated hack or crap equipment/software.


Could you post the e-mail on here as we might have to refer to its contents at some stage. Just their bit would be fine.

[angelbud]

I have been away for 3 days and have come back to no trade for 3 days - i will be going into see my business manager monday morning and have sent an email notifying my intention to claim compensation.
It does not take much these days to have an email system updating there customers of a situation - this is totally disgusting service from HSBC.:mad:

[jd1977]

Well here we are the 4th! DAY.

https://secure-epayments.hsbc.com/cgi-bin/HSBC_Engine

And still blank as at 1am 31st March 2008.

So much for "up and running this evening" yesterday.

Worldpay and Nat West have just gained a new customer. Too late HSBC

[IridiumCorp]

Hi All,

Some of you contacted us via PM and email to see if we can help. The answer is yes we can. Please feel free to send a note to support@iridiumcorp.co.uk. There are a number of things we can do and will look at each of your cases individually. We will be doing the connections as fast as we can and in the order they come in.

Iridium Support

[matty262]

7.40 on Monday morning - still not working. The home page is there but no message giving any update as they promised yesterday. I assume sales we did get on Friday before it went down won't be settled today! How much worse can HSBC get!

My mistake - 7.45 and it appears to be working but I have been checking since 6.30 and it has only just started to work for me

[benjo132]

It seems that Matty262 above was right, as soon as Monday morning comes round and the engineers come back to work things start working again!

[sculli]

Lucky you! but will you ever trust it again?
08.17 and I still have nothing.

[sculli]

I take it back - it is working on my website, however I am still unable to log in.

[Grassman]

you can get a good idea by the time it comes back on again when the technical team walk in Monday morning at 9am in whichever country they are in and switch the server off and back on again to reset it.
I really hope you're not right.... but :eek:

[TimmoB]

THis has been a big problem, this weekend i will have lost lots of sales, i do have a backup in place which is Paypal, but clients seem reluctant to use it, i think they think that they are buying from ebay if they use paypal.

I started the process of moving to Protx on saturday and hopefully that will be live by wed.

I will be writing a strong letter to my Business manager, informing him that this weekends, debarcle has cost at least one customer, me, and I urge the rest of you to do the same.

Good Luck.

[divorce-online]

I take it back - it is working on my website, however I am still unable to log in.

As at 10.13 am it still is not working.

We are on the phone as apparently the time stamp on the site is now wrong because of daylight saving and my IT manager is trying very hard not to shout at them.

They are saying we need to change the clock on our server to GMT? This has never been an issue before.

Has anyone else got this problem?

[jd1977]

Looks like the server monkey came in this morning and switched the commodre 64 back on running Windows 95.

[divorce-online]

The thing is they could not or would not explain why we have never had to do this before in 3 years of using this service.

We are signing up for protx as we speak as we have used it before when we bought another site and never had any problems.

[divorce-online]

Just to let everyone know we are back up and running at 11.31 after our hosts changed the time on the server by winding it back 2 minutes!!!

Thanks for all your help it has been good to know there are others out there to vent ones spleen at.

Any advice on obtaining compensation as we go through our business managers will be really helpful.

[jd1977]

deleted post

[jd1977]

The thing is they could not or would not explain why we have never had to do this before in 3 years of using this service.

We are signing up for protx as we speak as we have used it before when we bought another site and never had any problems.

I think they are very embarressed this time, and recognise the January problem on top of the last 3 days, if you speak to your branch manager he will give you more details.

Cant say anymore but it looks like HSBC are now taking seriously and will not be wanting to lose any custom at all.

[CLSLTD]

My branch commercial manager told me to write to:
servicequality[at]hsbc.com

Dear Sir/Madam,
We are extremely unhappy with the level of service from ePayments.
I have a catalogue of complaints that I do not have time to list here - but my branch manager Simon xxxx is fully aware of the issues and has been in extensive correspondence with me about it.
ePayments is not functioning as a payment gateway. Its is regularly offline. We are not notified of this. They deny theres a problem.
We spent many f thousands of pounds to create an integration with ePayments - so we are stuck with it.
Urgent action is required.
I have been communicating with other angry customers about this and we are considering a Class action against you.
I expect your swift response.
Jonathan Money

[CLSLTD]

I have also written to my area Merchant Services manager demanding discussions about compensation. My branch have given me extra £5k overdraft without any questions btw.
If anyone is interested in joining us to take them on for compensation - we are looking for 10k in compensation...
Jon

[jd1977]

I'm discussing my case on an individual level, as my b manager has been helpful so far, I have emailed a previous weeks sales figures for the 3 days to the email address they provided. So I am hoping they act upon that.

In the meanwhile, I'm sure the local branches themselves will cover any overdraft issues due to shortage of cashflow. They are pretty willing if you speak to them directly.

Try not to lose your temper in a direct manner with any staff, as previously proved with the bank charges reclaim: they might stop your account if they feel you are going in all "guns blazing." This will take some tact, but it's looking good so far :)

[Dwebs-Ltd]

http://www.theregister.co.uk/2008/03/31/hsbc_epayments_badness/

[jd1977]

http://www.theregister.co.uk/2008/03/31/hsbc_epayments_badness/

Not a good advert is it, I think they realise this aswell because the correspondance I have received today , has been in all fairness really good.

[Pelenna]

What correspondance? I think, like last time, if you haven't been in constant contact with them they won't bother to contact you - I know I haven't had any communications from them at all! Just waiting to hear back from streamline now, and then I'll bother to ring HSBC - to cancel.

[matty262]

Well done to all of you who have managed to have any contact at all with your branch manager. I called my "Relationship" Manager at 9am this morning then again at 11.30am and then tried a further two times this afternoon with no response at all. A colleague of hers took a message and I explained briefly what the problem was and when I called back, the same colleague told me that she had contact epayments to verify that it wasn't just me having problems and that it was all OK now and I would be getting a letter of apology (I bit my lip until it nearly bled at that point). I explained that it was not another apology I wanted - it was assistance with handling the damage to our cashflow that I was looking for and she said my manager would call me back. I then sent an e-mail to secure e-payments giving them details of our daily sales for March up until Friday, our sales for each Friday, Saturday and Sunday since January and the projected sales for the last weekend based on the orders we received upto the time the payment system collapsed and gave them the financial compensation I would be pursuing based on the average of all these figures. I also stressed how disappointed I was not just with the failure of the system but also the lack of communication and copied in my Relationship Manager. I had read receipts from both parties at 4.30 and 5pm but still not a single word from anyone as to how, or if, they intend to assist me. Frustrating!

[jd1977]

You'll probably hear back tommorow matty, and within 5 days regarding the compensation.

It does seem each persons outcome is based upon the availability of one's commercial manager, they will also have access to internal information on this and can advise accordingly. I know when I phoned my manager they had phones ringing all over the place in the background, he even said this too.

[DerekB]

Just an update on my situation.

I copied all my email communications to secure epayments to my commercial manager over the weekend.

This morning the system was up and running around 7.45am.

I received a call from a lady at the call centre at 9.50 apologising for the inconvenience caused by the outage and advising me it was up and running. I politely told her that I was well aware the system was back as I had been personally checking it every 15 minutes during waking hours since it failed. I also advised that it was somewhat more than an "inconvenience" as we had lost thousands of pounds in orders. I did, however, thank her for calling. After all, its not her fault!

Around 10.15 I received a call from my commercial manager, who apologised profusely (even though its beyond his remit). He was very annoyed that he was not made aware of any problem by the HSBC command structure. He promised to fully investigate, and to support my claim for compensation. He was particularly concerned at the total lack of communication with customers and that lessons appeared to have not been learned from the January incident. I have to give him 10/10 for his honesty and helpfulness.

Later this afternoon I received an email from epayments apologising for the late reply to my emails, promising a frank and full reply to all the points raised by tomorrow, inviting me to submit my claim for compensation and promising to respond to this within five working days.

I think some butt has and is being seriously kicked at long last. Heads really should roll at HSBC for this.

[benjo132]

You might have seen this already but it is on the Times Online. I cant post the link as I havnt had 15 posts on here but perhaps some one else can. Do a search in google news for HSBC and its about the 8th result.

But its good to see them getting a roasting about it, and it says in the article that they have agreed to compensate customers. I have just sent my claim in.

[brownie]

has been in all fairness really good.

I should hope so, if the certificate expiry reason is correct I'm astounded.

An institution this size should be monitoring certificate expiry dates, sheer negligence.

[jd1977]

http://business.timesonline.co.uk/tol/business/money/consumer_affairs/article3655174.ece

Hi Brownie, in relation to the HSBC staff have been helpful in all fairness, but the certificate lapse was just silly really for a large organisation.

[jd1977]

Is the system down AGAIN?!

Just tried to place an order online as a customer had difficulty

"Communication Failure

We have been unable to proceed with your transaction due to a communication failure.

Please Close your browser window and confirm the status of your order with the merchant"

Anyone else having any problems today?

[jd1977]

edit: working again now, must be a browser or connection issue.

phew

[wunpac]

Hi there,

Is anyone having issues with Epayments again? I've just called the call centre and they acknowledge an issue but say its only with a few customers. I'm not inclined to believe them from past experience.

[julie@affordablelighting]

yes were off again the call centre says they know this and has been off for an hour

[CLSLTD]

We texted by our monitors at 18.54 - the URL is dead. So its a complete server failure again.
Amazing.
Jon

[NickW]

Yes, unbelievably the Secure E-payments website is down again. (since 5:30pm GMT)
[ we'll it's not a total stretch of the imagination after the way they delt with the last outage ]

One of my staff who was on the phone to HSBC regarding this 15 minutes ago, was then promised full restoration of service in the next 10 minutes by someone at HSBC...

[julie@affordablelighting]

probably back on when they get in in the morning

[DerekB]

Words fail me. This is going to cost them big time.

The most annoying thing is how bad it makes us look to our customers.

Anyone got the HSBC phone number - that is , if they are answering it?

[TheBeekeeper]

I've just joined this Forum after searching Google this evening with "failure down HSBC secure epayments". What a surprise it comes out as No 1! HSBC managers clearly went to the same business school as British (T5) Airways managers.

[nicola]

Hi I found this blog on the last service outage, I had hoped (:| silly me) that would be the last of the problems.

Anyway they are answering the phone number is 0800 587 0633. They told me half an hour ago the service would be restored in about an hour and it was less of a major problem to last time. Now why do I have very little confience in that a) they even have a clue what the problem is b) their capabilities of fixing it this evening.

As for last time they made us a compensation offer within 2 days which was fair and we accepted. We dealt directly with the customer service team at Leicester Head Office.

So yet again I spend an evening sat with the office telephone, mobile chip and pin unit and laptop on my knee hoping to catch customers mid order!

Large glass of wine someone please ... :)

[wunpac]

It's now 22:39 and aparently everything will be working in 30 mins. Fingers Crossed.

I think the lesson I have learnt from all these problems and one that HSBC needs to learn is that you need to have redundancy built into your systems. I'm frustrated at the magnitude of incompetancy on display from HSBC. I would never have believed a multi-national giant could be such a joke. I'm also kicking myself that we don't have back-up systems ourselves considering the losses that occur in a situation like this.

Can anyone recommend another payment service provider? Low transaction fees and an XML API are priorities

Thanks for the Freephone no. by the way Nicola. At least I'm not having to pay to complain anymore.

[DerekB]

At 9.55pm, the latest information as follows:
UK call centre shut at six. All calls are being re-routed to the section that normally deals with shop terminals.
Earlier advice, at 6.30 was that the system was being shut down for fifteen to twenty minutes for maintenance and would be restored thereafter. The next (and latest) update they have was a carbon copy of what they were saying Sunday of last week "The management apologises etc etc...IT working to resolve...no timeframe...etc" ....sounds ominous to me. They are still saying that it is not a major problem on the scale of the earlier outage this month...but then "they would say that wouldn't they?" to use a famous quote. It is certainly a major problem to the credibility of my business with my customers, and I assume, to many thousands of other businesses in the UK. I will be seeking compensation again from them, please everyone, make sure that you claim. I have also emailed my commercial manager (twice) this evening, and asked him to ensure that this matter is raised at the highest levels within HSBC. We have the right to answers, and cannot be expected to believe single paragraph assurances that the problems have been addressed and rectified. Just one final point, if you ring the call centre please be courteous, remember its not the fault of the guy or lady on the phone..I know they have had people shouting and screaming at them this evening. Its understandable we want to vent our anger, but they are not the people to do it to!

[jd1977]

Great *adds this outage time to the next compensation bill*

Not working for me either, what time did it first go off around 5pm?

[DerekB]

Last successful order we received was at 5.38pm today...sometime between then and 6.30pm I believe.
Make sure you also charge for your time in your claim- I am claiming for the entire evening as I have been checking the gateway constantly every ten minutes and updating my companys website accordingly. When you put in your claim for losses, make sure you include compensation for your personal time spent. I have charged this to them at £30 per hour in January (which was paid) and in the recent claim (which I submitted on Saturday) at £30 per hour for Friday and Saturday and £60 per hour on Sunday. I will be claiming £60 per hour for this "overtime" - my leisure time is scarce enough.

[jd1977]

Blimey Derek you stuck it to them, I never claimed anything in Jan? could I still make a claim for this. Can you remember what date it was?

[angelbud]

This is becoming very inconvenient now - they must acknowledge the many effects this has on businesses using the payment gateway -not only the loss of sales, time spent while the system is not working but also business reputation and loss of opportunity when making a claim.

[DerekB]

It was January 5th. My sales were down £2000 that day, I claimed for loss of profits and time spent. For the end of March failure I claimed for loss of profits, time spent, and also suggested that they should pay a further sum as a goodwill gesture to enable my faith in them to be restored and to ensure that I remained a customer of theirs. I will await their reply with interest on that one. (I pay around £900 a month in fees). Yes, I would claim for January..add it on to this one, say something like "I was prepared, at the time, to accept the losses caused by this one day failure, but in light of the two further failures that have occurred I now expect to be compensated for the earlier incident. I am aware that HSBC have compensated other customers for this, and so expect equal treatment".
Can't remember how many billion pounds they made in profit in their last financial year, but I am sure they can afford to compensate you for the losses you incurred.
I think they should spend some of it in updating their epayments framework too...their Amstrad 8256s are obviously overstretched!

[angelbud]

Thanks Derek...
I also did not claim for the outage in Jan - think i will add it to this one!

[jd1977]

http://gfxzone.planet-d.net/articles/zx_spectrum_graphics-pentagon_128.jpg
Yes was going to say, the spectrum has overheated again ;)

[jd1977]

Into the 8th April now 00:50 still no joy, hopefully someone will be in the morning to reset the server again.

[lobleyandy]

06:24am and things are still down. :-(

We use Protx as our backup so we are not losing too much business, but is anyone else having problems with 3D secure on protx?

Andy

[DerekB]

7.10 and still down...now over twelve hours for a problem we were initially told was going to be remedied in a matter of minutes.

I have just sent my third, blistering email to HSBC, copying in my commercial manager. Please ensure you do the same. They record all calls and emails, so this will aid you when you seek compensation. If no-one is recorded as calling or emailing they (wrongly) may assume that you weren't bothered or affected significantly.

I wonder how many people are, though, not going to be aware of this further major blow to our businesses until they get into work this morning and find that the normal overnight crop of orders is non-existant.

[Nomadsounds]

I have only just discovered this new problem, as usual thanks to a customer shouting! When exactly did it fail? I was having problems with a Swedish customer about 5pm yesterday. Was it OK then?

[DerekB]

8am...there is now a message on the HSBC call centre number. It advises that they hope to restore service this morning. Apparently they remain committed to providing us with regular updates. If I wasn't in such a bad mood I would almost smile at that. The message was not there when I tried calling at 7.15am.

[DerekB]

Failed around 6.30pm yesterday.

[Nomadsounds]

Thanks for the update. At least with this forum we can find out what is going/gone on! No thanks to HSBC!

[DerekB]

Exactly. Make sure you complain by email to secure-epayments at hsbc.com, copy your commercial manager in all correspondence. I have already emailed them to advise them how to handle their customer service in such situations, as they themselves have obviously not got a ****ing clue. For example, what is the use of saying they will post a message on the secure epayments website when the system is restored? I have made the point that because of this, I have now spent eight hours checking for this message every ten minutes, time that I will expect them to pay for. I have suggested that they offer customers the option of being advised by SMS as soon as the system is restored. Hardly rocket science. I am also this time attacking the issue of losses in a different way. We can all (roughly) calculate the actual lost orders for a period of time based on averages over a time period. What we cannot do is to quantify how many customers are lost for ever afterwards. I am asking for substantial compensation above my actual losses to enable me to generate business via advertising to compensate for the unquantifable losses that have undoubtedly been caused by these three incidents this year.

[jd1977]

12:03 still down, that's quite appauling. I havent phoned them yet either, cant be bothered, they should inform me instead.

[matty262]

I didn't realise until 8.30 this morning that it was down AGAIN! They had the nerve to offer me a small fraction of the compensation claim last week but that is a good idea of DerekB and I shall certainly be throwing in the extra advertising required if they come back again with anything less than I have asked for. My relationship manager has been pretty good though and increased our overdraft without a problem to make sure we were covered for the cashflow. I would urge you all though to get a backup system in place get it all up and running then drop secure epayments because this is not the last time this will happen as the system is clearly unstable. I now have protx as well plus I am changing my merchant account (still hsbc but not the secure e-payments one) and got a 10p reduction in debit card transactions and a 0.8% reduction in credit cards and the new account will not be linked in anyway to secure e-payments - and that has to be a good thing!

[sculli]

It would appear that the website is now functioning again. However all records of past sales and current unshipped have been deleted............
What a mess!

[NickW]

The secure ePayments website appears to be back online, however when trying to view reports the following errors are returned:

1004 - No Reports definition is configured for ReportName: 'CCE_T_SC_A_A_V2'.

1004 - No Reports definition is configured for ReportName: 'CCE_CurrentBatch_Total'.

Therefore, I am instructing my staff to avoid taking telephone orders using secure ePayments as there are clearly still database issues.

[DerekB]

I can now log in, but when trying to access any reports I get the following error message:A problem occurred while processing your request1004 - No Reports definition is configured for ReportName: 'CCE_T_SC_A_A_V2'.
Please contact your administrator if the problem persists
I assume that they have got the actual website back up, and are now putting a couple of new duracell batteries in the data disk drive units. Or perhaps waiting for someone to come back from the shops with more cheese for the mouse-wheel driveshaft.

[NickW]

Has anyone dared to put a manual transaction through or taken a web order yet?

[sculli]

Yes - I have just had two web orders come through but no customer details. Do I contact the customer now and say sorry HSBC have lost your card details or do I wait and see if they miraculously appear?

[NickW]

The incompetance of the HSBC secure ePayments team is staggering:

It seems that now they have prematurely allowed access to a system that is not functioning correctly, thereby causing further confusion and frustration, with the possible bonus of more expense for the merchants as their customers data goes missing.

Absolutely unbelievable.

[DerekB]

I think I will wait before advising customers to use. Back in January the nightmare of not knowing what data was lost or recoverable was even worse than not being able to receive orders. I am so glad that I did not follow the "official" early HSBC advise then, and contact all my customers to tell them that their card details had been lost! Unbelievable!

[webvictim]

We use the HSBC secure e-payments system too, and I agree that this downtime and the unprofessional way in which they have handled it is simply unbelievable.

[mkhilton]

I've just entered a manual transaction it worked / approved the sale but cannot now check as no reports

[jd1977]

same here, getting the following
A problem occurred while processing your request
1004 - No Reports definition is configured for ReportName: 'CCE_T_SC_A_A_V2'.

Please contact your administrator if the problem persists

If they have lost ALL past orders permanently , I will go balistic. As that will mean writing out customer refund cheques if refunds need to be done.

Had to do that for a couple of orders back in Jan

[jd1977]

Have they lost all yesterdays transactions up to 5.30 GMT? as nothing has been paid into my account today on the 8th April, for sales up to 5.30? please tell me they havent buggered that up.

[sculli]

At the moment there are no records of current or past sales showing on the site at all. I have just phoned secure epayments and they cannot say whether or not they will be able to retrieve any new order details. They have just taken the site off-line again as well.

[jd1977]

Hi Sculli, Confirmed, yes they have
https://www.secure-epayments.hsbc.com/cgi-bin/HSBC_Engine down again

"Our database teams of professionals have been working through the night" my sides are splitting.

[mattseo]

I'm sticking with Nochex never had one problem like that with them :-)

Also my clients who use hsbc have switched to nochex today and have started trading today.

[webmate]

It's amazing how large organisations like HSBC can get away with such situations. If one of our servers was down for 5 minutes, our customers would threaten us with legal actions.

[traveller]

Hi all,

After yet another outage im another busines customer searching on Google for anyone else who has had problems - theres a fair few of us!!! We didnt claim in January as they reduced our credit charges as away fo compensation but after these two we will be. Could someone possibly let me know the exact times the March outage was so we can put in compensation claim?

[mattseo]

Yes my customers :mad: caused me loads of stress. Fixed the problem by removing the hsbc intergration codes and replacing it with nochex details.

My clients were back in business within 2 hours thanks to nochex.com


Hi
On Saturday 05/01/08 at 9am the HSBC secure epayments system crashed and wasn't restored until the following day at 1 am. This is obviously a major incident for HSBC but an email I have received informs me that all the transactions we took on the Friday 4th have been lost and should be manually re-imput. Short of contactcting all the customers who placed orders on the Friday and asking for their card details again I am not sure how to do this. In addition our website was unable to accept orders on Saturday which is usually a busy day for us causing us to lose a days trading. Has anyone else on the forum been affected by this?

Regards

Howard

[jd1977]

March was 3 days, fri, sat, sun. if you look back on this thread more precise times should be available.

They have taken the system offline again, so remember to claim for the 5:30 GMT until 12:11, then again from the time it went off again 13:50 onwards.

[CLSLTD]

WHo wants the bad news.... my monitors just alerted me... down again. Was up from lunchtime to 15:10... so we got 3 hours out of it...

[jd1977]

I'm going to put in a claim straight through from 5:30pm yesterday onwards to whenever this nonsense ends, they can stick the "couple of hours uptime"

[mattseo]

I had so many phone calls from my clients blaming me

[peedeesee]

The system seemed to first go down at about 4pm BST on Monday. I started getting the dreaded 'Input queue full' response.

I immediately rang the 'help' desk. At first they denied all knowledge of any problems. After a bit of prodding they realised something was wrong and promised to call me back.

An hour later I rang again and was told that there was a problem for a few UK customers. At which point I asked what their other customers were doing.
I was then told that system was being restarted and would be back in a few minutes. Quite.

What are people using as an alternative. I'm currently seeting up Protx but this will cost me an additional 10p per transaction. Barclay's ePQG looked ok but there rates were high, e.g. debit cards = 23p + 1% !!!!

What sort of compensation are people getting, did they get last time, or are hoping to get?

Time to give my account manager another call. I think he's hiding under his desk...

[mkhilton]

Just Rang the Helpline some people are able to log on told it wouldn't be much longer before all services are back including reports etc , No data already processed has been lost . or so they say we shall see !!!!!!!

[mattseo]

But will it happen again is the question

rather deal with a company like nochex that gives you an account manager who you can call here in the UK :D

[jd1977]

We are near approaching 24 hours of distrupted service now. Dont know about everyone else but I will be claiming for the full 24 hours so far.

[NickW]

23 hours of downtime and no resolution in sight.

I'm certain whatever amount of compensation that HSBC intend to pay is not going to cover the the additional labour costs that have resulted for many of us, nor the unquantifiable damage done to businesses reputations.

If they had sold this unreilable service to US companies, they would be preparing for a class action style legal suit right now.

[DerekB]

file:///C:/Documents%20and%20Settings/derek.DEREK/Desktop/12.jpgMAINFRAME POWERSOURCE HAD DIED.

[DerekB]

sorry people...seems you can't copy photos into the text box.

[Halcyon]

We have been using HSBC for one week now. After weeks of trouble and lack of communication from HSBC to get it up and running, this is the final straw. I have gone back to WorldPay and will never use Secure e(or not)Payments again.

I think the whole system is being run on my old ZX81!!

[jd1977]

Has to be hosted somewhere online first Derek, wont show local file names.

Is it a pic you have got from the net? just send us the net link :)

I tried phoning my commercial manager today, no luck. Incidently the compensation from the last 3 day outage was paid into my account this morning.

Just missing yesterday's sales, and the current outage compensation and maybe 5th Jan if I push my luck. then it will be sorted then.

The biggest issue I have with this, is that it buggers up your cashflow running on an overdraft and relying on clearance times when there is a delay of cash going in an extra few days can really cause problems.

On the register again
http://www.theregister.co.uk/2008/04/08/hsbc_e_payments_problems/

[wunpac]

Hi Guys,

I've just had an email from my commercial manager. Not good news I'm afraid. I've copied it below

--------------
Further update.

The secure ePayments system will be suspended until 9am tomorrow so the database suppliers and providers of the system can undertake further testing and correction.

I will advise you if there is any change by 9am tomorrow

Regards
----------

[Motor-Mouse]

I've taken my own advice, not HSBC's and removed them completely from our web site. Are they not suposed to be offering a service to support us little people. I've had nothing but grief having only started using them a couple of weeks back and within hours they were out for the whole weekend and now we are still none the wiser for todays little escapades. I emailed them twice this morning still with no results. And yet I've had phone call after phone call from my potential customers which I'm having to let down nicely.
]
I think I'll shop else where.

Will you all join me?


Motor-Mouse.

[Halcyon]

Just got this from HSBC.....

We are extremely sorry to advise that we are still experiencing problems with our Secure ePayments system.

The service was operational from 12:00pm today but we have since encountered a number of further performance issues

As previously advised the issues that we are facing relate to the databases.

This is the first time that we have experienced this particular issue and we have engaged the help of both the database suppliers and the providers of the application.

In order that we can provide a stable platform to fully support your business, regrettably we have temporarily suspended the Secure ePayments service with effect from 15:30 today.

This suspension of service will allow us to undertake further correction and testing. The earliest the system will return will be 9:00am tomorrow.

The Bank's Executive Management are fully aware of the situation and appreciate your continued patience throughout this further period of service disruption.


WHOOPEE DOOO!!! What about our customers?

[Halcyon]

Hi Motor Mouse

Am in the same boat as you, new to HSBC and am voting with my feet. They will never process another card on my site.

[jd1977]

15:30 my arse, it's been a hash job all through the last 24 hours.

5:30 7th April - 9am 9th April compensation claim coming up!

[Motor-Mouse]

I've tacked on PayPal for the time being and alls well here.

How does it compare?

Well it's working for me and the sales are still dropping in so until they get their act together I'm going for plan B.

Plan C is in the pipeline as we type.

I sound a bit bitter, sorry about this everone. I'm not usualy. Only I was the fool that paid the setup fees and sent a member of staff home this morning because there were no orders to process.

[matty262]

We were able to move completely to Protx today and although HSBC still authorise the transactions, payments have been successful so I would urge everyone to get your compensation claims in, get set up on another payment gateway and move. Don;t let HSBC know that you are moving though because the 'loss adjusters' are already hard at work trying reduce the compensation they pay out so if you start threatening to leave, my guess is they'll put you at the bottom of the list. I got some pretty quick responses to my claim with some pretty poor offers of comp. but I mailed them at 9.00am this morning and said I would settle for nothing less than I originally claimed bearing in mind that the system was down yet again. I haven;t heard a word from them all day and my relationship manager seems to be hiding away - not answering the telephone and not responding to messages - they are pathetic.

[stark77]

Hi,

We currently use secure e-payments with the XML API so we take the credit card details on our site as part of our order page and then submit them to HSBC.

Are we able to switch to using Protx in much the same fashion? i.e. POST the transaction details to them using XML and the process a response? If so, I take it this can be done using our existing merchant account with HSBC?

We do less than 1000 transactions a month, so £20 a month would seem to be the only additional cost - but are there any additional costs incurred from the HSBC side if we do this?

Thanks for any help you can give me - receiving the email tonight about how it won't possibly be back up until 9am tomorrow was the last straw. The idea the a technical problem would be mysteriously fixed as soon as regular working hours begin again only seems to imply that when 5pm comes round, everyone goes home and doesn't even attempt to fix anything until morning! I can't believe that a major international bank can be so incredibly poor at implementing a payment system.

[onlineforever]

Given these issues we are facing yet again, I decided today to get Protx set up, signed up and made a bloody mistake!
My hsbc merchant internet account can not be used with epayments and protx, it is catch 22.
Hsbc can create me an additional merchant account for Protx, wait time approx 7-10 days
OR
they can end my secure epayments service so I can stick Protx on, but a 3-4 day wait.
I have emailed them today about my compo claim for the last outage and will see how I get on.
IT HAD BETTER BE WORKING TOMORROW!!!!!!!!

[sculli]

We were able to move completely to Protx today and although HSBC still authorise the transactions, payments have been successful so I would urge everyone to get your compensation claims in, get set up on another payment gateway and move. Don;t let HSBC know that you are moving though because the 'loss adjusters' are already hard at work trying reduce the compensation they pay out so if you start threatening to leave, my guess is they'll put you at the bottom of the list. I got some pretty quick responses to my claim with some pretty poor offers of comp. but I mailed them at 9.00am this morning and said I would settle for nothing less than I originally claimed bearing in mind that the system was down yet again. I haven;t heard a word from them all day and my relationship manager seems to be hiding away - not answering the telephone and not responding to messages - they are pathetic.

Hi Matty
I don't think it makes any difference - HSBC know I am moving to Protx (sadly not until the end of the month as I have to have my site totally redesigned to be able to do this - long story and expensive!) however they settled my first compensation claim without any quibble and I received payment yesterday. Having logged regular complaints against secure epayments for the last two years (mostly ignored) I feel I am due a massive compensation claim, however it is neither wise nor beneficial to be greedy. I will make do with cover for the orders I have lost, a bit of extra admin time and a gesture towards the loss of goodwill, but how long is a piece of string.................!
One little tip to anyone who doesn't know this, head all correspondence to HSBC 'Without Prejudice' this may help your claim along.

[TimmoB]

This email seems to work quite well, I suggest you all give it a go.

MichaelGeoghegan@hsbc.com

We have now moved fully to Protx and taken hsbc off our website.

Good luck all.

Timmo.

[TimmoB]

Given these issues we are facing yet again, I decided today to get Protx set up, signed up and made a bloody mistake!
My hsbc merchant internet account can not be used with epayments and protx, it is catch 22.
Hsbc can create me an additional merchant account for Protx, wait time approx 7-10 days
OR
they can end my secure epayments service so I can stick Protx on, but a 3-4 day wait.
I have emailed them today about my compo claim for the last outage and will see how I get on.
IT HAD BETTER BE WORKING TOMORROW!!!!!!!!

I was told that too by hsbc, but then they told me I could use both on the one merchant account, and that is how it has been operating since the outage 2 weeks ago.

I would check with them again.

[webmate]

Secure-epayments seems to be back online. Let's hope it stays there for a while ;-)

[jd1977]

Seems to be working again now
https://www.secure-epayments.hsbc.com

anyone check records ?

[angelbud]

can login but i cannot make a transaction

[sculli]

Hmm - yes, it is working and at a glance I would say all records are there. I am now too tired to check them all - just hope they are all there in the morning.

[turner2000]

The orders we took in the two hours it was working this afternoon are showing up ok. I don't want to speak to soon but . . . .

[DerekB]

It appears to be functional, but I have not tried reconnecting to it via the website as no notice is posted on the site to say it is functional. I will carry out a couple of tests shortly.

Like others I received the following email from secure epayments at around 5pm today:

"We are extremely sorry to advise that we are still experiencing problems with our Secure ePayments system.

The service was operational from 12:00pm today but we have since encountered a number of further performance issues

As previously advised the issues that we are facing relate to the databases.

This is the first time that we have experienced this particular issue and we have engaged the help of both the database suppliers and the providers of the application.

In order that we can provide a stable platform to fully support your business, regrettably we have temporarily suspended the Secure ePayments service with effect from 15:30 today.

This suspension of service will allow us to undertake further correction and testing. The earliest the system will return will be 9:00am tomorrow.

The Bank's Executive Management are fully aware of the situation and appreciate your continued patience throughout this further period of service disruption."

I replied as follows:

Thank you for your email.

I note that it has taken twenty three and a half hours for you to contact me.

I cannot let the errors in your email go uncorrected.

Firstly, you advised both in statements issued by your call centre personnel yesterday evening, and subsequently when you eventually posted a recording on your answerphone, that a message would be posted on the secure e-payments website when the system is operational.

As a consequence of that, I have been checking the website approximately every ten minutes throughout yesterday evening (until 1.30am this morning), and subsequently from 6.30am this morning. At around 12.15 I was able to log in to the epayments website. No message was posted, and attempts to access any data were met with a failure message.
This continued until approximately 3.30pm this afternoon, when the website appeared to be taken off-line again.

As there was no message to say the site was functional, and as data could not be accessed, you cannot accurately claim that the service was operational for this period.

Secondly, you state "As previously advised the issues that we are facing relate to the databases.". Previously advised to whom exactly? This email is the first communication that we, and others, have received from you.

In your recorded message of last night and this morning, you promised, I recall, to provide full and honest reports of the situation.

Would you kindly explain how the glaring discrepancies in your email correlate with this promise.

I Iook forward to receiving your reply.


I would suggest that everyone responds in a similar vein.

[sculli]

Too risky to leave until the morning. Have just processed the unshipped orders and all seems to be working.

[mkhilton]

Same here all appears to be working including reporting

[turner2000]

I would suggest that everyone responds in a similar vein.

I would do if we had actually received any correspondance from HSBC.

The message stating 9am as the earliest restart time is still playing on the phone.

[stoley]

Epayments admin system working for me but still getting connection error when connecting to Epayments to put payment through.

[stoley]

Epayments admin system working for me but still getting connection error when connecting to Epayments to put payment through.

[sculli]

Same here all appears to be working including reporting

My shipped orders from yesterday aren't showing up on the report. Have printed off the ones I have just shipped in case of any further problems.

[jd1977]

I'm checking the 7th April records before 5.30pm compared to the admin report, and it seems the orders before that time are there, it will be processed tonight.

Unshipped orders is coming back with a blank result?

[DerekB]

All data appears to be present. I have tried reconnecting from our website, but the link is not yet operational. For purposes of compensation claims the website should still be regarded as down.

[onlineforever]

I think it is wworking now- connecting through from our site okay.

[Dwebs-Ltd]

Thing is when everyone starts business tomorrow the system will probably fall over i.e. everyone processing payments from the past couple of days etc

[jd1977]

Well at least it is a sign they do work beyond 5pm hours, probably this push has come from high management towards the IT team to "sort this asap as we are getting claims again"

I am still going to claim between 5:30pm 7th April - 9am 9th April, this 'on and off' nonesense , telling customers to , try again, no wait , come back again, sorry try again is BS.

[Dwebs-Ltd]

Well at least it is a sign they do work beyond 5pm hours, probably this push has come from high management towards the IT team to "sort this asap as we are getting claims again"

I am still going to claim between 5:30pm 7th April - 9am 9th April, this 'on and off' nonesense , telling customers to , try again, no wait , come back again, sorry try again is BS.

Trust me they would work after 5PM if there’s a major problem especially a bank. They have 24x7 IT staff; banks don't stop working after 5pm you know :)

Cash machines are usually updated after 2AM in the morning i.e. system updates etc and there’s plenty of other things that have to be looked after.

[matty262]

Given these issues we are facing yet again, I decided today to get Protx set up, signed up and made a bloody mistake!
My hsbc merchant internet account can not be used with epayments and protx, it is catch 22.
Hsbc can create me an additional merchant account for Protx, wait time approx 7-10 days
OR
they can end my secure epayments service so I can stick Protx on, but a 3-4 day wait.
I have emailed them today about my compo claim for the last outage and will see how I get on.
IT HAD BETTER BE WORKING TOMORROW!!!!!!!!

You can definately use your existing account. i was told the same thing but if you speak to the Terminal department in Leicester they can set it up so that your current HSBC account accepts Protx. Thats how we've done it and it works fine but we are still switching to a new merchant account which is what they ahve told you you need to do. the reason we are moving it anyway is that the new account gives better rates anyway. Call the office in Leicester and ask to speak to the Terminal Connections (I think thats what they are called) Department and they should sort out for you. We had Protx working within 2 days of signing up with them.

[onlineforever]

cool-will try that. what is sthe number for leicester then? i always get through to india.

[envyhairuk]

I just switched to Nochex - best thing i have done :-) I just cant take this risk again with hsbc

[matty262]

cool-will try that. what is sthe number for leicester then? i always get through to india.

Hi onlineforever - sorry I have been out all day. I remembered that I just signed up for Protx and gave them my normal merchant account number and they contacted hsbc as part of the setup so hsbc terminal dept then called me to confirm the request and set it up. They were in Leicester but I have lost the phone number but Protx should sort it for you if you give them you merchant number. Hope this helps.

[peedeesee]

Hi onlineforever - sorry I have been out all day. I remembered that I just signed up for Protx and gave them my normal merchant account number and they contacted hsbc as part of the setup so hsbc terminal dept then called me to confirm the request and set it up. They were in Leicester but I have lost the phone number but Protx should sort it for you if you give them you merchant number. Hope this helps.

Same for us too. Apparently HSBC have been getting rather a lot of these requests from Protx over the past few days...

[benjo132]

Has anybody else been having a few problems trying to claim compensation for this latest outage. Last time it was very straight forward and was settled in 2 days but I have a feeling that it might be more of a struggle this time. I was even sent this email from one person:

Thank you for your recent e-message.

I am formally obliged to advise you of the bank's complaint handling that you can view by logging on to and select 'How to complain' which sets out the relevant details for you.

However, please find below the postal address of the Service Department:

Level 40, HSBC Bank Plc,
8 Canada Square,
London,
Post code E14 5HQ

I quickly replied telling him that I would not be complaining this way and will be taking the same action as I took only 8 days ago. Has anybody had any offers of compensation yet?

Thanks,

Ben

[FunkyBears]

We have just been informed that HSBC secure epayments was unavailable from 11.30 Friday 28 March until 0730 Monday 31st March ! Can anyone advise if they have had any luck with compensation claims?
Thanks
Paul
PS: They have just informed us that it was down again from 6.30pm for another 28 hours ! It's not the loss of immediate sales that is most worrying but the loss of customers. We have worked out the averge lifetime value of one of our customers is between £100 and £150 so everytime this happens it is very expensive in the long term.

[turner2000]

We received a letter from HSBC yesterday informing us of the outage from 28th March to 31st March. It said that claims for reimbursement for direct losses and loss of net profits should be sent to the normal ePayments email address. I don't have the letter with me now, I can't remember the name of who sent it.

[wunpac]

Anyone having problems again?

[divorce-online]

Yes it appears to be down again.

[decworld]

This is a best reason to take payment details and process them off line later, you have the CC information and no third party can cause a payment to be lost.

In addition you can see what is been declined and why, how many people measure how many people exit the payment system when it complains about a card error pre attempted auth.......

[sysops]

This is a best reason to take payment details and process them off line later, you have the CC information and no third party can cause a payment to be lost.

You are joking, right?

[decworld]

No,

I am been quite serious.

We process over 100 order a day.

All card details are taken with mals-e, downloaded a csv spreadsheet and then fed in bulk to the PDQ terminal.

Works very well and not reliant on any third party gateway.

And cards can be re tried and customer mistakes corrected, ie postcode typos etc.

[sysops]

ROFL!!

Would be grateful for your site address, to ensure my credit card number doesn't go anywhere near it.

[decworld]

What, because your card details are safer in HSBC database than Mals-e. Mals-e having almost no downtime in the last 9 years where as HSBC cant even take the data in the first place.

I really do not see your point.

[debbidoo]

No,

I am been quite serious.

We process over 100 order a day.

All card details are taken with mals-e, downloaded a csv spreadsheet and then fed in bulk to the PDQ terminal.

Works very well and not reliant on any third party gateway.

And cards can be re tried and customer mistakes corrected, ie postcode typos etc.


Wow... have you any idea how insecure that is? What do you do with the CSV when you've finished using it? Are you PCIDSS compliant?

I really hope for your sake that none of your customers find this thread...

[sysops]

When I place an order online, carefully ensuring I type my CC details into an https form, the last thing I want is for these details to then be downloaded to someone's PC and manually typed into a terminal.

That is what is known in the industry as a GREAT BIG GAPING SECURITY HOLE.

[decworld]

SSL to download the data.

Deleted once used.

Class 4 PCI compliant.

Merchant services (barclays) fully happy with procedures.

I think you should consider more than 140,000 merchants use mals-e (a lot do use gateways I accept) for many years.

[sysops]

Class 4 PCI compliant


As verified by whom?



I think you should consider more than 140,000 merchants use mals-e (a lot do use gateways I accept) for many years.

A lot of people do a lot of very stupid things every single day - the fact that a lot of people do something should never be used as justification, or as proof that this thing is ok.

[decworld]

Class 4 is self cert.

Its not proof its OK, its proof it works for many business for many years.

If people want to reply on third party gateways thats up to them, the point I was/am making is that other options exist, have done so for many years.

Referring to the system that stores CC details as "someones pc" is quite ridiculous.

I suppose at protx they store details one "someones server".

[debbidoo]

When you download the CSV onto your computer, does it contain credit card numbers and CV2 numbers as well as personal data like names and addresses? That's what my concern would be... if that CSV was on your computer and some spyware found its way onto your computer, that could be seriously dodgy...

[sysops]

Class 4 is self cert.

Its not proof its OK, its proof it works for many business for many years.

Even if your security is sufficient (which I don't believe for a second), surely you see that this is a very inefficient way of processing payments.

You have to employ someone full time to key in payments (and trust this person with all these CC numbers).

And what about failed payments? Around 12% of our payments fail on first attempt (this is after Luhn and sanity checks). Presumably you contact these people individually to check card details?

Crazy...

[Cleveregg]

Glad I found this thread, I am getting a call from HSBC ePayment team today to complete sign up...I think I tell them i'm going elsewhere!!!

[onlineforever]

fingers crossed - it has been fine since the problems the other month.

[kevin555]

When it comes to taking payments I don't think you should be keeping your fingers crossed - doesn't smack of much confidence.

[AndYc42]

Its not just HSBC though every gateway suffers from downtime. Its a fact that payment gateways will suffer from downtime every now and then.

If you can offer alternative payment methods on your site (even pay by cheque) then you will have less chance of loosing sales next time it goes belly up.

You may be interested in our shopping cart system www.absoluteshoppingcart.co.uk (http://www.absoluteshoppingcart.co.uk) , it allows you to use multiple gateways to at the same time and they can be switched on and off at the click of a mouse. A number of our customers use this feature, and find it very useful.

Lots of people now have Paypal and Google accounts, if you offer these payment solutions as well, you can actually gain sales because your customers find them a more convenient way to pay.

[CLSLTD]

OK ePayments has been down for us all day - we can't login and its not processing any payments. Usual story - they are denying any problem saying its us... is anyone else getting this?

[sysops]

So you're the one still using ePayments...

Seriously, given the recent track record, why are you still on that system?

[Dwebs-Ltd]

Its not always super easy to move payment systems it can also be costly depending on implementation

[sysops]

Its not always super easy to move payment systems it can also be costly depending on implementation

No, it's pretty easy, and a hell of a lot less costly than losing a day's business...

[CLSLTD]

Thanks for those helpful suggestions...
To anyone 'still using ePayments': Are you having any problems? ePayments deny there is any issue - but we cannot find anything wrong our end... I just need to know if we need to spend the weekend looking to our code - or writing angry emails to ePayments...

[CLSLTD]

Finally resolved our latest ePayments problem - they had accidentally deleted our 'master account' which suspended our other account. However it took 4 days of us insisting it was not a problem our end before they finally decided to take a look and discovered it was theirs... hopeless.

[kevin555]

You admit they're hopeless and still you do business with them - at what point will you move?

[CLSLTD]

We been trying to move to WorldPays Select Invisible, which looks really good, but they are really strict on their PCI compliance and its taken 3 months to get McAfee certificate. So we on our way away from HSBC this week! Finally.