Payment integrations — what breaks in production and why

Payment gateways are straightforward to integrate and difficult to operate. The problems appear months later, and almost none of them are in the happy path.

Integrating a payment gateway is not hard. The documentation is good, the SDKs work, and you can usually take a test card on day two.

Operating a payment integration is a different discipline entirely. The problems arrive months after launch, they arrive in small numbers, and they are all in the paths nobody tested.

The order of operations that causes the worst bugs

Here is the single most common structural mistake:

  1. Charge the card.
  2. Create the order.

It works perfectly until the process dies between the two — a deploy, an app pool recycle, a timeout, a transient database error. Now the customer has been charged and has no order. They have a bank statement proving they paid and your system has no record of it. That is a support call that starts badly and gets worse.

The correct order is to create the order in a pending state first, then charge, then mark it paid. If the charge never completes, you have a pending order you can reconcile. If the process dies after the charge, you have a record to reconcile against. You will still have failures — you are coordinating two systems that cannot share a transaction — but they become detectable rather than silent.

Webhooks are not a convenience

Many teams treat the gateway's webhook as a nice-to-have because the redirect back to the site already tells them the payment succeeded.

The redirect is the unreliable channel. The customer closes the tab. Their phone loses signal on the confirmation page. The bank's 3-D Secure step takes them somewhere unexpected. In all of those cases the money moved and your application never found out.

The webhook is the source of truth. The redirect is a user-experience detail. Build it in that order, or you will be reconciling by hand.

Three things the webhook handler must get right:

  • Verify the signature. An unauthenticated endpoint that marks orders as paid is exactly as dangerous as it sounds.
  • Be idempotent. Gateways retry. You will receive the same event more than once, and processing it twice must not charge twice, email twice or ship twice.
  • Return 200 quickly. Acknowledge, queue the work, process it out of band. A slow handler causes retries, which causes duplicates, which causes the problem above.

Money is not a floating point number

It should not need saying, and yet. Use decimal end to end, store the currency alongside the amount, and never let a rounding decision happen implicitly.

Multi-currency adds a specific trap: the exchange rate at the moment of the transaction must be stored with the transaction. If you recalculate historical totals using today's rate, your reports will disagree with your accounts, and the accounts will be right.

The failures that are not really failures

A meaningful share of declines are not technical faults. Insufficient funds, a card expiring, the customer's bank declining a foreign transaction, a 3-D Secure challenge abandoned halfway.

Logging all of these as errors means the genuine failures — a misconfigured endpoint, an expired API credential, a TLS problem — get buried in noise. Separate "the payment was declined", which is a normal business outcome, from "we could not talk to the gateway", which is an incident. Only the second one should wake anyone up.

Wallets bring platform problems, not payment problems

Apple Pay and Google Pay are pleasant to use and awkward to operate, because the failures are environmental rather than logical.

Domain verification files that must be served from an exact path. Certificates with expiry dates nobody is tracking. Behaviour that differs between browsers and between a native app and mobile Safari. Sandbox environments that do not quite mirror production.

None of it is difficult. All of it needs to be written down somewhere other than one developer's memory, because the day a certificate expires is the day you find out nobody knew it existed.

What to build alongside the integration

The integration itself is perhaps half the work. The rest is what lets you operate it:

  • A reconciliation report. Gateway transactions against your orders, run daily, differences surfaced. This finds problems before customers do.
  • A payment audit log that records every gateway interaction, including failures, with enough context to answer "what happened to this specific payment" months later.
  • A staging environment that genuinely talks to the gateway's sandbox, so you can test the failure paths rather than hoping.
  • Credential expiry in a calendar, owned by a person.

Payment code is some of the least forgiving code in a business application: it involves real money, an external party you do not control, and customers who notice immediately. The teams that operate it calmly are not the ones who integrated most cleverly — they are the ones who assumed it would go wrong and built the tools to find out when it did.

All insights Talk to us about this

Let's work together

Dealing with this yourself?

If this is close to a problem you are living with, we would be glad to hear the details and tell you what we would look at first.