How to make a payment API safe to retry: idempotency keys, done properly in Go

The first time I watched a customer get charged twice, nothing had gone wrong. No bug in the charge logic, no bad input, no exception. My code charged the card exactly the way I wrote it to — and then charged it again, cleanly, for the same thing. The double charge didn't come from my code failing. It came from my code succeeding, twice, because the network between my server and the client is less reliable than the code inside my function.

This article is a build-along. By the end you'll have a payment endpoint in Go that is safe to retry: it survives lost responses, two identical requests arriving at the same instant, mismatched retries, and a crash in the middle of a charge. I'll build it in stages, and at each stage I'll break the previous version on purpose, because every hole in this problem is invisible until you see the exact sequence of events that exposes it.

The failure we're solving#

Start with the endpoint most of us write first.

func (s *Server) CreateCharge(w http.ResponseWriter, r *http.Request) {
    var req ChargeRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        http.Error(w, "bad request", http.StatusBadRequest)
        return
    }
    charge, err := s.rail.Charge(req.CustomerID, req.AmountCents, req.Currency)
    if err != nil {
        http.Error(w, "charge failed", http.StatusBadGateway)
        return
    }
    s.db.SaveCharge(charge)
    json.NewEncoder(w).Encode(charge)
}

Every line is correct. It breaks anyway, on this exact sequence:

1. Client sends the request.
2. Server charges the card. The money moves.
3. The connection drops before the response reaches the client.
4. The client, having received nothing, cannot tell whether the charge happened. So it retries.
5. The server charges the card a second time.

Here is that sequence — the failure is invisible until you see the timing:

The root cause, stated precisely: a retry is byte-for-byte identical to a first attempt, so the server cannot distinguish them unless the client gives it a way to. Nothing in that endpoint lets the server tell "new charge" apart from "the charge you already did and the client didn't hear about."

The mechanism: name the operation#

An operation is idempotent if performing it many times leaves the same state as performing it once. We make a create-operation idempotent by attaching a name to it — a client-generated unique value, the idempotency key — that is identical on the first attempt and every retry, and different for every distinct operation.

The key names the operation, not the HTTP request. "Charge customer 42, 2000 paise, INR, for order 4821" gets one name; every attempt to make that one thing happen carries it.

Client side, it's a header:

key := uuid.NewString() // v4 UUID: enough entropy that two clients never collide
req.Header.Set("Idempotency-Key", key)

Two hard constraints on the key, both of which prevent real bugs:

• Generate it with real randomness (a v4 UUID). If two independent operations can collide on a key, one of them gets the other's result.
• Put no meaning inside it. Never use an email, a user ID, or anything identifying. The key is opaque. Encoding data into it just leaks that data into every log line and index that touches the key.

One operation, one name — every attempt carries the same header; a different charge gets a different key:

Version 1: look, then act (and why it's not enough)#

The obvious first implementation:

func (s *Server) CreateCharge(w http.ResponseWriter, r *http.Request) {
    key := r.Header.Get("Idempotency-Key")
    if key == "" {
        http.Error(w, "missing idempotency key", http.StatusBadRequest)
        return
    }
    if saved, found := s.db.LookupCharge(key); found {
        json.NewEncoder(w).Encode(saved) // replay the first result
        return
    }
    var req ChargeRequest
    json.NewDecoder(r.Body).Decode(&req)
    charge, _ := s.rail.Charge(req.CustomerID, req.AmountCents, req.Currency)
    s.db.SaveCharge(key, charge)
    json.NewEncoder(w).Encode(charge)
}

This fixes the sequential retry: first attempt saves under the key, the lost-response retry finds it and replays. Good. But it has a race that production will find within a day.

The client didn't send its retry after the first request finished — it sent it because it gave up waiting, while the first request may still be running. So two requests with the same key run concurrently. Here is the race the lookup-then-write pattern leaves open:

Both looked before either wrote. Both charged. This is the concurrent duplicate, and it's a different failure from the sequential retry: the check-then-act pattern has a window between the check and the act, and the second request fits inside it.

Version 2: claim the key atomically#

The fix is to stop asking whether the key exists and instead try to claim it, letting the database's uniqueness constraint be the referee. Exactly one racing request wins the insert; the loser learns someone else owns the key.

The schema — this is the core data structure, worth getting exactly right:

CREATE TABLE idempotency_keys (
    key           TEXT PRIMARY KEY,        -- the claim itself is the lock
    request_hash  TEXT NOT NULL,           -- fingerprint of the original request
    status_code   INT,                     -- filled in once the work finishes
    response_body BYTEA,                   -- the exact bytes to replay on retry
    state         TEXT NOT NULL,           -- 'in_progress' | 'completed'
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

The claim, as a single atomic statement. ON CONFLICT DO NOTHING means the insert either creates the row (we won) or affects zero rows (someone else owns it):

INSERT INTO idempotency_keys (key, request_hash, state)
VALUES ($1, $2, 'in_progress')
ON CONFLICT (key) DO NOTHING;

In Go, the endpoint now has three cases to handle — this is the heart of the whole implementation:

func (s *Server) CreateCharge(w http.ResponseWriter, r *http.Request) {
    key := r.Header.Get("Idempotency-Key")
    if key == "" {
        http.Error(w, "missing idempotency key", http.StatusBadRequest)
        return
    }
    var req ChargeRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        http.Error(w, "bad request", http.StatusBadRequest)
        return
    }
 
    won, err := s.db.ClaimKey(r.Context(), key, req.Fingerprint())
    if err != nil {
        http.Error(w, "internal error", http.StatusInternalServerError)
        return
    }
 
    if !won {
        // The key already exists. Fetch its current state.
        rec, _ := s.db.LoadKey(r.Context(), key)
 
        if rec.RequestHash != req.Fingerprint() {
            // Same key, different request. Refuse — see Version 3.
            http.Error(w, "idempotency key reused with different parameters",
                http.StatusUnprocessableEntity)
            return
        }
        if rec.State == "completed" {
            // Finished earlier. Replay the exact original response, status and all.
            w.WriteHeader(rec.StatusCode)
            w.Write(rec.ResponseBody)
            return
        }
        // Claimed but not finished: a request with this key is still running.
        http.Error(w, "a request with this key is in progress",
            http.StatusConflict) // 409 — client should back off and retry
        return
    }
 
    // We won the claim. We are the only worker for this key.
    charge, err := s.rail.Charge(req.CustomerID, req.AmountCents, req.Currency)
    if err != nil {
        body, _ := json.Marshal(map[string]string{"error": "charge failed"})
        s.db.CompleteKey(r.Context(), key, http.StatusBadGateway, body)
        w.WriteHeader(http.StatusBadGateway)
        w.Write(body)
        return
    }
    body, _ := json.Marshal(charge)
    s.db.CompleteKey(r.Context(), key, http.StatusOK, body)
    w.WriteHeader(http.StatusOK)
    w.Write(body)
}

Three things here are deliberate and matter for correctness:

• The PRIMARY KEY on key is the concurrency control. I wrote no locking logic; the database rejects the second insert. Reach for the tool that's actually built for this instead of hand-rolling mutexes.
• There's a third state, in_progress. A retry that lands while the first attempt is still running must not replay a result that doesn't exist yet and must not start a second charge. It gets a 409 and comes back later. This is why a mature payments API has a dedicated "key currently in use" error — it's this exact branch.
• I replay the stored response even on failure. I store the status and body whether the charge succeeded or returned an error, so a retry gets the same outcome the first attempt did, not a fresh roll of the dice.

Once a key is claimed, every later request with that key lands in one of three places:

Version 3: the same name must mean the same thing#

There's a request_hash in the schema and a fingerprint check in the code above. Here's the bug it prevents.

A key names an operation. So what should happen when a client sends a key it already used, but with a different body — "charge 2000" the first time, "charge 5000" under the same key later? Replaying the first result silently swallows the second, real request. Running the second means the key protected nothing. Both outcomes let one name point at two operations.

So on every reused key, I verify the incoming request matches the one I first stored under that key, by hashing the meaningful fields:

func (req ChargeRequest) Fingerprint() string {
    h := sha256.New()
    fmt.Fprintf(h, "%s|%d|%s", req.CustomerID, req.AmountCents, req.Currency)
    return hex.EncodeToString(h.Sum(nil))
}

Mismatch → reject with 422 (as in the Version 2 code) instead of guessing.

Version 4: when you save decides what a retry can see#

A subtle sequencing rule: claim the key at the moment real work begins — not a step earlier.

If a request fails validation (malformed body, missing field) before any work happens, don't burn the key on it. The client should fix the typo and retry with the same key successfully. The same logic applies to the checks that run ahead of your endpoint — authentication and rate limiting sit in front of the idempotency layer. A request rejected for a missing API key (401) or for arriving too fast (429) hasn't started work, so its key must remain reusable once the client fixes auth or slows down.

Concretely, that's why the Version 2 code decodes and validates the body before calling ClaimKey, and claims only when it's about to actually charge.

Version 5: surviving a crash in the middle#

The hardest case. Charging isn't atomic — it's reserve-locally, call-the-rail, record-the-result. If the server dies between "call the rail" and "record the result," the money moved but the database doesn't know it, and a naive retry charges again.

You can't wrap the external call in a database transaction; no ROLLBACK of yours reaches the card network. So instead: break the operation into stages, and after each stage write a marker in the same transaction that did that stage's work. Stage and marker commit together or not at all. A retry reads the marker and resumes from where the last attempt stopped.

func (s *Server) processCharge(ctx context.Context, key string, req ChargeRequest) (*Charge, error) {
    state, err := s.db.LoadState(ctx, key)
    if err != nil {
        return nil, err
    }
 
    // Stage 1: reserve intent locally, before any money moves.
    // ReserveAndMark writes the reservation row AND advances the phase in one tx.
    if state.Phase == PhaseStarted {
        if err := s.db.ReserveAndMark(ctx, key, req, PhaseReserved); err != nil {
            return nil, err
        }
        state.Phase = PhaseReserved
    }
 
    // Stage 2: the external call — the part that can't be rolled back.
    if state.Phase == PhaseReserved {
        // Pass OUR key to the rail too, so the rail can dedupe on its side.
        charge, err := s.rail.ChargeWithKey(key, req.CustomerID, req.AmountCents, req.Currency)
        if err != nil {
            return nil, err
        }
        // Crash between the line above and the line below?
        // The retry re-enters at PhaseReserved and asks the rail whether a
        // charge under this key already exists — it does — so it adopts it
        // instead of charging again.
        if err := s.db.RecordResultAndMark(ctx, key, charge, PhaseCompleted); err != nil {
            return nil, err
        }
        state.Charge = charge
    }
    return state.Charge, nil
}

The ordering in Stage 2 is the whole point: write local intent first, make the outside call second, never the reverse. Because intent is recorded before the call, a retry that lands mid-flight knows a charge might have gone out under this key, so it asks the rail (passing the same key) rather than firing blind. If the rail has it, adopt it; if not, make it. One charge either way.

This is the same discipline as reserving funds before you answer a request instead of after: decide what's true and write it down before the irreversible step, so that if you fall over, you fall over somewhere you've already reasoned about.

Version 6: retry like a good citizen (client side)#

Idempotency makes retries safe; it doesn't make them kind. Retrying instantly and repeatedly turns a momentarily slow server into a self-reinforcing outage, because every client retries in the same instant. So: wait, wait longer each time, and add randomness so clients don't synchronize.

func chargeWithRetry(ctx context.Context, c *PaymentClient, req ChargeRequest) (*Charge, error) {
    key := uuid.NewString() // ONE key for the whole operation, reused on every attempt
 
    backoff := 200 * time.Millisecond
    for attempt := 0; attempt < 5; attempt++ {
        charge, err := c.Charge(ctx, key, req) // same key every time
        if err == nil {
            return charge, nil
        }
        if !isRetryable(err) { // declined card, bad request → don't retry
            return nil, err
        }
        jitter := time.Duration(rand.Int63n(int64(backoff / 2)))
        time.Sleep(backoff + jitter)
        backoff *= 2 // 200ms → 400ms → 800ms → ...
    }
    return nil, fmt.Errorf("charge failed after retries")
}

Two independent properties, and you need both: the same key every attempt (so the server dedupes), and exponential backoff with jitter (so a struggling server recovers instead of being buried). A key without backoff protects your data and endangers your server; backoff without a key protects your server and endangers your data.

And retry only what might differ next time — timeouts, dropped connections. A declined card won't change on retry; asking again just wastes the attempt.

Two smaller decisions#

Key lifetime. Expire keys after ~24h and reap them. A retry that's coming is coming within minutes. But if a reaped key can ever be reused, make that a deliberate choice — a "new" charge under a recycled name is exactly the bug this article exists to prevent.

Key scope. Scope keys to the endpoint and the caller, not globally, so the same value used by different callers for different operations can't collide. Unique within the place it's used is all it needs to be.

Reference checklist#

Hold your own implementation against this:

• Client generates one v4-UUID key per operation, sends it on every retry.
• Server rejects requests with no key on create/money endpoints.
• Body is validated before the key is claimed.
• Key is claimed with a unique-constraint insert (ON CONFLICT DO NOTHING), not a look-then-write.
• Three states handled: new → work; completed → replay stored response; in-progress → 409.
• Stored response includes status code and body, and is replayed on failure too, not just success.
• Request fingerprint stored and checked; reused key + different body → 422.
• Front-door rejections (auth/rate-limit/validation) do not consume the key.
• Non-rollbackable steps: local intent committed before the external call; external provider deduped on the same key; retries resume from the last marker.
• Client uses exponential backoff with jitter, same key throughout, transient-only retries.
• Keys expire and are scoped to endpoint + caller.

Closing#

None of this makes failure go away. Networks still drop, servers still restart at the worst moment. What changes is that every failure now lands somewhere you've already handled: a lost response on a key that replays, two simultaneous requests on a claim only one can win, a mid-charge crash on a marker that says where to resume, a struggling server on backoff instead of a stampede.

The whole shift is smaller than it sounds — you stop treating each request as a fresh event and start treating it as a named operation that may arrive more than once. The first time I saw a customer charged twice, it was because my code assumed every request was the first one. Once every operation has a name and every retry lands on a system that remembers what it already did, the double charge doesn't happen. Not because failure stopped — because failure became survivable.