Idempotency and Retries Through Proxies: How a Repeated Request Doesn't Create a Duplicate Order
Table of contents
- Introduction: one click, two orders, and a tunnel where responses get lost
- Fundamentals: what is idempotency and why is a repeated request dangerous?
- Deep dive: the server contract and the anatomy of a safe retry
- Method 1: generating and passing the idempotency key on the client side
- Method 2: server-side implementation of a key store
- Method 3: a retry protocol through a proxy that doesn't break idempotency
- Method 4: when the server doesn't support idempotency keys
- Common mistakes: how teams create duplicates themselves
- Tools and resources for implementation
- Case studies and results
- Faq: hard questions about idempotency and retries
- Conclusion: a contract instead of hope
Introduction: One Click, Two Orders, and a Tunnel Where Responses Get Lost
Picture an integration that's been running for years. Your service sends orders to an external system, the traffic flows through a pool of mobile proxies to distribute requests across regions and stay under the rate limits of any single address. Everything's stable. And then accounting drops a report on your desk: over the month, the partner got forty-seven twin orders. Same items, same amounts, created three to four seconds apart. Nobody changed anything. What happened?
The most ordinary thing happened: the connection through the proxy broke on the way back. The server received the request, created the order, sent a response, but the response never reached the client. The client honestly decided the request had failed and retried it. The second request also went through. That's how a duplicate is born, and no retry policy on its own protects you from it.
Let's draw a line right away. We've covered elsewhere how to properly retry requests after a 429 error, how long to wait between attempts, and how to build exponential backoff: that's the topic of retry policy. Today the conversation is different. We're talking about the contract between client and server that makes a retry safe or dangerous regardless of how you actually execute it. That contract is called idempotency, and its main tool is the idempotency key.
Why is this topic especially acute when working through proxies? Because a proxy adds another segment to the chain where the response can get lost. Mobile proxies also change the external IP on rotation, and the server may well see the second request coming from a different address. If your duplicate protection relies on session, IP, or a time window, it falls apart. If it relies on an idempotency key, it doesn't care where the retry came from.
In this guide, you'll learn how idempotency works at the protocol and business-logic levels, how to generate and store keys, how to design a server-side key store that withstands concurrent retries, what to do if the external API doesn't support keys, and what mistakes even experienced teams stumble on. We'll walk through case studies with numbers, put together checklists, and answer uncomfortable questions. This material is aimed at integration developers, architects, and tech leads who work with orders, payments, and any operations where "twice" means "bad."
Fundamentals: What Is Idempotency and Why Is a Repeated Request Dangerous?
A plain-English definition
An operation is called idempotent if performing it multiple times produces the same result as performing it once. Pressing an elevator call button ten times is idempotent: the elevator arrives once. Adding another unit of a product to a cart is not idempotent: each press changes the state. Transferring money is definitely not idempotent, unless the system is specifically designed so that repeating a transfer with the same identifier does nothing.
In HTTP, there are methods that are idempotent by specification: GET, PUT, DELETE, HEAD. And there's POST, which by default is not considered idempotent. POST is most often used to create orders, process payments, and send messages. In other words, the riskiest operation lives in the least idempotent method. Coincidence? More like a historical pattern you just have to work with.
Three outcomes of any request
When a client sends a request and gets no response, it ends up in a state of uncertainty. There are exactly three possible scenarios:
- The request never reached the server. Retrying is safe: nothing happened.
- The request reached the server, the server performed the operation, and the response was lost. Retrying is dangerous: the operation will run a second time.
- The request reached the server, the server started processing but hasn't finished yet. Retrying is doubly dangerous: two operations will run in parallel.
The core problem is that the client can't distinguish between these scenarios. A timeout looks the same in all three cases. So does a broken connection. This leads to a fundamental principle of distributed systems: reliable delivery is only possible as "at least once" (at-least-once). The "exactly once" (exactly-once) guarantee can't be obtained at the transport level; it can only be emulated at the application level by making retries harmless. Idempotency is that emulation method.
How a proxy lengthens the chain of uncertainty
Without a proxy, there's one network segment between client and server. With a proxy, there are two: client — proxy and proxy — server. The response can get lost on either one. And if it gets lost on the first segment, the proxy has already received the response, the server has already completed the operation, and the client will never find out.
Mobile proxies add their own specifics. They rotate IP on a schedule or on demand, switch between base stations, and experience natural fluctuations in radio channel quality. Each of these events can break the TCP connection at an arbitrary moment. What's more, the break most often happens not during transmission of the request body (it's small and flies out quickly), but while waiting for the response, when the connection is just being held open. That's exactly the moment the server is processing your order. That's exactly the moment when a break gives the maximum chance of a duplicate.
What is an idempotency key
An idempotency key (Idempotency-Key) is a unique identifier for a business operation that the client generates once and sends with every attempt to perform that operation. The server remembers the key and the result. On a repeated request with the same key, the server doesn't perform the operation again but returns the saved result. The client can retry as many times as it wants: the order will be created once.
Pay attention to the wording: the key identifies an operation, not an HTTP request. This distinction becomes central later when we dig into implementation mistakes.
Deep Dive: The Server Contract and the Anatomy of a Safe Retry
The contract as an agreement on behavior
Idempotency isn't a header or a library. It's a contract in which the server promises the client specific behavior on retries, and the client promises to follow the rules for forming keys. A good contract answers the following questions:
- Which entity is the unit of idempotency: order, payment, order line, batch?
- How does the client pass the key: header, body field, URL component?
- What happens if the key matches but the request body is different?
- What happens if the first request is still running and a second one with the same key arrives?
- How long does the server remember the key?
- What gets returned on a retry: the exact same response byte-for-byte, or the current state of the entity?
- How is a retry distinguished from a new request in logs and metrics?
If even one of these questions has no answer in the API documentation, the contract is incomplete, and in uncertain situations behavior will depend on implementation, version, and the phase of the moon.
Three states of a key
On the server, every key has a lifecycle of three states. Understanding this cycle explains ninety percent of duplicate problems.
- Absent. The key is being seen for the first time. The server must atomically reserve it and start processing.
- In-progress. The first request is being processed. A second request with the same key must not trigger a second processing. Options: return 409 Conflict indicating the client should wait, block until the first completes and return its result, or return 202 Accepted with a link to the status.
- Completed. The result is saved. A retry returns the saved result without side effects.
The most common architectural mistake is skipping the middle state. The developer checks "is the key in the database," doesn't find it, starts processing, and a parallel retry at the same moment does the same thing. Two threads, two checks, two orders. The check and the reservation must be a single atomic operation: insert with a unique index, conditional write to Redis, transaction with a lock.
Request fingerprint
The key identifies the operation, but an operation with what content? If the client, due to a bug, sent key K with an amount of 1,000 rubles, then retried with the same key and an amount of 10,000, what should the server do? Silently return the result of the first operation — dangerous: the client will think it paid 10,000. Perform the second — that violates idempotency.
The correct answer: the server stores a request fingerprint — a hash of the normalized body and significant headers. On a retry with the same key but a different fingerprint, it returns a 422 Unprocessable Content or 400 with an explicit code like idempotency_key_reused_with_different_payload. This protects against client logic errors and against forged retries.
Storing the response: snapshot or current state
There are two approaches to what gets returned on a retry. A snapshot — the server saves the status, headers, and body of the original response and returns them unchanged. This is strictly idempotent and predictable. The current state — the server finds the created entity by key and returns its current representation. This is more convenient if the order changed status between attempts, but less predictable.
In payment systems, the snapshot dominates: the client must get exactly the response from the first attempt. In order management systems, a hybrid is often used: a snapshot for error codes, the current state for success. The main thing is that the choice is fixed in the contract.
What a proxy changes at the contract level
Technically, a proxy doesn't participate in the idempotency contract: it's transparent to headers and body. But it changes probabilities. Over a direct connection, losing a response when the operation succeeded is a rarity, a fraction of a percent of a percent. Through a chain with a mobile proxy and rotation, this event becomes regular. We've observed integrations where, with five-minute IP rotation, roughly 0.3–0.8% of requests ended in a connection break during the wait-for-response phase. At ten thousand orders a day, that's thirty to eighty potential duplicates daily. Without an idempotency key, each one becomes a real duplicate.
There's a second subtlety. Many servers build "repeated request from a different IP" checks into their anti-fraud logic. When a mobile proxy rotates, the retry almost always comes from a new address. If the server treats this as suspicious activity and rejects the retry without consulting the key store, the client won't get the saved result and will remain in uncertainty. A good contract checks the key before any network-based heuristics.
Method 1: Generating and Passing the Idempotency Key on the Client Side
Where the key is born
The main rule: the key is generated once per business operation, before the first send attempt, and saved along with the intent to perform the operation. If the key is created inside the HTTP client on every call, the retry will get a new key, and all protection is nullified. This is such a common mistake that we'll call it out separately in the antipatterns section, but we'll repeat it here: the key lives at the level of the order service, not at the transport level.
The practical sequence looks like this:
- A user or internal process initiates order creation.
- The service creates a local "order intent" record with pending status and generates a key.
- The key and payload are saved to the local database in a single transaction.
- Only after the transaction commits does sending to the external API through the proxy start.
- All retries read the key from the local record rather than generating a new one.
- After receiving the response, the local record is updated with the result and the external order ID.
This order guarantees that even a restart of your service in the middle of retries won't lead to a key change: it sits in the database.
Key format
The key must be globally unique and unpredictable for third parties. Working options:
- UUID version 4 — classic, 122 bits of randomness, works everywhere.
- UUID version 7 or ULID — they include a timestamp in the high bits, which makes them convenient for sorting and indexing on the server database. In 2026, this is already the standard choice for new systems.
- A deterministic key derived from a natural identifier — for example, a hash of the cart ID, user ID, and cart version. Useful when the client itself can lose state and regenerate the key from the same data.
What to avoid: sequential numbers (easy to guess someone else's key and get their response), short strings, keys that depend only on time with low precision.
How to pass it
The de facto standard is the Idempotency-Key header. The IETF specification draft fixes exactly this name, and most modern APIs adhere to it. An example call through a proxy:
curl -x http://user:pass@proxy.example:8080 -H "Idempotency-Key: 018f3a2c-7b1e-7c3d-9a4e-1f2b3c4d5e6f" -H "Content-Type: application/json" -d '{"cart_id":"c-9921","amount":149000}' https://api.partner.example/v1/orders
Some APIs accept the key in the request body (a field like client_reference_id, external_id, request_id). That's also a working option, but with one caveat: if the server uses a body field as the key, it must be part of a unique index, otherwise it's just a label, not protection.
Key scope
The key must be unique within one client account and one endpoint. A good server composes the internal storage key as a combination of account ID, method, path, and the passed key. This protects against collisions between different clients and against accidentally reusing a key for a different operation.
Client-side checklist
- The key is generated before the first attempt and stored in the database along with the intent.
- All retries use the same key.
- The request body on retry matches the first attempt byte-for-byte.
- The client can handle 409 (operation in progress) as a signal to wait and retry, not as an error.
- The client can handle 422 with a fingerprint mismatch code as a logic error requiring intervention.
- The client logs the key in every record related to the operation, so duplicates can be investigated.
- The client doesn't reset the key when switching proxies, rotating IPs, or failing over to a backup pool.
Method 2: Server-Side Implementation of a Key Store
If you own the server that your clients or partners call through a proxy, this section is for you. If you're only the client, it will help you assess the quality of someone else's implementation from the documentation and behavior.
Storage schema
The minimal table or structure for keys contains the following fields:
- scope_key — composite key: account, method, path, passed Idempotency-Key. Unique index.
- fingerprint — hash of the normalized request body.
- state — one of: in_progress, completed, failed.
- response_status, response_headers, response_body — response snapshot, populated on completion.
- resource_id — ID of the created entity, if applicable.
- locked_at, completed_at, expires_at — timestamps for handling stuck operations and cleanup.
Atomic reservation
The heart of the implementation is an operation that, in a single action, either creates a record in the in_progress state or reports that the record already exists. In a relational database, this is an insert with conflict handling on a unique index. In Redis, it's the SET command with NX and EX flags, which writes the value only if the key doesn't exist and sets the TTL immediately. No "first SELECT, then INSERT": a parallel retry will slip in between them.
Pseudocode at the logic level:
- Compute scope_key and fingerprint.
- Try to insert a record with state = in_progress. If the insert succeeds, proceed to performing the operation.
- On conflict: read the existing record.
- If the fingerprint doesn't match — return 422 with a mismatch code.
- If state = in_progress — return 409 with a Retry-After header or wait for completion (within a reasonable timeout).
- If state = completed — return the saved response snapshot with a header marking the retry, e.g.,
Idempotent-Replayed: true.
Executing and committing the result
The second trap hides here. The operation completed, the order was created, but the process crashed before writing the result to the key store. The record stays in the in_progress state forever. A retry gets 409, then 409 again, and so on ad infinitum.
There are three approaches to solving this:
- Single transaction. Creating the order and updating the key record happen in one database transaction. The most reliable option if the order and keys live in the same database.
- Key as part of the entity. The idempotency key is stored directly in the orders table with a unique index. Then the order itself is the key record; no separate table is needed. Simple and reliable for cases with one entity per operation.
- Recovery of stuck records. A background process finds records in in_progress state older than a threshold (say, 60 seconds), checks whether the operation was actually performed (by resource_id or by searching for the entity), and either completes the commit or moves the record to failed, allowing a retry.
Handling first-attempt errors
What to store if the first attempt ended in an error? It depends on the error type. Validation errors (400, 422) are deterministic: a retry with the same body will produce the same result, so they can be cached. Infrastructure errors (500, 503, database timeout) are not deterministic: if they're cached, the client can never successfully perform the operation. The correct behavior: on an infrastructure error, release the key (delete the record or move it to failed with the ability to re-acquire), so the next retry tries again.
Key TTL
Keys can't be stored forever: the table will grow. But too short a TTL is dangerous too: if the client retries half an hour after a network failure, the key must still be alive. The practice of large payment providers converges on 24 hours as a minimum. For e-commerce orders, 24–72 hours is reasonable. For operations the client might retry after prolonged outages (say, catalog syncs once a day), the TTL should be at least twice the sync interval.
Observability
Every retry should leave a trace. Metrics worth collecting: number of retries hitting completed (harmless retries), number of 409s (concurrent retries), number of 422s (fingerprint mismatch — almost always a client bug), age of the oldest in_progress record (an indicator of stuck operations). The ratio of retries to unique operations is a direct measure of network quality between client and server. If it spikes, proxy channel quality has degraded or rotation has changed somewhere.
Server-side checklist
- Key reservation is atomic.
- There's a distinct in_progress state and a clear response to concurrent retries.
- The body fingerprint is stored, and a mismatch is rejected with an explicit error.
- Result commit is transactional, or there's a mechanism to recover stuck records.
- Infrastructure errors aren't cached; deterministic ones are.
- TTL is at least 24 hours and documented.
- Retries are marked with a header and visible in metrics.
- Key checks happen before network anti-fraud heuristics.
Method 3: A Retry Protocol Through a Proxy That Doesn't Break Idempotency
Retry policy is the topic of a neighboring article, but there's a narrow layer where retry policy and idempotency intersect: which specific signals from the proxy chain mean a retry is needed, and how the retry must not destroy the contract. Let's unpack it.
Distinguish proxy failure from upstream failure
When working through a proxy, the client gets errors from three sources, and they need to be handled differently:
- Connection errors to the proxy. Failed to connect to the proxy, the proxy returned 407 (authentication required), the proxy closed the connection before sending the request. The request definitely didn't reach the server. Retrying is safe even without a key, but with a key it's still better.
- Proxy errors when calling the upstream. The proxy returned 502 Bad Gateway or 504 Gateway Timeout. Here uncertainty is maximal: 502 usually means the upstream didn't respond correctly but might have performed the operation; 504 means the proxy got tired of waiting while the server might still be processing. Retry only with a key.
- Upstream responses that passed through the proxy. 500, 503, 429 from the target server. Their semantics are determined by the target API's contract. Retry with a key per retry policy rules.
Special attention is deserved by a connection break while reading the response. The headers already arrived, but the body got cut off. This is the nastiest case: you might have seen the 201 Created status, but you didn't get the order ID from the body. A retry with a key returns the full response snapshot. Without a key, you'll get a second order and still never learn the first order's ID.
IP rotation as an event, not a background
If your mobile proxy pool rotates addresses on a schedule, sync the sending of critical operations with stability windows. A simple trick: query the proxy service's API for the time until the next rotation and don't start creating an order if the time until rotation is less than your response wait timeout. This doesn't replace idempotency but reduces the number of retries, and therefore the load on the key store and the partner's anti-fraud system.
If rotation happens on demand, don't trigger an IP change between attempts of a single operation without a reason. Some teams do so "to bypass stickiness," but the server, seeing the same key from a new address, may enable additional checks. The key will save you from a duplicate, but the retry may slow down.
Timeouts: two boundaries instead of one
Over a direct connection, you usually set one timeout for the response. Through a proxy, you need at least two: a timeout for establishing the connection to the proxy (short, a few seconds) and a timeout for waiting for the upstream response (longer, accounting for order processing time). If you merge them into one, a short timeout will constantly trigger on slow processing, generating excess retries, while a long one will make you wait extra seconds for an unreachable proxy.
Important: the response wait timeout must not be shorter than the actual order creation time on the server. Otherwise, you'll systematically break the connection right at the moment of processing, and every operation will turn into a pair of "broken attempt plus retry." The key will protect against duplicates, but the server will waste resources processing for nothing.
Retrying through a different proxy
A sensible strategy: the first retry goes through the same proxy (maybe the connection just flickered), the second through a different proxy from the pool (maybe the issue is with a specific channel). The key doesn't change. The body doesn't change. Only the route changes. This way you separate network issues from request issues and get diagnostic information: if the second proxy succeeded immediately, the problem was in the channel.
Step-by-step safe retry protocol
- Read the key and body from the local intent record.
- Send the request through the current proxy with two timeouts.
- On success: save the result, finish.
- On 409 from the server: wait the indicated time or a reasonable interval, retry with the same key.
- On 422 with a fingerprint mismatch code: stop, raise an incident, don't retry automatically.
- On a proxy connection error: switch the proxy, retry immediately.
- On 502, 504, a break, or a timeout: apply the backoff policy, retry with the same key; from the second retry onward — through a different proxy.
- After exhausting attempts: leave the record in the unknown state, hand it off to the reconciliation queue (see Method 4).
Point eight is critical. The state "I don't know whether the order was created" is a legitimate state, and it must not be silently collapsed into either "created" or "not created."
Method 4: When the Server Doesn't Support Idempotency Keys
The ideal world ends at the first partner API whose documentation is missing the word "idempotency." What do you do if you're the client and the server simply creates an order for every POST? There are several techniques, from simple to architectural.
Natural key in request fields
Many APIs, lacking a formal Idempotency-Key, accept a field like external_order_id, client_reference, merchant_order_number and promise uniqueness on it within the account. That's idempotency under a different name. Check the documentation and, more importantly, check the behavior: send two requests with the same external ID in the test environment and see whether a duplicate error comes back or a second entity is created. Documentation promises and actual behavior diverge more often than you'd like.
Read before retry
If the server allows searching for entities by your external ID or by parameters (for instance, a list of orders filtered by time and amount), then before every retry, do a check request: has an order with my characteristics already appeared? This doesn't give a strict guarantee (a window can slip between the check and the retry), but it reduces the probability of a duplicate by an order of magnitude. The scheme:
- The first attempt ended in uncertainty.
- Wait an interval sufficient for processing to complete on the server.
- Perform a GET filtered by external ID or a unique combination of fields.
- If the order is found — record its ID, no retry needed.
- If not found — retry the POST.
For search to work, include a marker in the order by which it can be found: a comment, a custom field, a unique line number. Even if the server ignores it as a key, it will return it in the response, and you'll be able to reconcile.
Splitting the operation into reservation and confirmation
Some APIs support a two-phase model: first a draft or hold is created (an operation that can be safely retried or canceled), then confirmation by the draft ID. Confirmation by ID is idempotent by nature: you can't confirm one draft twice. If such a model is available, it's preferable to one-step creation even when keys exist, because it reduces the uncertainty window of the riskiest step.
Outbox pattern on your side
Even if the server guarantees nothing, you can guarantee that from your side every operation goes out with exactly one intent. A transactional outbox is a table of outgoing operations where a record lands in the same transaction as the business change (say, moving a cart to the "being processed" status). A separate sender process picks records from the outbox and executes requests through the proxy. Each record has a unique identifier that you use as the key or as the external_id. Even if the sender crashes mid-work, the record stays in the outbox, and the retry goes out with the same identifier.
Reconciliation as the last line of defense
Reconciliation is a regular process that compares the list of your operations against the list of entities on the partner's side and finds discrepancies: orders that exist on the partner's side but aren't linked on yours; operations in the unknown state; duplicates by matching characteristics. Reconciliation doesn't prevent a duplicate, but it finds it in minutes rather than a month later in an accounting report. For operations over unstable channels, this is a mandatory component, not an option.
A practical approach: run reconciliation every 10–15 minutes for operations older than your maximum retry timeout, automatically link found entities to unknown records, and send detected duplicates to a cancellation queue with manual confirmation if the amount is above a threshold.
Saga and compensations
If an order is part of a chain of actions (warehouse reservation, payment, delivery), a duplicate in one link breaks the whole chain. The saga pattern assumes that each step has a compensating action: cancel the order, refund the payment, release the reservation. The compensation must also be idempotent, otherwise you'll be fighting duplicates of cancellations. When designing an integration, immediately write out the pairs of "action — compensation" and check for each that a retry is safe.
Common Mistakes: How Teams Create Duplicates Themselves
Mistake 1. The key is generated inside the HTTP client
The most common. A wrapper over the HTTP library adds an Idempotency-Key header with a new UUID on every call. The retry mechanism inside the same library re-sends — and gets a new key. Outwardly, everything looks right: the header is there. In reality, there's no protection. Verify in the server or proxy logs that the key matches on retries.
Mistake 2. The key changes along with the body when "fixing" the request
The client got a 400 due to an invalid field, fixed the field, and retried with the same key. A server with fingerprint checking returns 422. A server without fingerprint checking returns the cached 400 error, and the client keeps banging its head against the wall. A fixed request is a new operation, and it needs a new key.
Mistake 3. Checking the key via SELECT, then INSERT
Covered above, but let's repeat: a parallel retry slips between the read and the write. This is especially common when working through proxies, because client timeouts trigger retries at intervals of a few seconds, and order processing takes about that long. Only atomic reservation.
Mistake 4. Idempotency by IP or session
The server "remembers" that the same request came from this IP just now and rejects the retry. When a mobile proxy rotates, the retry comes from a different address, and the protection fails. The reverse case is bad too: legitimate different orders from one proxy address (you have many clients behind one IP, after all) get rejected as duplicates. The key must be explicit and passed by the client.
Mistake 5. TTL that's too short
Keys live five minutes, "to keep Redis clean." The network outage lasts twenty minutes. The retry queue reaches the operation half an hour later. The key is forgotten, the order gets created a second time. The TTL must cover the longest realistic delayed-retry scenario.
Mistake 6. Caching infrastructure errors
The first attempt failed with a 500 due to a temporary database outage. The server saved the 500 as the result under the key. All subsequent retries get 500 from the cache, even though the database came back up long ago. The client is forced to change the key, losing the link to the original operation. Only deterministic results can be cached.
Mistake 7. Collapsing the unknown state
After exhausting attempts, the client marks the operation as failed and lets the user place the order again. But the server performed the first operation. The user gets two orders and a double charge. The unknown state must exist explicitly and be resolved through reconciliation, not through optimistic assumptions.
Mistake 8. Retrying non-idempotent operations "because it's easier"
A universal retry layer repeats all requests the same way, including POSTs without keys. This works until the first network storm, after which hundreds of duplicates appear within an hour. The retry layer must know which requests are allowed to be retried and must refuse to retry non-idempotent ones without an explicit key.
Mistake 9. One key per batch
The client sends an array of a hundred orders with one key. The server processed sixty and crashed. A retry with the same key gets a 409 or a partial snapshot. How do you figure out which forty need to be created? For batches, you need either a key per item, or a server guarantee of atomicity for the entire batch, or a response with per-item status. Without that, batches over an unstable channel are a source of chaos.
Mistake 10. Missing key in the logs
A duplicate happened, and you're trying to figure out whether it was a retry with the same key (server bug) or two different operations (client bug). There's no key in the logs. The investigation turns into guesswork. The key must be in every log line on both sides and in the proxy log if the proxy service supports header logging.
Tools and Resources for Implementation
On the client side
- Identifier libraries. UUIDv7 and ULID generators exist for all major languages. Choose ones that guarantee monotonicity within a millisecond if keys go into the server's database indexes.
- HTTP clients with separate timeouts. Make sure your client allows setting a separate timeout for connecting to the proxy and for reading the response. Most modern clients can do this, but a single timeout is often the default.
- Queues with delivery guarantees. Any broker system with processing acknowledgments will do for implementing the outbox. What matters is that the message identifier is stable across redeliveries — it's convenient to use as the basis for the key.
- Proxy management APIs. Quality mobile proxy services have a programmatic interface that lets you query channel state, time until rotation, and change IP on demand. Integrating with this API lets you avoid starting critical operations before a rotation and consciously change the route between attempts.
On the server side
- Relational database with unique indexes. The most reliable option for a key store if volumes allow. Insert with conflict handling is an atomic operation out of the box.
- Redis or equivalent. For high-load systems: SET with NX and EX flags for reservation, a separate key for the response snapshot. Remember the risk of data loss on node failure and configure persistence.
- Idempotency middleware. For popular web frameworks, ready-made middleware layers exist that implement the basic contract: reservation, fingerprint, snapshot. Check how they behave on concurrent retries and on a crash between execution and commit — that's exactly where good implementations diverge from formal ones.
- API gateways. Some gateways can perform deduplication by header at the edge. This takes load off the application, but the gateway usually doesn't know about the business result and may cache infrastructure errors. Use it as a first filter, not as the only protection.
Testing
- Network fault injection. Proxy-based testing tools that can break the connection after sending a request but before the response, delay the response beyond the timeout, return 502 and 504. Without such tests, you won't see how your system behaves in the only scenario that matters.
- Concurrency tests. Simultaneously send N requests with the same key. Expected result: exactly one created entity, N-1 responses that are either 409s or snapshots. Any other result is a defect.
- Restart-during-operation test. Kill the process after creating the order but before committing the key. Verify that the stuck-record recovery mechanism worked.
- Rotation test. First attempt through one IP, retry through another. Verify that the server returned the snapshot rather than rejecting the retry with a heuristic.
Standards and documents
The IETF specification draft on the Idempotency-Key header is the starting point for formulating the contract: it defines the header name, the semantics of conflicts, and recommended response codes. The public documentation of major payment providers is a good model for how to describe a contract for external integrators: TTL, behavior on body mismatch, handling of concurrent requests. The section on idempotent methods in the HTTP semantics specification explains why PUT and DELETE can be retried and POST cannot, and it's worth rereading before designing endpoints.
Case Studies and Results
Case 1. An order aggregator and 47 duplicates per month
A company-aggregator passed orders to a logistics partner's system through a pool of mobile proxies rotating every five minutes. Volume — about 12,000 orders a day. There were no idempotency keys: the partner API didn't document them. Duplicates were discovered once a month via financial reconciliation, each costing an average of 900 rubles for cancellation, refund, and manual work.
Diagnosis from the proxy logs showed: 0.4% of requests ended in a connection break after the body was sent and before the response was received, mostly during rotation minutes. The retry went out three seconds later, and in 30% of cases the first attempt had already completed by then. The math matched the observed duplicates.
The solution came in three steps. First, it turned out that the partner API accepts an external_id field and rejects duplicates on it, though the documentation mentioned this in a single line. The field started being populated with a stable identifier from the local outbox. Second, rotation windows were synced with sending: new orders weren't sent 15 seconds before rotation. Third, reconciliation was added every 10 minutes. The result over a quarter: zero duplicates, retry count down 70% thanks to rotation syncing, average order time up 1.2 seconds due to waiting for windows — an acceptable price.
Case 2. A payment service and concurrent retries
A fintech service implemented idempotency on its side: the header was accepted, the key was saved in the database, and on retry the snapshot was returned. Duplicates still appeared, rarely but regularly: about one in 40,000 operations. The investigation showed that the key check was done via a read followed by a write, and clients with aggressive timeouts (two retries within a second through different proxies) slipped into the window between them.
It was replaced with an insert with a unique index, and an in_progress state was added with a 409 response and a Retry-After header of one second. A concurrent retry metric was added. It revealed something unexpected: 2% of all operations get at least one 409, meaning clients retry far more aggressively than assumed. The service updated the documentation with timeout recommendations, and two months later the 409 rate dropped to 0.3%. Duplicates — zero for six months.
Case 3. A marketplace integration and batches
A seller uploaded stock and prices to a marketplace in batches of 500 items through a proxy. Batch idempotency was at the level of the whole request. On a connection break, the retry was either rejected (batch in progress) or returned a snapshot of a partial result from which it was impossible to tell which items had been updated. The seller concluded "nothing updated" and re-uploaded, creating races between the old and new batches.
The fix: switch to batches with per-item operation IDs and request per-item status after an uncertain completion. The batch size was reduced to 100 items so that processing time wouldn't exceed the timeout. The number of "stuck" updates dropped from dozens a day to zero, and the stock discrepancies that had regularly led to customer order cancellations disappeared.
Case 4. An automated agent and cascading retries
A 2026 trend: more and more integrations are performed by autonomous agents based on large language models that call APIs according to instructions. A team deployed such an agent for procuring from suppliers. On an uncertain response, the agent "reasoned" and decided to repeat the operation, generating a new request. There were no keys, since the agent formed the call anew each time. In the first week — 11 duplicate purchase orders.
The fix: the agent was forbidden to form creation requests directly. It creates an intent record in an outbox through an internal tool, and a deterministic service with keys and reconciliation performs the sending. The agent only receives the final status. Duplicates disappeared, and along the way full traceability appeared for who initiated each procurement and why. The general lesson: autonomous systems with retry ability are a new and powerful source of duplicates, and an idempotency barrier is mandatory for them.
FAQ: Hard Questions About Idempotency and Retries
Do I need an idempotency key for PUT and DELETE if they're idempotent by specification?
Formally, no: repeating a PUT with the same body produces the same state. But there are nuances. If PUT returns 201 on create and 200 on update, a retry returns a different status, and the client may interpret it differently. DELETE on retry returns 404 instead of 204, which is often treated as an error. A key with a response snapshot makes behavior predictable. Also, if PUT triggers side effects (sending a notification, writing to an audit log), they'll run again. Method idempotency isn't equal to the idempotency of all side effects.
What if the server returns 409 in_progress but the first attempt actually died?
That's the server's responsibility: it must have a mechanism to recover stuck records past a time threshold. The client should wait on 409 not indefinitely, but a limited number of times (say, up to two minutes), then move the operation to unknown and hand it off to reconciliation. If the server systematically keeps in_progress longer than reasonable, that's an implementation defect that should be reported to the API owner.
How should the key work if the request goes through multiple proxies in a chain?
No special handling: the key is passed in a header and transparently traverses any chain. The only thing worth checking is whether some intermediate node strips non-standard headers. Some corporate gateways filter headers by an allowlist. A simple test: send a request through the whole chain to your own debug endpoint and verify the header arrived.
Can I use a hash of the request body as the key and store nothing on the client?
Yes, with caveats. Two different orders with identical content (same item, same amount, same buyer) will get the same key, and the second will be rejected as a retry. To avoid this, a unique component is added to the body — a cart ID or a local sequence number. Then the hash from the body becomes a deterministic operation key. This approach is convenient for clients without reliable local storage, but it requires strict body normalization: changing the field order in JSON will change the hash.
How long should I wait before a retry so the server can finish the first attempt?
This is a retry policy question, but from a contract standpoint one thing matters: if the server supports in_progress with a 409, you don't need to wait "by eye" — the server itself tells you the operation is still running. If the server doesn't have such a state, the interval before a retry must exceed the 99th percentile of the operation processing time on the server. You can learn it from the documentation, from metrics of successful requests, or empirically on a test bench.
Should read operations be made idempotent?
GET is idempotent by definition and doesn't need a key. But there's an exception: a "read with a side effect," for example a GET that marks a notification as read or consumes a rate limit. That's an architectural smell; such operations are better refactored into a POST with a key than to bolt idempotency onto a GET.
What should happen to keys during a migration or API version change?
Include the endpoint version in the key scope on the server. Then a key used in v1 won't collide with the same key in v2. For the client, the rule is simple: a migration means new operations with new keys; old keys aren't carried over. If an operation was in unknown at migration time, resolve it via reconciliation first, then migrate.
How do I explain to the business why we should spend a sprint on idempotency if duplicates "almost never happen"?
Calculate the cost of one duplicate: cancellation, refund, support, reputation, sometimes partner penalties. Multiply by the observed frequency. Then show that the frequency grows non-linearly during network incidents: on a calm day zero duplicates, on the day of a telecom operator outage — hundreds per hour. Idempotency is insurance with a known premium and an unknown but potentially huge payoff. Plus it has a side benefit: the ability to retry requests aggressively without fear, which speeds up recovery after failures.
Does idempotency increase server load?
Insignificantly: one extra write and one extra read per operation. But it reduces load in failure scenarios, because a retry returns a snapshot in milliseconds instead of fully re-processing the order. During a retry storm, a server with idempotency handles the load better than one without it — counterintuitive but confirmed in practice.
Conclusion: A Contract Instead of Hope
Let's put the main points together. Any network delivers requests on an "at least once" basis. Proxies — especially mobile ones, with rotation and radio channels — make retries the norm, not a rarity. Retry policy answers the question "when and how to retry." Idempotency answers the question "why the retry won't break anything." Without the second answer, the first is dangerous.
A safe retry rests on three pillars. The operation key, born before the first attempt and living in your database, not in the HTTP client. The server contract with atomic reservation, an in_progress state, a body fingerprint, a response snapshot, and a reasonable TTL. Reconciliation as the last line of defense for operations stuck in uncertainty. If the server doesn't provide a contract, the role of the key is taken over by a natural identifier, and the role of the store — by your outbox.
What to do next, today:
- Find all POST operations in your integrations that are retried automatically. For each, answer: is there a key, where is it generated, does it stay the same on retries?
- Read the partner API documentation for external_id fields and the Idempotency-Key header. Verify the behavior in a test environment with two identical requests.
- Add an unknown state to your operation model and a reconciliation process for it.
- If you own the server, run three tests: concurrent retries, a crash between execution and commit, and a retry from a different IP.
- Put the ratio of retries to unique operations on a dashboard. That's your health indicator for the channel through the proxy.
Idempotency is a rare case where engineering discipline converts directly into business peace of mind. A contract designed once protects for years against duplicates, retry storms, and midnight calls from accounting. And, honestly, there's a special pleasure in looking at a chart of thousands of retries through an unstable channel and seeing a confident zero in the duplicates column.