Introduction: Why Streaming HTTP Through Proxies Became a Mass Problem

Picture this: you send a request to a language model, and instead of the familiar "typing" of tokens, the screen stays silent for forty seconds and then dumps the entire answer in one chunk. Or a monitoring dashboard that's supposed to refresh every second shows data half a minute late. Or a parser listening to a trading platform's event stream gets batches of a hundred events instead of a steady trickle. Nine times out of ten, the culprit isn't the server or the client. It's the thing standing between them: a proxy, load balancer, CDN, or corporate gateway that decided to "help" by accumulating the entire response before passing it along.

The topic of streaming HTTP responses through intermediate nodes was long considered niche. In 2026, it's become mainstream. Token streaming from LLM APIs, server-sent events in web interfaces, long-polling in messengers and bots, progress bars for long exports, streaming parser output — all of this runs on top of plain old HTTP that simply isn't in a hurry to close the response. And all of it breaks the moment a node with buffering enabled appears in the path.

In this guide, we'll examine three classic mechanisms — chunked transfer encoding, Server-Sent Events, and long-polling — specifically from the perspective of traversing proxies. One important caveat up front: no WebSocket or gRPC here. Those have their own dedicated material, and the difference is fundamental. There, a protocol upgrade happens or binary framing is used over HTTP/2, and the proxy needs to recognize and allow that upgrade. Here, it's a regular HTTP response with no Content-Length header that the server keeps open. No handshake, no protocol switch. Seems like it couldn't be simpler. But that very simplicity is what creates the problem: the proxy sees nothing unusual and applies standard buffering rules designed for short responses.

What you'll get from this material: an understanding of how a stream works at the byte and header level; a map of every place where a response can get stuck; ready-made configurations for nginx, HAProxy, Envoy, and client libraries; a framework for choosing between the three mechanisms; diagnostic checklists; and real-world case studies with numbers. This material is aimed at developers, DevOps engineers, and anyone using proxies to work with APIs, scraping, and automation.

Fundamentals: How a Regular HTTP Response Becomes a Stream

Two Ways to Communicate Response Length

In HTTP/1.1, there are two fundamental ways to tell the client where the response body ends. The first is the Content-Length header: the server declares the exact number of bytes in advance. The second is Transfer-Encoding: chunked: the server sends the body in chunks, each chunk starting with its size in hexadecimal, and the whole thing ending with a zero chunk. There's also a third, archaic option — just closing the connection after the last byte — but with keep-alive it's practically unused.

Why does this matter for streaming? Because Content-Length requires knowing the size in advance. If the server generates the response gradually — token by token, event by event — it physically can't calculate the length before transmission starts. The only option in HTTP/1.1 is chunked. In HTTP/2 and HTTP/3, chunked encoding as such doesn't exist: the body is transmitted in DATA frames anyway, and the end is marked with an END_STREAM flag. But the logic is the same: a response with no declared length that can last indefinitely.

What "Not Closing the Response" Means

An analogy. A regular HTTP response is a package: you receive a box with the weight written on it, you open it, and you know everything inside. A streaming response is a conveyor belt: boxes come one after another, and you don't know when the belt will stop. The server sent headers with a 200 status, sent the first chunk — and waits. New data arrives — it sends the next one. The connection stays open for minutes, hours, sometimes days.

From TCP's perspective, nothing unusual is happening: the same socket, the same segments. From HTTP's perspective, it's a single request and a single response, just a very long one. And here's where the root of all proxy problems lies: the intermediate node sees the beginning of the response, doesn't see the end, and has to decide what to do with the bytes that have already arrived. Pass them along immediately? Or wait, accumulate more, and send them more efficiently? By default, most proxies choose the latter.

Three Mechanisms: Commonalities and Differences

  • Chunked streaming — the lowest level. The server just writes bytes to the body and calls flush. The content format is arbitrary: text, NDJSON, binary data. The client decides how to parse it.
  • Server-Sent Events — a standardized text format on top of a chunked stream. Content-Type: text/event-stream, events separated by blank lines, with data, event, id, and retry fields. Browsers support it natively via EventSource with automatic reconnection.
  • Long-polling — strictly speaking, not a stream. The client makes a request, the server holds it open until data appears or a timeout hits, responds, and closes. The client immediately makes the next request. It's a stream simulation using a sequence of regular responses, each of which can have a Content-Length.

What do they have in common? In all three cases, the response to a single request is delayed for an indefinite time, and the proxy has to survive this: not kill the connection on timeout and not delay the data in a buffer. The difference is how much each mechanism depends on the transparency of the intermediate node. Chunked and SSE depend on it critically. Long-polling — minimally, and that's its main advantage.

The Proxy's Role in the Chain

Between the client and server in modern infrastructure, there's rarely nothing. A typical chain: client application — outbound proxy (say, mobile or residential) — internet — CDN — cloud load balancer — reverse proxy — application. Each of these nodes is a potential buffering point. And each is configured by default to optimize short responses: collect, compress, deliver whole. For streaming, that optimization is a disaster. We'll peel back these layers one by one and see what happens at each.

Deep Dive: Where and Why a Stream Gets Stuck at an Intermediate Node

Why Proxies Buffer in the First Place

Before cursing buffering, it's worth understanding its logic. A reverse proxy like nginx buffers for three reasons. First — protecting the backend from a slow client: if the client on a mobile connection reads ten kilobytes per second while the backend has already generated the response, it's more efficient to grab everything from the backend at once, free up its worker, and then calmly serve the client from the buffer. Second — compression: gzip and brotli work more efficiently on large blocks. Third — the ability to set Content-Length and correctly handle a backend error before sending headers to the client.

For a typical API with responses of a few kilobytes, all of this is reasonable and saves resources. For streaming, it's the exact opposite: by delaying bytes, the proxy destroys the very value of the stream. That value is in timeliness, not volume.

Map of Eight Buffering Points

Let's walk through the chain and mark every point where data can get stuck. Keep this list handy when diagnosing.

  1. The application. The framework might buffer output itself. WSGI servers, PHP with output_buffering enabled, Node.js frameworks with compression middleware. Flush isn't called — data sits in the process buffer.
  2. The reverse proxy in front of the application. nginx with proxy_buffering on (enabled by default) accumulates the response in buffers defined by proxy_buffer_size and proxy_buffers, and when they overflow — into a temporary file on disk. Apache mod_proxy buffers less but has its own settings. Caddy doesn't buffer by default, but flush_interval is worth checking.
  3. Compression on the proxy. Even with proxy_buffering off, enabled gzip accumulates data up to the compression window size. This is one of the most insidious points: buffering is formally disabled, but the stream comes in bursts.
  4. L7 load balancer. HAProxy in http mode passes data as it arrives, unless http-buffer-request is enabled, but is limited by tune.bufsize. Envoy streams by default, but compression or Lua filters can accumulate the body.
  5. CDN and cloud gateways. Many CDNs buffer responses for caching and compression. Some providers don't support streams longer than a certain time or require explicit configuration. Cloud API gateways often have a hard response time limit around 30 seconds and don't support body streaming at all.
  6. The client-side outbound proxy. The HTTP proxy you use to reach the internet — corporate, mobile, residential — is also an HTTP parser. If it's built on nginx or Squid and configured to buffer, the stream will get stuck there. Quality proxy services pass chunked responses transparently, but you need to verify this in practice.
  7. TLS and TCP. Data can be delayed in TLS records if the library collects them up to 16 kilobytes. The Nagle algorithm delays small TCP segments until the previous one is acknowledged. For a stream of small chunks, this adds tens of milliseconds to each.
  8. The client library. Python's requests reads the entire body into memory unless stream=True is passed. A number of HTTP clients buffer up to chunk_size.

Eight points. If even one of them accumulates data, you'll see the stream as bursts or one big batch at the end. And the nastiest part: behavior depends on volume. On small responses, everything looks fine, but under load or on long responses, the buffer fills up, writing to temporary files kicks in, and latencies skyrocket. No wonder many teams only discover the problem in production.

HTTP/1.1 vs HTTP/2 in the Context of Streaming

In HTTP/1.1, one connection — one response at a time. A long-lived response occupies the connection entirely. Browsers limit simultaneous connections to a single host to six, so six open SSE streams block all other requests to that host. A proxy working over HTTP/1.1 with the backend also holds one connection per stream.

In HTTP/2, multiplexing removes this limitation: a hundred streams live in a single TCP connection. But there's a subtlety — flow control at the protocol level. If the receiver doesn't expand the window via WINDOW_UPDATE, the sender stops. Some proxies implement the forwarding of these frames sloppily and create artificial pauses.

It's important to understand: a proxy can accept a request over HTTP/2 from the client and go to the backend over HTTP/1.1. It repackages DATA frames into chunks and back. This is a normal operation, but it's precisely at the conversion boundary that timely flush often gets lost — the proxy waits until a "convenient" frame size accumulates.

Timeouts — The Second Stream Killer

Buffering delays data. Timeouts kill the connection. nginx has proxy_read_timeout — the maximum time between two consecutive read operations from the backend, 60 seconds by default. If the server stays silent longer than a minute, nginx closes the connection and returns a 504 to the client. For long-polling with a 90-second wait, this is a guaranteed failure. HAProxy has timeout server and timeout tunnel. Cloud load balancers have an idle timeout, often 60 or 350 seconds. Mobile carriers have NAT timeouts that reset inactive connections after a few minutes.

Hence the iron rule: a stream without periodic activity is doomed. Heartbeat messages every 15-30 seconds aren't an option — they're a mandatory requirement for any long-lived response.

Mechanism Selection Matrix

Before moving to practice, let's lock in the selection framework. Answer four questions.

  • Do you control the entire proxy chain? Yes — chunked and SSE work. No, there are unknown gateways in the chain — start with long-polling or plan for degradation to it.
  • Who's the client? A browser — SSE with EventSource gives reconnection for free. Your own code — chunked with NDJSON is more flexible.
  • What's the nature of the data? Fragments of a single entity (tokens of one response) — chunked or SSE. Discrete independent events — SSE or long-polling.
  • What's the frequency? Dozens of events per second — chunked or SSE with batching. One event per minute — long-polling with a long timeout or SSE with heartbeat.

Method 1: Chunked Transfer Encoding — Low-Level Streaming Without Overhead

When to Choose Pure Chunked

Pure chunked streaming is appropriate when the client is your own code, not a browser; when the data format isn't text or doesn't fit the event model; when you need maximum control over every byte. Typical examples: streaming NDJSON from an API, streaming parser output, downloading large exports with a progress indicator, proxying LLM responses in your own format.

Step-by-Step Server-Side Setup

  1. Make sure the application writes in pieces. After each logical portion, call flush. In Python with an ASGI server, that's a generator yielding bytes; in Node.js — res.write followed by flush if compression is involved; in Go — the http.Flusher interface after each Write.
  2. Don't set Content-Length. Most frameworks will switch to chunked automatically if the length isn't specified. Verify that the response over HTTP/1.1 has Transfer-Encoding: chunked.
  3. Disable compression at the application level for streaming routes or make sure it flushes along with the data. A compromise is gzip with a synchronous flush after each chunk; this works but reduces the compression ratio.
  4. Set the X-Accel-Buffering: no header. This is an nginx-specific header that disables buffering for a specific response without changing the global config. Some other proxies understand it too. Also add Cache-Control: no-cache, no-transform — the second value asks intermediate nodes not to modify the body.
  5. Configure the reverse proxy. For nginx in the streaming route's location: proxy_buffering off; proxy_cache off; proxy_http_version 1.1; proxy_set_header Connection ''; proxy_read_timeout 3600s; gzip off;. The proxy_http_version 1.1 directive is mandatory: by default nginx talks to the backend over HTTP/1.0, where chunked doesn't exist, and nginx is forced to buffer until the connection closes.
  6. Separate your routes. If compression is needed for the rest of the API, enable it only in non-streaming location blocks.
  7. Check the Nagle algorithm. In nginx, tcp_nodelay is enabled by default, but in the application with small frequent chunks, you may need to explicitly set TCP_NODELAY on the socket.

HAProxy and Envoy Configuration

HAProxy. In the backend section, don't use option http-buffer-request. Make sure timeout server is large enough for your scenario, and for truly long streams set timeout tunnel. Disable compression for streaming paths via an ACL, or rely on the no-transform header: HAProxy respects Cache-Control: no-transform and won't compress such responses.

Envoy. Buffering is disabled by default, which is great. But check two timeouts: stream_idle_timeout in the HTTP connection manager (five minutes by default) and route timeout (fifteen seconds by default — this is the one that most often kills streams). For streaming routes, set the timeout to 0s, which means no limit, and rely on idle timeout combined with heartbeat.

Chunked Stream Verification Checklist

  • The response has Transfer-Encoding: chunked, or uses HTTP/2 without Content-Length.
  • The X-Accel-Buffering: no header is present in the application's response.
  • Content-Encoding is absent or compression flushes along with the data.
  • The first body byte arrives within a second after the headers.
  • Intervals between chunks at the client match the generation intervals on the server within 50 milliseconds.
  • The connection survives longer than a minute without breaking when there's a heartbeat.
  • The proxy logs show no entries about buffering to a temporary file.

Method 2: Server-Sent Events — The Standard for Server-to-Client Events

Format Structure

SSE is simply a convention for laying out events in a text stream. The response has Content-Type: text/event-stream. Each event is a set of "field: value" lines ending with a blank line. Fields: data — the payload (can repeat, lines joined with a newline); event — the event type name; id — an identifier for resumption; retry — the recommended reconnection delay in milliseconds. A line starting with a colon is a comment, and it's the perfect heartbeat: the browser ignores it, while the proxy sees activity on the connection.

The beauty of SSE is in the asymmetry. The complexity is offloaded to the spec and the browser, while the server stays trivial. No handshake, no binary framing. Just text that flows. That's exactly why SSE became the de facto standard for streaming language model responses: the vast majority of major LLM APIs emit tokens in text/event-stream format. It's surprising that a technology described in specs over fifteen years ago is enjoying a second renaissance in 2026.

Why SSE Is Especially Sensitive to Buffering

An SSE event weighs from a few dozen bytes to a couple of kilobytes. The default nginx buffer is 4 or 8 kilobytes for the first block and eight blocks of 4-8 kilobytes after that. That means the proxy will accumulate fifty to several hundred events before delivering the first. For a chat with a model, it looks like this: the user waits fifteen seconds, then the whole paragraph appears instantly. The illusion of typing is destroyed — and that's the whole reason streaming was implemented in the first place.

Step-by-Step SSE Implementation Through a Proxy

  1. Response headers: Content-Type: text/event-stream; charset=utf-8, Cache-Control: no-cache, no-transform, X-Accel-Buffering: no, and for HTTP/1.1 also Connection: keep-alive.
  2. Heartbeat. Send a comment line like : ping every 15-25 seconds. This is below any reasonable idle timeout and keeps carrier NAT tables current.
  3. Event identifiers. Assign each event a monotonically increasing id. On reconnection, the browser will send the Last-Event-ID header, and the server can resend what was missed. Without this, any break at the proxy means data loss.
  4. The retry field. Set a reasonable value, say 3000 milliseconds, so clients don't create a reconnection storm when the server restarts.
  5. nginx configuration — same as for chunked. Additionally, make sure gzip is not applied to text/event-stream: if gzip_types includes text/*, add a separate location with gzip off.
  6. Connection planning. Each open SSE stream over HTTP/1.1 is a separate connection between the proxy and the backend. For a thousand simultaneous clients, you need a thousand connections. Use HTTP/2 on the client side and an async server on the backend.
  7. Handling closure. The server should detect client disconnection and stop generation. Through a proxy this happens with a delay: the server only learns of the break when it tries to write. For expensive operations like model inference, this is wasted seconds of GPU time. The solution is regular heartbeat writes that trigger a write error on a broken connection.

SSE and HTTP/2: Nuances

Over HTTP/2, SSE works without the six-connection limit — that's the main reason to enable HTTP/2 on the front-end proxy. However, when translating HTTP/2 to HTTP/1.1 toward the backend, some proxies add a small delay due to frame aggregation. Make sure the first event goes out immediately, not waiting for a frame to fill. If the backend also speaks HTTP/2, delays are minimal, but verify that the proxy correctly forwards WINDOW_UPDATE and doesn't throttle the stream with flow control.

SSE Stream Design Framework

  1. Define the event types and their frequency. Rare events need heartbeat; frequent ones need batching of multiple records within one event.
  2. Choose a resumption strategy: keep a ring buffer of the last events on the server for handling Last-Event-ID, or rely on a full resync on reconnection.
  3. Set a limit on the stream's duration. A good practice is for the server to close the stream itself every 5-10 minutes, with the client transparently reconnecting. This protects against state accumulation and bypasses hidden infrastructure timeouts.
  4. Plan for degradation: if the stream doesn't pass through the proxy, automatically switch to long-polling.

Method 3: Long-Polling — A Stream of Sequential Regular Responses

How It Works and Why It's Still Relevant

Long-polling is the oldest of the three. The client sends a request, the server doesn't respond immediately but waits for an event up to a given timeout (usually 20-60 seconds). An event appears — the server delivers the response and closes it. Nothing appears — it delivers an empty response. The client immediately sends the next request. From the outside, it looks like a stream with minimal latency, but each response is ordinary, with a Content-Length, fully compatible with any proxy, any cache, and any gateway.

Why is long-polling alive and well in 2026? Because it gets through everywhere. Through corporate gateways that cut chunked. Through cloud API gateways with response time limits. Through proxy servers with unpredictable buffering. The most famous example is the update-fetching method in the Bot API of a major messenger, where the timeout parameter sets the wait duration. The vast majority of bots have worked this way for years, through all sorts of proxies, without issues.

Five Rules for Designing Long-Polling Through a Proxy

  1. The server timeout is shorter than the shortest timeout in the chain. If the proxy has a proxy_read_timeout of 60 seconds, the server wait should be no more than 50. If there's a cloud gateway with 30 seconds in the chain, no more than 25. Otherwise, instead of an empty response, the client gets a 504, and the stream of errors drives monitoring crazy.
  2. Cursor or offset. The client passes the identifier of the last received event. The server returns everything that appeared after it. The protocol becomes idempotent: a lost response simply leads to re-delivery of the same events.
  3. Jitter on reconnection after errors. Exponential backoff — 1, 2, 4, 8 seconds — plus a random addition. Without jitter, a thousand clients hit the server simultaneously after a failure.
  4. Zero pause after a successful response. The next request goes out immediately. Any pause here is lost latency.
  5. Keep-alive is mandatory. Without it, each request is a new TCP connection and a new TLS handshake. This adds 50 to 300 milliseconds to each cycle, and through a remote proxy — even more.

Step-by-Step Implementation

  1. Server: the endpoint accepts offset and timeout, waits for an event on an internal queue or via a subscription, and responds with an array of events and a new offset. An empty array is a normal response, not an error.
  2. Proxy: standard config, but proxy_read_timeout should be no less than the server timeout plus ten seconds of margin. Buffering can stay enabled — the response is short and complete.
  3. Client: a loop handling three outcomes — data, empty response, error. A unified connection pool, keep-alive, and a client timeout greater than the server timeout.
  4. Monitoring: the share of empty responses, average wait time, frequency of 5xx and client timeouts.

When Long-Polling Loses

At high event frequency, the overhead of headers and the request-response cycle becomes noticeable. With thousands of clients, the server needs to hold thousands of pending requests — without an async stack, you can't cope. And of course, long-polling is awkward for streaming a single long response where you need a sequence of fragments of one entity. Language model tokens via long-polling are theoretically possible but practically clumsy.

Method 4: Consuming Streaming Responses Through an Outbound Proxy

Scenario Specifics

So far, we've been looking through the eyes of a server owner. Now let's flip the picture: you're the client, reaching out to someone else's streaming API through a proxy server, and only the client side is under your control. Typical tasks: collecting data from streaming APIs through an address pool, working with LLM streaming from automation scripts, distributing API load across different IPs, tracking events on platforms in real time, testing your own service from different networks.

Three things matter here. The proxy itself must transparently pass chunked responses. The client library must read the stream as it arrives. And you must be able to distinguish a problem at the proxy from a problem at the target server.

Checking a Proxy for Stream Transparency

  1. Take a known streaming endpoint — a public test service that emits lines with a delay, or your own endpoint with a known chunk schedule.
  2. Make the request directly and through the proxy using curl with the -N flag (disables curl's own output buffering) and the --trace-time flag for timestamps. Compare when the fragments arrive.
  3. If through the proxy all fragments arrive in one piece at the end — the proxy buffers completely. If in portions of a few kilobytes — partially. If they match the direct request — it's transparent.
  4. Test a long stream (5-10 minutes) with heartbeat. A break at a fixed time reveals the proxy's idle timeout.
  5. Test HTTPS. Through a proxy using the CONNECT method, a tunnel is created, and the proxy doesn't parse HTTP inside — there can be no HTTP-level buffering, only TCP delays. This is one reason why streaming APIs through a CONNECT tunnel usually work more smoothly than through a regular HTTP proxy that parses requests.

A nuance about mobile proxies: the cellular network itself adds jitter, and carrier NAT aggressively closes inactive connections. For long streams through a mobile proxy, a heartbeat every 15-20 seconds is a necessity. IP rotation on a schedule must account for open streams: changing the address mid-SSE connection is guaranteed to break it. Align rotation with the stream lifecycle — close the stream, change the address, open a new one with the cursor.

Client Libraries: What to Enable

  • Python, requests: stream=True in the call, then iter_lines or iter_content with a small chunk_size. Set the timeout as a tuple (connect, read), where read is the allowed time between chunks, not the total response time.
  • Python, httpx: client.stream and aiter_lines for async code; for SSE, specialized wrappers that parse the event format.
  • Python, aiohttp: response.content.iter_any returns data exactly as it came from the socket, without repackaging — ideal for measuring real intervals.
  • Node.js: fetch with response.body read as a ReadableStream; for SSE, format parser libraries.
  • Go: a regular http.Client, reading resp.Body through bufio.Scanner with a buffer large enough for big events.

In all cases: don't call methods that read the entire body — text, json, read without arguments. Otherwise the client will wait for the connection to close, and you'll blame the proxy for buffering that isn't there.

The "Three Points" Diagnostic Protocol

When a stream behaves strangely, measure the time to first byte and the distribution of intervals between chunks in three configurations: directly to the server from the same machine; through the proxy to the server; through the proxy to a reference streaming endpoint. The comparison unambiguously localizes the problem. Through the proxy to the reference works well, but to the target server is bad — the target server's infrastructure is at fault. To the reference is bad — the proxy is at fault. Directly is also bad — your code or the network is at fault. This protocol takes ten minutes and saves days of arguments.

Common Mistakes When Working With Streams Through Proxies

  1. Disabled proxy_buffering, forgot about gzip. The most common mistake. The stream is formally not buffered, but compression accumulates data up to the window size. Symptom: bursts of a few kilobytes.
  2. Left proxy_http_version at 1.0. nginx by default talks to the backend over HTTP/1.0, where chunked doesn't exist, and is forced to wait for the connection to close. Symptom: the whole response arrives at the end.
  3. No heartbeat. The stream lives exactly until the first idle timeout in the chain — 60 seconds at nginx, 15 at Envoy per route, a few minutes at NAT. Symptom: breaks at consistent intervals.
  4. Long-polling timeout longer than the proxy timeout. The client gets a 504 instead of an empty response, monitoring goes red, and events may duplicate.
  5. Relying on the proxy's global setting. When moving to a new environment or changing CDN, streaming silently breaks. The solution is X-Accel-Buffering and no-transform at the response level: the application defines the behavior, not the proxy administrator.
  6. No Last-Event-ID and cursors. Any break means data loss. Through a proxy, breaks happen noticeably more often than directly.
  7. Reading the entire body on the client. stream=True forgotten, the client waits for completion, the developer blames the proxy.
  8. IP rotation during an open stream. The connection breaks, there's no reconnection, data is lost.
  9. Testing on short responses. The buffer doesn't have time to fill, everything looks great. In production, on long responses, writing to a temporary file kicks in, and latencies skyrocket.
  10. Mixing streaming and regular routes in one location. You either break streaming or lose compression and buffering for regular responses, which hurts the entire API's performance.
  11. Ignoring client disconnection. Through a proxy, the server learns of the break late and continues expensive generation. For LLM inference, this is real money.
  12. Using chunked where long-polling is needed. Through a corporate gateway or cloud API gateway with a response limit, the stream won't pass, but long-polling will without questions.

Tools and Resources for Configuration and Diagnostics

Reverse Proxies and Load Balancers

  • nginx — the gold standard of flexibility. Key directives: proxy_buffering, proxy_http_version, proxy_read_timeout, gzip, support for the X-Accel-Buffering header.
  • HAProxy — minimal buffering out of the box. Watch timeout server, timeout tunnel, and compression.
  • Envoy — streams by default. Route timeout and stream_idle_timeout are critical.
  • Caddy — doesn't buffer by default; flush_interval set to -1 explicitly enables immediate flush.
  • Traefik — passes streams transparently; timeouts are configured at the entrypoints level.

Diagnostics

  • curl with -N, -v, and --trace-time shows when each fragment arrives. The -w parameter with time_starttransfer and time_total variables gives the time to first byte and total time.
  • tcpdump and Wireshark — to check whether bytes are being delayed at the network level or are held up higher.
  • mitmproxy — for viewing streams while preserving chunk structure. Be sure to enable streaming mode, otherwise it becomes a buffering point itself.
  • Browser developer tools — the EventStream tab for SSE shows every event with a timestamp.

Test Endpoints and Libraries

Public HTTP testing services have endpoints that emit a given number of lines with a delay — they're ideal for checking proxy transparency. Your own minimal server in any language, emitting a chunk every second with a timestamp, is even better: you control both sides. For SSE on the server, virtually any async framework has a ready-made response class for text/event-stream. For SSE on the client outside a browser, format parsers exist for Python, Node.js, and Go. For long-polling, no special libraries are needed — an HTTP client with a connection pool suffices.

Monitoring

Metrics worth collecting: time to first body byte; the distribution of intervals between chunks at the 50th, 95th, and 99th percentiles; stream lifetime and close reason; the number of active streams on the proxy and on the backend; the share of reconnections with Last-Event-ID. A sharp increase in the interval between chunks alongside a growing volume is a sure signal that buffering has kicked in.

Case Studies and Results: How This Looks in Real Projects

Case 1. LLM Streaming Through nginx: From 11 Seconds to 310 Milliseconds

A team deployed a chat based on a language model. Locally, tokens appeared smoothly. After deployment behind nginx, users saw a blank screen for 8-12 seconds, then the whole response at once. Diagnostics using the "three points" protocol showed: directly to the application, the first token arrived in 280 milliseconds; through nginx — in 11 seconds. The causes were classic: proxy_buffering on and proxy_http_version 1.0 by default. After adding proxy_buffering off, proxy_http_version 1.1, gzip off for the route, and the X-Accel-Buffering: no header in the application's response, the time to first token through the proxy became 310 milliseconds. A side discovery: when a tab was closed, inference continued to the end of the response. After implementing a heartbeat with write error checking, GPU time spent on abandoned requests dropped by roughly a quarter.

Case 2. SSE Dashboard and Mobile Networks

A monitoring service delivered events over SSE. On wired internet, everything worked, but from mobile devices, streams broke after 2-4 minutes with no errors on the server. The cause was carrier NAT timeouts on inactive connections. The solution: a heartbeat comment every 20 seconds plus a server-forced stream close every 10 minutes with proper Last-Event-ID. Complaints about a "frozen dashboard" dropped to practically zero, the average continuous stream duration grew from 3 to 10 minutes (limited now by intentional closure), and no event loss was recorded during reconnection.

Case 3. Scraping a Streaming API Through a Pool of Mobile Proxies

A team was collecting events from a platform delivering data as an NDJSON stream through a pool of mobile proxies. Two problems were observed: some streams arrived in bursts of 8 kilobytes, and some broke during address rotation. Testing showed that for unencrypted requests, the proxy parsed HTTP and buffered responses, while through a CONNECT tunnel to an HTTPS endpoint, it passed the stream transparently. Switching to HTTPS completely solved the first problem. The second was closed with synchronization: IP rotation happens only after the stream is properly closed, and reconnection uses the last event's cursor. Data completeness grew from about 91 to 99.7 percent.

Case 4. A Long-Polling Bot Through a Constrained Gateway

A developer ran a bot on infrastructure where outbound requests went through a corporate gateway with a 30-second response limit. Initially, a 60-second wait timeout was used — half the requests ended in gateway errors, updates were duplicated. After reducing the timeout to 25 seconds and adding exponential jitter on errors, the error rate dropped from 48 to under 0.5 percent. The average event delivery latency didn't change — it's determined by the moment the event occurs, not the wait duration.

Case 5. Switching to HTTP/2 for SSE

A web app opened 3-4 SSE streams per tab. With several tabs open, the browser exhausted the six-connection limit, and regular requests queued up behind the streams. Enabling HTTP/2 on the front-end proxy moved all streams into one connection. Page load time with five open tabs dropped from 9 to 1.5 seconds. A side discovery: after the switch, stream_idle_timeout needed to be increased, since the previous settings were designed for HTTP/1.1.

FAQ: Deep Questions About Streaming Responses Through Proxies

How does streaming HTTP fundamentally differ from WebSocket from a proxy's perspective?

WebSocket requires a protocol upgrade: the proxy must recognize the Upgrade and Connection headers, switch into tunnel mode, and stop parsing HTTP. Streaming HTTP requires nothing — it's a regular response that doesn't end for a long time. The paradox is that WebSocket through a proxy either works or doesn't work at all, while streaming HTTP always "works," but may work poorly: data arrives, but with a delay. That's why streaming problems are harder to notice and diagnose.

Why doesn't X-Accel-Buffering: no always help?

The header is understood by nginx and nginx-compatible proxies. A CDN, cloud gateway, or a different type of proxy will ignore it. Moreover, X-Accel-Buffering doesn't affect gzip — compression needs to be disabled separately. And if there's a layer between the application and nginx that strips non-standard headers, it won't reach nginx.

Can a streaming response be compressed?

Technically yes: gzip supports synchronous flush, and the server can compress each chunk with immediate delivery. The compression ratio drops, however, since the dictionary doesn't have time to fill. In practice, compression is disabled for SSE and LLM tokens: saving a few percent of traffic doesn't justify the risk of delays. For streams of large JSON objects, compression with flush can yield a tangible gain.

How to choose between SSE and long-polling?

If the client is a browser, events come more often than once every few seconds, and you control at least the reverse proxy — SSE. If there are uncontrolled nodes with response time limits in the chain, events are rare, or maximum compatibility is needed — long-polling. Mature systems implement both options and switch automatically based on the first connection's result.

Does HTTP/3 affect the buffering problem?

HTTP/3 runs on QUIC and eliminates head-of-line blocking at the transport level, but buffering lives higher — in the proxy's logic. A proxy with HTTP/3 on the front end and HTTP/1.1 on the back end buffers exactly the same way. What does actually change is behavior in mobile networks: QUIC handles network changes and losses better, so streams over HTTP/3 break less often on the client side.

How do you tell that it's the outbound proxy buffering, not the target server?

The "three points" method: the same request directly, through the proxy, and through the proxy to a reference streaming endpoint. Additionally, try HTTPS through CONNECT — if the stream becomes smooth, the proxy was parsing and buffering unencrypted HTTP.

What heartbeat interval should you choose?

Shorter than the shortest timeout in the chain, with a twofold margin. For typical infrastructure — 15-25 seconds, for mobile networks closer to 15. Too frequent a heartbeat means extra traffic and load on thousands of connections; too rare means breaks.

What if the stream needs to live for hours?

Don't do it. Design a server-forced close every 5-15 minutes with a cursor for resumption. This protects against memory accumulation, leaks, and invisible timeouts, and also simplifies deployment: you restart servers without worrying about hour-long connections.

How does a proxy affect detecting client disconnection?

Now there are two connections between the client and server. When the client disconnects, the proxy learns immediately but only informs the server on the next write attempt or per its own rules. The server will only notice the break on write. Hence the practice of regular heartbeat writes for early detection of the break and stopping expensive operations.

Does a mobile proxy need special configuration for streams?

From the user's side, the proxy itself usually isn't configurable. But you need to: prefer HTTPS for a transparent tunnel; align IP rotation with the stream lifecycle; keep the heartbeat at no more than 15-20 seconds; use keep-alive and a connection pool; and verify each pool for transparency before launching to production.

Conclusion: Summary and Next Steps

Streaming HTTP isn't exotic and isn't a separate protocol. It's a regular response that isn't in a hurry to close. That's exactly why it's so vulnerable to intermediate nodes: the proxy sees nothing unusual and applies rules designed for short responses. Buffering delays data, timeouts kill the connection, compression masks buffering — and all of this at eight different points in the chain.

The three mechanisms — chunked, SSE, and long-polling — solve the same problem with different degrees of dependence on proxy transparency. Chunked gives maximum control at the cost of handling it yourself on the client. SSE gives a ready-made event model and reconnection in the browser but requires attention to every node. Long-polling gets through anything but pays for it with the overhead of frequent requests. The choice is determined by four questions: do you control the chain, who's the client, what's the nature of the data, and what's its frequency.

What to do today:

  1. Walk through the map of eight buffering points for your infrastructure and mark which ones you control.
  2. Add X-Accel-Buffering: no and Cache-Control: no-cache, no-transform to streaming route responses — regardless of the current proxy configuration.
  3. Check proxy_http_version, proxy_buffering, gzip, and proxy_read_timeout in nginx or their equivalents in your proxy. Move streaming routes into separate blocks.
  4. Implement a heartbeat at least every 25 seconds and a forced stream close every 5-15 minutes with a cursor.
  5. If you're consuming third-party streams through a proxy — run the "three points" protocol on each address pool and switch streaming requests to HTTPS.
  6. Set up monitoring of time to first byte and intervals between chunks. This is the only way to learn about buffering before your users do.

A proxy isn't the enemy of a stream. A properly configured proxy adds fault tolerance, load distribution, and backend protection. The only task is to explain to it: this response is special, it needs to be forwarded as it comes. Now you have everything to make that explanation in its language.