Proxy Pool Observability: Metrics, Logs, Alerts, and Diagnosing Degradation in a Minute

Introduction: Without Metrics, the Problem Looks Like "Everything's Slow"

A familiar scene. A message pops up in the work chat: "The proxies are slow." Five minutes later, a second one: "And there are lots of errors." Then begins what engineers privately call archaeology. Someone opens a console and manually runs curl through three random addresses. Someone else stares at a CPU load graph that has nothing to do with proxies. Someone writes to the provider's support with a message along the lines of "something's wrong on your end," without a single number in hand. An hour passes. It turns out only one segment of the pool degraded, only in one country, and it started not "just now" but two days ago—nobody just noticed.

The difference between a team that spends an hour on this kind of story and a team that spends a minute isn't experience or talent. The difference is that the second team decided in advance what to measure, what to write to logs, and what to alert on. When you have a chart of success rate broken down by country and carrier, the phrase "everything's slow" turns into a concrete statement: "Since 14:32, p95 time-to-first-byte on mobile proxies of carrier A in region B has grown from 1.8 to 6.4 seconds, the error rate hasn't changed, and the retry rate has tripled." With that kind of statement, you can actually work: redistribute load, open a ticket with the provider backed by data rather than emotions, or figure out the problem isn't with the proxies at all but with the target resource.

In this guide, we'll walk through observability around a proxy pool end to end. You'll learn which four signals give you almost all the useful information, which fields must appear in every log line and which should never appear, why average response time lies and how to calculate percentiles correctly, how to label data by dimensions so you can spot degradation in a single slice, how to build alerts that fire rarely but meaningfully, and how to distinguish three typical problem patterns on a dashboard in a minute. At the end, we'll cover a minimal tech stack, real cases with numbers, and answer common questions.

Let's set boundaries right away. We won't discuss in detail choosing a specific IP from the pool, health-checking proxies, or quarantining faulty addresses here: that's covered in a separate article. This piece is only about two things: measurement and alerting. Everything described below applies to legitimate engineering scenarios: monitoring prices on public marketplaces, verifying ad display across regions, testing geo-dependent content of your own services, and collecting open data in compliance with resource policies.

A Bit of Fundamentals: What Observability Means for Proxies

The classic definition says that observability is a property of a system that lets you understand its internal state from external outputs. Translated into engineering: you can ask the system a new question you didn't anticipate and get an answer without deploying new code. Monitoring answers known questions ("is the service alive?"), while observability answers unknown ones ("why did requests to this domain through mobile addresses become slower specifically on Tuesdays?").

The three pillars of observability are well known: metrics (numeric time series, cheap to store, easy to aggregate), logs (events with context, expensive to store but rich in detail), and traces (the path of a single request through components). For a proxy pool, the picture is a bit specific. Between your code and the target site sits external infrastructure you don't control: the exit address, the carrier, the route to the destination. You can't install an agent on a mobile tower. So the only point where you measure everything is your own client sending the request through the proxy. Every request through the pool should leave two traces behind: a metrics increment and a structured log line. Traces are useful but overkill for most teams, and we'll only come back to them in the FAQ.

Why does this matter specifically for proxies? Because proxy degradation almost never looks like an outage. A server not responding is simple. A proxy, on the other hand, is "a bit slower," "slightly more likely to return connection errors," "sometimes drops the download halfway through." Each of these symptoms individually is below a human's attention threshold. Together, they double the cost of data collection and break deadlines. Slow degradation can only be caught through measurement.

Four Signals for Proxies: Success, Latency, Retries, Traffic

In reliability engineering, there's the concept of "golden signals": latency, traffic, errors, and saturation. For a proxy pool, we adapt this idea and get four signals—enough for 90% of situations. Why four and not twenty? Because each additional chart on a dashboard reduces the chance that the on-call engineer looks at the right one. Four signals fit on one row of the screen and can be read in ten seconds.

Signal 1: Success Rate

Success rate is the ratio of requests that completed with a useful result to the total number of requests over a time window. The key word is "useful." For a proxy, success isn't just HTTP 200. Success is when the target resource responded with what you expected, and the response arrived in full. A connection dropped mid-body isn't a success, even if the headers carried a 200 code. A 403 from the target site isn't a success either, but it's probably not the proxy's fault, and we distinguish that through outcome classification.

Rule of thumb: classify the outcome of every request into a small number of categories. For example: ok (expected response received), proxy_connect_error (couldn't establish a connection to the proxy), proxy_auth_error (proxy rejected authentication), upstream_timeout (proxy responded but the target resource didn't respond in time), upstream_4xx, upstream_5xx, truncated (connection dropped before the end of the body). This classification is the foundation for all further analysis. The success rate is calculated as ok over total, but you also watch the share of each error category separately, because they point to different culprits.

Signal 2: Latency by Percentiles

For proxies, it makes sense to measure latency at three points: time to establish the connection to the proxy, time to first byte from the target resource (TTFB), and total request time. The first tells you about the proxy node's availability. The second is about the route quality from proxy to target and the target's response speed. The third is about channel throughput. If only the third grows while the first two stay stable, the problem is bandwidth, not availability. If the first grows, the problem is between you and the proxy. Latency must be measured in percentiles, not averages, and there's a dedicated section on that below.

Signal 3: Retry Rate

Retry rate is the most underrated signal. Most clients are written so that on error they automatically retry the request through a different address. This is right from a reliability standpoint and terrible from an observability standpoint: the success rate stays at 99%, but under the hood every request is executed three times. You pay triple for traffic, the target resource gets triple the load, and on the dashboard everything is green. Retry rate exposes hidden degradation before it becomes visible degradation. It's calculated as the ratio of attempts to logical requests: a value of 1.0 means no retries, 1.5 means every other request was retried.

Signal 4: Traffic Consumption

Traffic consumption is a signal directly tied to money, because many pricing plans—including Proxeon's—bill by data volume. But it also has a diagnostic role. A sharp increase in bytes per successful request means the target resource started serving different pages: heavy stub pages, redirects to other sections, changed markup. A sharp decrease means that instead of full pages, you're getting short error responses. Count traffic in both directions and always normalize by the number of successful requests: "kilobytes per successful request" is far more informative than "gigabytes per hour."

What This Looks Like in Code

Below is a minimal set of metrics for Prometheus in Python. Note the label set: pool, country, operator, proxy type. The specific proxy address is deliberately absent from labels, and we'll explain why in the dimensions section.

from prometheus_client import Counter, Histogram

LABELS = ["pool", "country", "operator", "proxy_type"]

REQUESTS = Counter("proxy_requests_total", "Логические запросы через пул", LABELS + ["outcome"])
ATTEMPTS = Counter("proxy_attempts_total", "Попытки, включая ретраи", LABELS + ["attempt_kind"])
TTFB = Histogram("proxy_ttfb_seconds", "Время до первого байта", LABELS, buckets=(0.1, 0.25, 0.5, 1, 2, 4, 8, 16, 30))
CONNECT = Histogram("proxy_connect_seconds", "Установление соединения с прокси", LABELS, buckets=(0.05, 0.1, 0.25, 0.5, 1, 2, 5))
BYTES = Counter("proxy_bytes_total", "Байты через пул", LABELS + ["direction"])

def observe(ctx, outcome, ttfb, connect, bytes_in, bytes_out, attempt):
base = dict(pool=ctx.pool, country=ctx.country, operator=ctx.operator, proxy_type=ctx.proxy_type)
kind = "first" if attempt == 1 else "retry"
ATTEMPTS.labels(**base, attempt_kind=kind).inc()
if outcome in ("ok", "upstream_4xx", "upstream_5xx", "truncated", "upstream_timeout"):
TTFB.labels(**base).observe(ttfb)
CONNECT.labels(**base).observe(connect)
BYTES.labels(**base, direction="in").inc(bytes_in)
BYTES.labels(**base, direction="out").inc(bytes_out)
if outcome == "ok" or attempt == ctx.max_attempts:
REQUESTS.labels(**base, outcome=outcome).inc()

Note a detail: the logical requests counter increments only on success or on exhausting attempts. This way, the success rate reflects the outcome for the user of your code, while the attempts counter reflects the actual work done by the pool. The difference between them is the retry rate.

Table: Metric, What Its Growth Means, First Action

What to Log per Request—and What Not to Log

Metrics answer "what's happening," logs answer "with what exactly." When the dashboard shows a growth in errors in the slice "country X, carrier Y," you need to figure out whether all addresses in this slice are to blame or just two out of thirty. Metrics with a per-address label would blow up storage, but logs won't—because you don't continuously aggregate logs, you filter them on demand.

Required Log Line Fields

Every request attempt through the pool should produce one structured record, preferably in JSON format. Here's the set of fields we consider minimally sufficient:

• ts — timestamp in UTC with milliseconds. Not local time. Never.
• request_id — a logical request identifier shared across all its attempts. Lets you pull the entire history of one request with a single command.
• attempt and max_attempts — attempt number and the limit. Without the attempt number, you can't tell the first error from the third.
• proxy_id — an internal proxy identifier. Not an address with port and not a connection string. An internal ID you can map to the real address through a separate secured table.
• proxy_type, country, operator, pool — the same dimensions as in metrics. This lets you jump between dashboard and logs without recalculation.
• target_host — the target domain name. Not the full URL: query strings often carry tokens and personal data.
• method — HTTP method.
• status — HTTP response code, if received, otherwise null.
• outcome — that very outcome classification discussed above.
• error_class — exception class or short transport error code if outcome isn't ok.
• connect_ms, ttfb_ms, total_ms — three latencies in milliseconds.
• bytes_in, bytes_out — response body size and request size.
• session_id — if you use sticky sessions, the session identifier; this lets you see that errors concentrate in long-lived sessions.
• retry_reason — for attempts starting from the second: the outcome of the previous attempt.

Example line:

{"ts":"2026-03-14T11:32:07.412Z","request_id":"7f3a9c1e","attempt":2,"max_attempts":3,"retry_reason":"upstream_timeout","proxy_id":"px-2841","proxy_type":"mobile","country":"RU","operator":"op_a","pool":"pricing","target_host":"shop.example","method":"GET","status":200,"outcome":"ok","error_class":null,"connect_ms":184,"ttfb_ms":1420,"total_ms":2310,"bytes_in":48211,"bytes_out":612,"session_id":"s-19b"}

One such line weighs about 400 bytes. At 50 requests per second, that's roughly 1.7 gigabytes per day uncompressed and 200–300 megabytes compressed. Quite manageable even on a modest budget.

What Must Never Appear in Logs

Here we move from engineering to responsibility. The list is short, but every item matters:

1. Proxy access credentials. The most common leak is a connection string like http://login:password@host:port that ends up in the log in full because a developer logged the config "for debugging." Logs get copied into tickets, forwarded to colleagues, stored for years. Credentials in them are credentials available to anyone who has ever seen the log.
2. The Proxy-Authorization header and any other authorization headers, including those for the target resource.
3. Response body. It's huge, it may contain third parties' personal data, and it's almost never needed for proxy diagnostics. If you need to know what the target served, save a hash of the body and its length; the response itself goes to separate storage with a short retention period and only on a sample.
4. Cookies in either direction.
5. Full URL with query parameters. Host and, if necessary, path without parameters are enough.
6. Real proxy address in plaintext if the log goes to shared storage. An internal ID is sufficient.

A practice we recommend: implement redaction at the level of a single function through which every proxy-related record passes. Don't rely on developers' discipline.

import re

CRED_IN_URL = re.compile(r"(https?|socks5h?)://[^/@:]+:[^/@]+@")
SENSITIVE_HEADERS = {"proxy-authorization", "authorization", "cookie", "set-cookie"}

def redact(record: dict) -> dict:
for key, value in list(record.items()):
if key.lower() in SENSITIVE_HEADERS:
record[key] = "[redacted]"
elif isinstance(value, str):
record[key] = CRED_IN_URL.sub(lambda m: m.group(1) + "://[redacted]@", value)
record.pop("body", None)
return record

Sampling and Storage

Do you need to log every attempt? Errors—yes, all of them. Successful attempts can be sampled: at intensities above a hundred requests per second, 5–10% of successful lines is enough to estimate latencies, because metrics already contain the full picture. Importantly, the sampling decision must be made by request_id, not randomly per attempt: otherwise you'll get a second attempt without the first. Retention for detailed logs is usually 7–14 days; aggregates and metrics live longer.

Percentiles Instead of Averages: Why the Average Hides the Problem

Imagine a pool of a hundred addresses. Ninety of them respond in 0.8 seconds. Ten respond in 9 seconds because they landed on an overloaded base station. The average response time is 1.62 seconds. A week ago, when everything was fine, the average was 1.4. A 15% increase. Who would react to that? Nobody. Meanwhile, every tenth of your requests waits nine seconds, and if you have an 8-second timeout, every tenth request fails with an error.

The arithmetic mean describes bell-shaped distributions well. Latency through a proxy is never bell-shaped. It's a long-tailed distribution: most requests are fast, but some are very slow, and those determine user experience and cost. Moreover, latency through a pool is often bimodal: there's a "normal" hump and a "problematic addresses" hump. The average lands between them, at a point where almost no real requests exist.

What p50, p95, p99 Mean

A percentile is the value below which a given share of observations falls. p50 (median)—half the requests are faster than this value. p95—95% are faster, 5% slower. p99—only one percent are slower. For a proxy pool, we recommend looking at three percentiles with different purposes:

• p50 shows the "typical" request. If p50 grows, most requests degraded. This is a rare and serious signal.
• p95—the working metric for alerts. It's sensitive to problematic segments (in the example above, p95 immediately shows 9 seconds) but resilient to single outliers.
• p99—the metric for timeouts and capacity planning. Your timeout should be slightly above the "healthy" p99, otherwise you'll cut off normal requests and turn them into retries.

How to Calculate Accurately

If your data volume is small (thousands of values in a window), calculate exactly: sort and take the element by index.

def percentile(values, p):
if not values:
return None
ordered = sorted(values)
k = (len(ordered) - 1) * p / 100.0
lo = int(k)
hi = min(lo + 1, len(ordered) - 1)
return ordered[lo] + (ordered[hi] - ordered[lo]) * (k - lo)

ttfb_window = [0.8, 0.9, 0.7, 9.1, 0.85, 8.7, 0.78, 0.92, 0.81, 0.79]
print(percentile(ttfb_window, 50), percentile(ttfb_window, 95))

Exact calculation doesn't scale: to compute p95 over a week across millions of requests, you'd need to store all values. That's why production systems use approximate structures.

How to Calculate at Scale: Histograms

Prometheus and compatible systems store a histogram as a set of per-bucket counters: how many observations were below 0.1s, below 0.25s, below 0.5s, and so on. The percentile is reconstructed by linear interpolation within a bucket. Accuracy depends on how well the bucket boundaries are chosen. For proxy latencies, a logarithmic scale from 100 milliseconds to 30 seconds works well: 0.1; 0.25; 0.5; 1; 2; 4; 8; 16; 30. If your healthy p95 is around 2 seconds and the timeout is 15, this grid gives an error within 20–30% on the tail, which is enough for alerts and diagnostics.

histogram_quantile(0.95,
 sum by (le, country, operator) (
rate(proxy_ttfb_seconds_bucket{pool="pricing"}[5m])
 )
)

Note the sum by (le, ...) inside. This is a critically important detail: histograms from different instances of your client must be summed first, then the quantile computed. The reverse operation—computing p95 per instance and averaging—is mathematically meaningless. The average of percentiles is not a percentile. This is one of the most common mistakes, and it can hide a problem just as reliably as a plain average.

If you need high tail accuracy (for example, you sell SLAs on latency to internal customers), use structures like t-digest or HDR histograms, which deliver sub-percent accuracy in fixed memory. For most teams, standard histograms suffice.

Breakdown by Dimensions: Country, Carrier, Proxy Type

Here's a scenario that illustrates why dimensions matter. You have a pool of 400 addresses; 5% of them are mobile addresses of one carrier in one region. That segment fully degraded: success rate there fell to 10%. What will the overall success rate chart for the pool show? A drop from 99% to 94.5%. Noticeable but not critical. The on-call engineer thinks "the network is acting up" and moves on. Meanwhile, the client consuming data from that region gets nothing.

Dimension breakdown (called labels in Prometheus terminology) solves exactly this problem: it lets you decompose the aggregate into slices and see that the problem is localized. But labels have a cost, and you need to understand it.

Which Dimensions to Choose

A good dimension has two properties: it has a small, bounded number of values, and its breakdown answers a real diagnostic question. For a proxy pool, we recommend four dimensions in metrics:

• proxy_type—mobile, residential, datacenter. Different types degrade differently and cost differently.
• country—the exit address country. Usually 1 to 30 values.
• operator—the carrier or autonomous system through which the address exits. Problems are often localized right here: one overloaded base station or one bad peering on the carrier's side.
• pool—the logical purpose: which product or task uses this segment. Lets you separate incidents by business impact.

A fifth dimension worth considering is target_host, the target host. It's needed to distinguish proxy problems from target problems. But if you hit thousands of different domains, cardinality explodes. The compromise: keep only the top-20 hosts by volume in the label, and collapse the rest into an other value.

Cardinality Arithmetic

Each unique combination of label values is a separate time series. Let's count: 3 types × 10 countries × 15 carriers × 4 pools × 7 outcomes = 12,600 series just for the requests counter. A histogram with 9 buckets multiplies its series count by 10. Total: around 50–60 thousand series. That's comfortable for Prometheus. Now imagine adding a label with the proxy address (400 values). The series count grows 400-fold—up to 20 million. That's no longer "comfortable," that's storage death.

Hence the rule: the specific proxy address lives in logs, not in metrics. Metrics answer "which slice degraded," logs answer "which addresses inside the slice." This division of labor gives both speed and detail without explosive cost growth.

Queries That Find the Slice

The main diagnostic question: "where is it worst?" It's answered by sorting slices by success rate:

bottomk(5,
 sum by (country, operator, proxy_type) (rate(proxy_requests_total{outcome="ok"}[10m]))
 /
 sum by (country, operator, proxy_type) (rate(proxy_requests_total[10m]))
)

The second question: "is this a new problem or has it always been like this?" It's answered by comparison with the past:

(
 sum by (operator) (rate(proxy_requests_total{outcome="ok"}[10m])) / sum by (operator) (rate(proxy_requests_total[10m]))
)
-
(
 sum by (operator) (rate(proxy_requests_total{outcome="ok"}[10m] offset 1d)) / sum by (operator) (rate(proxy_requests_total[10m] offset 1d))
)

A negative value shows how much worse the slice got compared to a day ago. Such a chart on a dashboard turns "something somewhere is wrong" into "carrier B in country C is down 12 points versus yesterday."

Where Dimension Values Come From

Country and proxy type are usually known from the pool config. The carrier can be obtained from the provider's metadata: for example, in Proxeon's panel and API, addresses are tagged with geography and carrier info that you pull when initializing the pool and cache alongside the internal proxy ID. Don't determine the carrier at request time through external databases: it adds latency and creates an unnecessary dependency.

Alerts That Don't Make Noise: Thresholds, Windows, Hysteresis

The worst thing that can happen to an alerting system isn't a missed incident. The worst is when the on-call engineer gets 40 notifications a day, 39 of which mean nothing, and stops reading the fortieth. An alert that's ignored is worse than no alert at all, because it creates an illusion of control.

Principle 1: Alert on Symptoms, Not Causes

Don't alert on "proxy px-2841 returned an error." An individual address has the right to be sick; that's what the quarantine mechanism inside the pool handles, which we're not discussing here. Alert on what the consumer of your data feels: pool success rate below target, p95 above target, retries above target. You'll find the cause from the dashboard in a minute if you've done the dimensions right. The alert's job is to wake you up, not to explain.

Principle 2: Minimum Traffic Volume

A 50% success rate on two requests in five minutes is one failed request, not an incident. Any ratio rule must include a condition on the minimum number of requests in the window. Otherwise, at night when traffic drops, you'll drown in false positives.

Principle 3: Windows and Error Budget Burn Rate

If you've defined a target reliability level (say, 98% successful requests per month), you have an error budget: 2% of requests. A good alert reacts to the rate of budget consumption, not the instantaneous value. The practice that's become the reliability industry standard is multi-window alerts: a short window (5 minutes) catches sharp drops, a long window (1 hour) catches slow degradation, and both must be exceeded simultaneously to exclude brief spikes.

Principle 4: Hysteresis

Hysteresis means different thresholds for firing and resolving an alert. If the alert fires when the success rate is below 95% and resolves when it returns above 95%, a metric oscillating around 95% will generate a series of "fired—resolved—fired" notifications every few minutes. The fix: fire at 95%, resolve at 97%. Or, more simply, require the condition to hold continuously for a specified duration (in Prometheus, the for keyword), and also set a resolve delay.

Principle 5: Severity Levels and Routing

Not everything needs to be a phone call. Split alerts into two or three levels: critical (impacting data consumers right now—call or push to the on-call), warnings (slice degradation, retry growth, traffic anomaly—post to a channel, handled during business hours), informational (an instance not reporting data, approaching a traffic limit—daily digest).

Example Rule Set

groups:
- name: proxy-pool
 rules:
 - record: pool:success_ratio:5m
expr: sum by (pool) (rate(proxy_requests_total{outcome="ok"}[5m])) / sum by (pool) (rate(proxy_requests_total[5m]))
 - record: pool:success_ratio:1h
expr: sum by (pool) (rate(proxy_requests_total{outcome="ok"}[1h])) / sum by (pool) (rate(proxy_requests_total[1h]))
 - record: pool:request_rate:5m
expr: sum by (pool) (rate(proxy_requests_total[5m]))

 - alert: ProxyPoolSuccessLow
expr: pool:success_ratio:5m < 0.95 and pool:success_ratio:1h < 0.97 and pool:request_rate:5m > 0.5
for: 5m
labels: { severity: critical }
annotations:
 summary: "Доля успеха по пулу {{ $labels.pool }} ниже цели"

 - alert: ProxyPoolLatencyHigh
expr: histogram_quantile(0.95, sum by (le, pool) (rate(proxy_ttfb_seconds_bucket[10m]))) > 4 and pool:request_rate:5m > 0.5
for: 10m
labels: { severity: warning }

 - alert: ProxyPoolRetryRateHigh
expr: sum by (pool) (rate(proxy_attempts_total[15m])) / sum by (pool) (rate(proxy_requests_total[15m])) > 1.4
for: 15m
labels: { severity: warning }

 - alert: ProxyPoolSliceDegraded
expr: (sum by (pool, country, operator) (rate(proxy_requests_total{outcome="ok"}[15m])) / sum by (pool, country, operator) (rate(proxy_requests_total[15m]))) < 0.85 and sum by (pool, country, operator) (rate(proxy_requests_total[15m])) > 0.2
for: 15m
labels: { severity: warning }

 - alert: ProxyPoolNoData
expr: absent(proxy_requests_total) or pool:request_rate:5m == 0
for: 10m
labels: { severity: warning }

Note the last rule. Missing data is the most insidious failure: the client crashed, there are no metrics, charts are flat and green, and everyone's calm. An alert on silence is mandatory.

Checklist for Every New Alert

1. Does the annotation state what the recipient should do? If not, the rule isn't ready.
2. Is there a condition on minimum traffic?
3. Is there a for delay or hysteresis?
4. Has the rule been tested against last month's history: how many times would it have fired and how many of those were real incidents?
5. Is it clear what severity level it belongs to and where it's routed?
6. Is there a feedback process: if the alert fired and turned out to be false, who and when reviews it?

Quick Dashboard Diagnostics: Three Typical Patterns

A dashboard isn't a gallery of charts, it's a decision-making tool. A good proxy pool dashboard is structured as a funnel. Top row: four signals for the whole pool over the last 6 hours with yesterday overlaid as a thin line. Second row: the same signals broken down by country, carrier, and type—as a table or heatmap sorted by worst value. Third row: success rate and TTFB by top-10 target hosts. Fourth: the ratio of connect time, TTFB, and total time. Plus vertical markers for deploys and pool config changes.

With such a dashboard, diagnostics boils down to answering three questions: "did the whole pool degrade or a slice?", "did all targets degrade or one?", "which of the three latencies grew?" The combinations of answers yield three typical patterns.

Pattern 1: All Slices Degraded Simultaneously

All countries, all carriers, all proxy types show growing errors or latency synchronously, to the minute. External providers and carriers don't break this way: they have no shared failure point on your side. So the problem is yours: your outbound channel, your DNS, file descriptor or local port exhaustion, a recent deploy that changed timeouts or the connection pool size, CPU contention with another process on the same machine. First action: check the deploy marker and the proxy connect time. If connect time grew and TTFB (minus connect) is stable, the problem is between you and the proxy nodes, and it's almost certainly on your side or your network provider's side.

Pattern 2: One Slice Degraded

One carrier in one country, or one proxy type, shows a drop while other slices are stable. This is the most common pattern. The cause is on the carrier's side (network segment overload, scheduled maintenance) or in a segment of the proxy provider's infrastructure. First action: temporarily reduce this slice's weight in load distribution (the mechanism is beyond this article's scope) and report it to the provider's support with concrete data: the slice, start time, metrics before and after, and example request_ids. Proxeon support responds to these faster than to "everything's bad on your end," simply because it receives ready localization. Second action: check the logs for the slice to see whether all addresses within it degraded or just a few.

Pattern 3: Success Rate Dropped for One Target Host, Latencies Normal

All pool slices still perform well across all hosts except one. For this host, the share of upstream_4xx is growing, or the bytes-per-request volume changed. The proxies aren't at fault here at all. The target resource changed its behavior: introduced rate limiting, changed page structure, started serving different content for some requests. First action: reduce request intensity to that resource and verify that your collection complies with its usage policy. Second: check the content on a sample and update response validation. Absolutely do not increase retries or rotate addresses more often in this situation—it will only increase load on the resource and your traffic costs.

Pattern 4 (Bonus): Retries Grow, Success Holds

Success rate is 99%, latencies are tolerable, but the retry rate grew from 1.1 to 1.8. This is hidden degradation. It's already costing you money and will become visible degradation as soon as the retry budget runs out. First action: check which outcome triggered the retries (the retry_reason field in logs), then follow one of the first three patterns.

60-Second Diagnostic Checklist

1. Seconds 0–10: top row of the dashboard. Which of the four signals diverged from yesterday's line? When exactly?
2. Seconds 10–25: slice table. One slice red or all? If all—Pattern 1, go to deploy markers and your own infrastructure.
3. Seconds 25–40: target host table. One host or all? If one—Pattern 3, proxies aren't at fault.
4. Seconds 40–50: latency ratios. Did connect, TTFB, or total time grow?
5. Seconds 50–60: formulate one sentence: "Since [time], in slice [slice] for [target], [metric] grew from [before] to [after], presumably due to [hypothesis]." This sentence goes to the incident channel and the provider ticket.

Typical Mistakes That Nullify the Value of Observability

We've seen dozens of monitoring rollouts around proxies. The mistakes repeat with astonishing consistency.

• Average response time as the primary metric. Already covered: the average hides the tail, and the tail is where the problem lives.
• Averaging percentiles across instances. Sum histograms, not pre-computed p95s.
• Proxy address as a metric label. Cardinality explosion and storage failure within a month.
• Access credentials in logs. One "for debug" line—and a password sits in shared log storage for a year.
• Retries counted as success. If you only count the final outcome, you'll never see that the pool is working three times as hard as it needs to.
• 429 and 403 from the target resource logged as proxy errors. This is a classification error that sends the investigation in the wrong direction and damages the relationship with the provider.
• One latency instead of three. Without separately measuring connect, TTFB, and total time, you can't distinguish an availability problem from a bandwidth problem.
• Alert without minimum traffic. Nighttime noise and loss of trust in alerting.
• Alert on every individual address. That's a job for quarantine, not for human notification.
• No alert on missing data. The client crashes—charts stay green.
• Local time in logs. Correlating with provider data and colleagues in another timezone becomes torture.
• Dashboard without deploy markers. Half of incidents coincide with a code or config change. Without a marker, you'll look outside when the cause is inside.
• Storing detailed logs forever. Expensive and pointless: after two weeks, details aren't needed, and aggregates already live in metrics.

Minimal Stack Without Overengineering

Observability easily turns into a project for its own sake. A three-person team spins up a distributed message queue, a columnar database cluster, and a tracing system—for a client doing 20 requests per second. Don't do that. The stack should match the scale.

Level 1: One Service, Up to 50 Requests per Second

Structured JSON logs to a file or stdout and one script that aggregates them once a minute and writes the four signals into a simple database or even the same log—that's enough. The dashboard is any tool that can plot a chart from a table. Alerts are a cron script that compares aggregates against thresholds and posts a message to a messenger. This stack takes a day to set up and covers 80% of needs.

Level 2: Multiple Services, 50–1000 Requests per Second

The standard kit: Prometheus for metrics, Grafana for dashboards, Alertmanager for alert routing, Loki or a similar log store with label-based indexing. Clients export metrics via a built-in HTTP endpoint and write JSON logs to stdout; an agent collects them. Recording rules pre-compute success ratios so dashboards open instantly. This is the configuration described in the examples above, and it covers the vast majority of teams.

Level 3: Dozens of Services, Thousands of Requests per Second

Here it makes sense to adopt OpenTelemetry as a unified collection layer, a columnar log store capable of arbitrary analytical queries over billions of rows, and possibly tracing for complex chains. But even at this level, the four signals and log structure don't change. Only transport and storage change.

Provider Data as a Complement

The proxy provider sees its side: traffic consumption per port, active connection count, last rotation time. Proxeon's panel and API expose this statistics, and it's useful to pull it alongside your own metrics—at least for traffic reconciliation: if your byte counter and the provider's counter diverge by more than 10–15%, you have a measurement problem (most often headers, TLS handshakes, or truncated responses aren't being counted). But provider data doesn't replace yours: only you know which request was successful from the task's perspective.

Client Wrapper as a Single Measurement Point

Regardless of stack level, all measurements should pass through one wrapper over the HTTP client. No "we'll measure here but forget there." Schematically:

import time, json, sys, requests

def fetch_via_proxy(ctx, url, timeout=(5, 20)):
t0 = time.perf_counter()
outcome, status, err = "ok", None, None
ttfb = connect = total = 0.0
bytes_in = 0
try:
session = ctx.session # уже настроена на прокси по внутреннему proxy_id
resp = session.get(url, timeout=timeout, stream=True)
ttfb = time.perf_counter() - t0
status = resp.status_code
body = resp.content
bytes_in = len(body)
total = time.perf_counter() - t0
if status >= 500: outcome = "upstream_5xx"
elif status >= 400: outcome = "upstream_4xx"
elif not ctx.validate(body): outcome = "invalid_content"
except requests.exceptions.ProxyError as e:
outcome, err = "proxy_connect_error", type(e).__name__
except requests.exceptions.ReadTimeout as e:
outcome, err = "upstream_timeout", type(e).__name__
except requests.exceptions.ChunkedEncodingError as e:
outcome, err = "truncated", type(e).__name__
finally:
total = total or (time.perf_counter() - t0)
observe(ctx, outcome, ttfb, connect, bytes_in, len(url), ctx.attempt)
record = redact({"ts": ctx.now_utc(), "request_id": ctx.request_id, "attempt": ctx.attempt,
"proxy_id": ctx.proxy_id, "proxy_type": ctx.proxy_type, "country": ctx.country,
"operator": ctx.operator, "pool": ctx.pool, "target_host": ctx.host(url),
"status": status, "outcome": outcome, "error_class": err,
"ttfb_ms": int(ttfb * 1000), "total_ms": int(total * 1000), "bytes_in": bytes_in})
sys.stdout.write(json.dumps(record, ensure_ascii=False) + chr(10))
return outcome, status

Note the invalid_content outcome: a 200 response that failed your validation (for example, a page without the expected data block) is not a success. It's precisely these responses that most often mask a change in the target resource's behavior.

Case Studies: What Observability Looks Like in Action

Case 1: One Carrier, One Country, Minus Two Hours of Downtime per Week

A price-monitoring team for public marketplaces was working through a pool of roughly 300 mobile and residential addresses at about 40 requests per second. Before adding operator labeling, they saw one thing: overall p95 TTFB occasionally spiked from 1.6 to 3–4 seconds for several hours, and the success rate dipped by 3–5 points. Investigating each such episode took one to two hours and usually ended with "it went away on its own." After adding country and operator labels, the very first episode was localized in two minutes: mobile addresses of one carrier in one region, with p95 there at 11 seconds versus a norm of 2. The remaining 85% of the pool was perfectly fine. The team reduced this slice's weight in load distribution, sent the provider a report with timeframes and metrics, and a few hours later received confirmation of scheduled maintenance at the carrier. Result for the quarter: diagnostic time dropped from 90 minutes to 5, the number of episodes didn't change, but their impact on overall p95 became indistinguishable.

Case 2: Hidden Retries and +38% to the Traffic Bill

Another team discovered a 38% traffic consumption increase over a month with an unchanged task count and a stable 98.5% success rate. The first hypothesis was heavier target pages. The bytes-per-successful-request metric refuted it: 46 kilobytes, same as before. But the retry rate—which nobody had been watching—grew from 1.08 to 1.52. Analysis of the retry_reason field in logs showed that almost all retries were triggered by the upstream_timeout outcome for one target host that had become slower to respond but fit within the timeout on the second attempt. The proxies weren't the cause at all. The team raised the read timeout for that host from 8 to 15 seconds (the healthy p99 was 11), reduced request intensity to it, and brought traffic consumption back to the previous level in three days. Moral: retry rate is a metric about money.

Case 3: From 200 Notifications a Week to Six

A third team inherited an alerting system with the rule "success rate for any slice over 1 minute below 90%" with no traffic constraint. At night, when some slices processed two or three requests per minute, the rule fired dozens of times. The on-call engineers muted the sound. One real incident—a whole segment's availability dropping for four hours—was noticed in the morning through complaints. Redesigning the rules took one day: a minimum traffic condition was added, two windows (5 minutes and 1 hour) were introduced, a for delay of 10 minutes was added, slices were moved to the warning level, and only a single critical was left at the pool level. Result: about six notifications per week, four or five of which required real action. The on-call engineers turned the sound back on.

FAQ: Common Questions About Proxy Pool Observability

Do I need tracing if I have metrics and logs?

For most teams—no. A request through a proxy doesn't pass through ten microservices; it has one hop. The trace's role here is played by request_id in logs: with it, you gather all attempts of one request. Tracing becomes useful when the proxy client is just one part of a long chain (task queue, parser, validator, database loader) and you need to understand exactly where time is lost.

How do I measure TTFB if the HTTP client doesn't expose it directly?

Most clients return control after receiving headers, if you request streaming (stream=True or equivalent). The moment of return is the TTFB approximation. Connect time is more accurately measured at the transport level: many libraries have hooks for connection events, and some expose request phase information directly. If there are no hooks, you can measure the time to send the first request byte with a separate low-level measurement, but for practical purposes the streaming-read approximation is sufficient.

How do I distinguish a proxy error from a target resource error?

By where it occurs. If the connection to the proxy wasn't established, or the proxy returned its own code (for example, 407 for auth problems, or a specific gateway response), that's a proxy error. If the proxy accepted the request and you then received a code from the target server or a timeout waiting for a response, that's an error between proxy and target, or on the target itself. Classify at this boundary, then compare across slices: target errors are visible across all slices simultaneously, proxy errors—only for specific ones.

What's a healthy retry rate?

Depends on the goals and proxy type. For mobile addresses, 1.05–1.15 is normal: wireless networks are inherently less predictable. For datacenter, 1.01–1.05. More important than the absolute value is its stability: a 30% increase relative to the weekly baseline is a reason to investigate, even if the absolute value still looks modest.

How do I pick alert thresholds with no history?

Start by observing without alerts: collect two weeks of data, look at the distribution of success rate and p95 by day and hour. Set the critical threshold so that over those two weeks it would have fired at most once or twice, and both times you'd agree it was an incident. Set the warning threshold so it fires several times a week. Then review every month: an alert that never led to action should either be tightened or deleted.

Should I store the response body at least for errors?

For transport errors, there's no body. For responses with unexpected content (invalid_content), it's useful to save the first few hundred bytes, or a hash and length, in separate storage with a one- to two-day retention, with mandatory personal data scrubbing. Storing full bodies isn't needed: they're heavy, they may contain third parties' data, and for proxy diagnostics it's almost always enough to know what type of page came back.

How do I reconcile my traffic accounting with the provider's data?

Count bytes in both directions, including headers, and compare daily sums by port or session against what the provider's panel shows. A 5–10% divergence is normal: you're not accounting for TLS overhead and truncated connections. A divergence above 20% is a reason to check whether you're only counting the response body and losing attempts that terminated with an exception before reading the body.

How long does it take to implement this approach from scratch?

For a single client with an existing HTTP wrapper: one day for metrics and log structure, one day for the dashboard, one day for alert rules, and another two weeks of observation before enabling critical alerts. If you have multiple clients in different languages, most of the time goes into agreeing on a unified set of outcomes and labels—it's important not to rush here, because renaming a metric after a year of accumulated history is very painful.

What if the provider data lacks carrier info for some addresses?

Introduce an unknown value and watch its share. If unknown exceeds 10–15% of the pool, your carrier breakdowns lose meaning—request complete metadata from the provider or manually tag addresses once when adding them to the pool.

Conclusion: One Minute Instead of an Hour

Back to the scene from the beginning. "The proxies are slow" is a phrase that contains no information. Everything described in this guide serves one purpose: to turn it into a sentence with numbers, a slice, a time, and a hypothesis. Four signals—success rate, percentile latency, retry rate, traffic consumption—give a picture of the pool's state. A structured log with request_id, proxy_id, attempt number, status code, three latencies, and size gives investigation details. Percentiles instead of averages expose the tail where real problems live. Breakdowns by country, carrier, and type localize degradation to a slice. Alerts with minimum traffic, two windows, and hysteresis fire rarely and meaningfully. A funnel dashboard answers three questions in a minute.

Where to start tomorrow? First step—a client wrapper with unified outcome classification and a single record function. Second—a TTFB histogram and an attempts counter with labels for country, carrier, and type. Third—a log redaction function that everything passes through without exception. Fourth—a dashboard with four charts and one slice table. Fifth—two weeks of observation and only then alerts. A month from now, you'll look at the retry rate chart and, quite possibly, be surprised: the pool was working noticeably harder than you thought. And a quarter from now, the phrase "everything's slow" will disappear from your chat—not because problems stop happening, but because you'll call each one by name.