Introduction: Why Proxy Layer Logs Deserve Their Own Conversation

Picture this. Overnight, your parser's conversion rate drops by a third. In the morning, the security team asks who went where through the corporate proxy gateway the day before yesterday. By lunch, a lawyer wants to know why the archives contain full URLs with client access tokens going back two years. All three questions boil down to a single entity — proxy layer logs. And almost always, it turns out the logs either don't contain what's needed, contain too much, or are stored in a way that makes finding anything in a reasonable amount of time impossible.

We've already covered proxy pool observability: success rate metrics, per-node latency, degradation alerts, dashboards. That was a conversation about how the system feels as a whole. Today's topic is different. A metric will tell you that the 429 error rate climbed from two to fourteen percent. A log will tell you exactly which account, through which mobile carrier IP, with which User-Agent header, and at which second got that response. Metrics answer "what's happening" — logs answer "what exactly happened and to whom." Here we're talking exclusively about record content, cutting sensitive data, file rotation, and retention policy.

Why does this matter right now? First, volumes. An average proxy gateway, even for a small marketing automation team, generates anywhere from a few hundred thousand to tens of millions of requests per day. One carelessly added line in the log format gets multiplied by those millions. Second, regulation. Data protection laws are getting stricter, and the proxy layer is where user identifiers, cookies, and authorization headers pass through in plain text. Third, economics. Storing terabytes of logs in a hot search index costs money, and often that money goes to data nobody will ever query.

From this guide you'll learn: which fields must be present in every proxy log record and why; which data you absolutely cannot write even "just in case"; how to set up masking and hashing so logs remain useful for investigations; how to build rotation and tiered storage; and how to determine a retention period that satisfies engineers, security officers, and lawyers alike. We'll provide ready-made templates, checklists, and break down real cases with numbers. Get comfortable — this one's substantial, but every part of it is battle-tested.

Fundamentals: What a Proxy Layer Log Is and What It's Made Of

Definition and Place in the Architecture

The proxy layer is the component between your application (a parser, an account manager, an ad account automation system) and the outside world. It accepts a request, picks an outbound address from the pool, sends the request, and returns the response. If you're working with mobile proxies, this layer also manages IP rotation, session-to-account binding, and carrier switching.

A proxy layer log is a sequence of records, each describing a single event in this component. Usually the event is a processed HTTP request, but events also include IP changes, connection errors to upstream nodes, requests rejected by an access rule, and process start/stop.

A useful analogy is a ship's logbook. You don't copy the contents of the cargo hold into it, but you record every course change, the time, the coordinates, the weather, and who was on watch. Likewise, a good proxy log doesn't store request bodies but records everything needed to reconstruct the event's path.

Log Types Within the Proxy Layer

In practice there are four streams, and mixing them into one file is a beginner's first mistake:

  • Access log — one record per processed request. The largest stream, the primary source for investigations and billing.
  • Error log — events that need attention: inability to connect to an upstream, timeouts, pool exhaustion, process crashes. Volume is tens to hundreds of times smaller than the access log, but the value per record is higher.
  • Audit log — who changed the configuration, who issued an access token, who added or removed a node from the pool. Written rarely, stored the longest.
  • Rotation/session log — the mobile-proxy-specific stream: when and why the IP changed, which carrier, which session is bound to which account. Without it, you can't answer why an account got hit with a security check.

Logging Levels

Classic levels — DEBUG, INFO, WARN, ERROR, FATAL — apply here too, but with a twist. For the access log, the level is almost meaningless: every request is INFO. Levels make sense for the error log and internal events. The practical rule: in production, the proxy layer runs at INFO; DEBUG is enabled surgically, for a specific client or node, and only for a limited time. Why? Because DEBUG mode in a proxy usually means dumping full headers, which is a direct path to leaking authorization data into files.

Formats: Text vs. Structure

A decade ago, the standard was a text format like Common Log Format, with fields separated by spaces. Today, the industry consensus is unambiguous: structured logs, usually in JSON, where every field is named. The reasons are simple. A structured record can be filtered by any field without regex, it's understood by all collection systems, and — important for our topic — specific fields can be easily removed or masked from it. Try cutting an Authorization header out of a text string where it could be anywhere, and you'll get the picture.

Deep Dive: The Anatomy of a Record and the Nature of Risk

The Three Questions Every Record Must Answer

Before discussing specific fields, let's agree on a criterion. A proxy layer access log record is useful if it can answer three questions:

  1. Who? Which client, account, or profile, which internal service initiated the request.
  2. Through what? Which outbound node, which IP, which carrier, which session was used.
  3. What was the result? Response code, duration, bytes transferred, whether a retry occurred.

Notice: this list doesn't include "what exactly was transmitted." Request and response content is application data, not proxy layer data. The proxy layer is transport, and it should log transport events. This is a fundamental principle of separation of concerns, and most sensitive-data-in-logs problems arise precisely from violating it.

Identifiers and Correlation

The single most valuable field in a modern log is the correlation identifier (request ID or trace ID). It's a unique string the application generates when creating a request and passes to the proxy layer via a service header. The proxy writes it to its log, the application writes it to its own. As a result, you can link an event in the parser, an event in the proxy, and an event in the IP rotation system in seconds. Without correlation, you'd be matching records by timestamp with second-level precision — hopeless when you're dealing with tens of thousands of requests per second.

The second most important identifier is the session identifier. For mobile proxies, a session is the period during which a specific account's requests go through the same IP. It's the session that links the access log to the rotation log. If an account gets blocked, you look at its sessions, see that three IPs from different regions changed within an hour, and understand why.

Categories of Sensitive Data in Proxy Traffic

Let's break down what actually passes through the proxy layer and poses a risk if it lands in logs:

  • Authorization data: Authorization, Proxy-Authorization, Cookie, Set-Cookie headers, X-API-Key and similar. A token in a log equals a token in the open for anyone with log access — and that's usually a much wider circle of people than those with secret access.
  • Personal data in URLs: email addresses in search parameters, phone numbers, user IDs from external platforms, names in paths. This is where data protection requirements kick in, and logs become a data store subject to regulation.
  • The proxy's own credentials: the login and password for proxy authentication that the client sends in a header. A classic leak scenario — the proxy logs the full connection string like user:pass@host in the error log on a failed connection.
  • Request and response bodies: login forms, payment data, private message contents. In a normal configuration, the proxy layer shouldn't see them at all, but the temptation to enable a dump during debugging is strong.
  • Indirect identifiers: the combination of User-Agent, client IP, and exact time allows de-anonymizing a person even without explicit personal data. Regulators in many jurisdictions classify such combinations as personal data.

The Legal Dimension

Precision is needed here. If user data flows through your proxy layer — your service's customers, subscribers, buyers — logs containing their identifiers become personal data processing with all the attendant obligations: a defined purpose, a limited retention period, protection during storage, localization. Logs are not "technical garbage," as they're often perceived. They're a database, just with an unusual structure. The engineering solution here is simple: minimize personal data in logs to the point where it stops being personal — through hashing and tokenization, covered below. That radically reduces the legal burden on logs.

The Economics of a Record

Another dimension — money. Take a hypothetical gateway handling ten million requests per day. A record in compact JSON without extra fields weighs about three hundred bytes. That's three gigabytes a day, about ninety a month. Add the full URL with all parameters and all request headers — the record balloons to one and a half to two kilobytes. That's already twenty gigabytes a day and six hundred a month. If this data sits in a hot search index with replication, the cost difference is an order of magnitude. Every field in the log schema is a financial decision made years in advance.

Method 1: Designing the Record Schema — What You Must Write

The Principle of Minimal Sufficiency

You design a log schema not from the question "what can we write" but from "what questions must we be able to answer." Make a list of investigation scenarios and verify that each one is covered by the available fields. Typical proxy layer scenarios: account block investigation, root-causing a spike in errors, client billing, limit compliance checks, security incident investigation, proving a specific request didn't go through you.

Mandatory Access Log Fields

Below is the core schema we recommend as a minimum for any proxy layer. Each field has an explanation of why it's needed:

  • ts — timestamp in UTC with milliseconds, ISO 8601 format. No local zones: with a distributed pool, nodes sit in different regions, and local time makes correlation impossible.
  • req_id — correlation identifier received from the client or generated by the proxy if the client didn't pass one.
  • client_id — internal identifier of the client or service that authenticated to the proxy. Not a login, not a password, but an internal key.
  • profile_id — identifier of the account or profile the request is made on behalf of, if your system operates such entities. This is the field that links the log to multi-accounting business logic.
  • session_id — proxy session identifier linking the request to a specific IP-binding period.
  • upstream_id — identifier of the outbound node in the pool. Not the IP itself but its internal code: an IP can be reused, a code cannot.
  • upstream_ip_hash — salted hash of the outbound IP. Allows grouping requests by address without storing the address in the clear long-term.
  • carrier and region — carrier and region of the outbound node for mobile proxies. Critical for analyzing how platforms behave toward different carriers.
  • method — HTTP method.
  • host — target domain. The domain, not the full URL.
  • path_template — the path with variable parts replaced, e.g. /users/{id}/posts. Retains analytical value without leaking identifiers.
  • status — response code from the target server.
  • proxy_status — internal processing status: success, timeout to upstream, rejected by rule, pool exhausted. Separates target-server errors from proxy's own errors.
  • duration_ms — total processing time; useful to also break down into connection time, time to first byte, and transfer time.
  • bytes_in and bytes_out — data volume. The basis for billing and anomaly detection.
  • retry — attempt number if the proxy retries.
  • tls — TLS version and whether the handshake succeeded.
  • ua_class — not the User-Agent itself but its class: mobile browser, desktop, app. The full User-Agent, if needed, is stored in a separate lookup table under a short code.

Sample Record

Here's what a record built on this schema looks like:

{"ts":"2026-03-14T08:21:37.412Z","req_id":"01HRX7K9M2P4Q","client_id":"c_0412","profile_id":"p_88213","session_id":"s_7f3a91","upstream_id":"n_msk_017","upstream_ip_hash":"a91f...c2","carrier":"op_a","region":"ru-mow","method":"GET","host":"example-platform.com","path_template":"/api/v2/feed/{id}","status":200,"proxy_status":"ok","duration_ms":842,"ttfb_ms":611,"bytes_in":1204,"bytes_out":48211,"retry":0,"tls":"1.3","ua_class":"mobile_android"}

A bit over three hundred bytes. Not a single secret, not a single personal identifier of external users, but a complete picture of the event. This record can answer all six investigation scenarios from the list above.

Mandatory Rotation Log Fields

For mobile proxies, the rotation log is no less important than the access log. Minimum schema: timestamp, session_id, upstream_id, rotation reason (scheduled, client-requested, error, request limit reached), hash of the old and new IP, carrier, duration of the previous session, request count in the previous session. It's this log that will tell you an account got a check not because of its actions' content but because its IP changed four times in ten minutes.

Step-by-Step Schema Implementation Guide

  1. Write out the investigation scenarios relevant to your team. No fewer than five.
  2. For each scenario, determine the minimum set of fields needed to cover it.
  3. Merge the sets and remove duplicates. That's your draft schema.
  4. Go through each field asking: does it contain a secret, personal data, or an indirect identifier? If yes — replace it with a hash, class, or template.
  5. Fix the schema as a versioned document. Every record should include a schema_version field.
  6. Implement validation: a record that doesn't match the schema should be rejected at the log-assembly stage, not discovered six months later.
  7. Run a test load and measure the average record size. Multiply by daily volume and compare to your storage budget.

Method 2: Cutting Sensitive Data — What You Must Never Write

The Field Blacklist

Let's start with absolute bans. This data must never enter any proxy layer log stream at any level, including DEBUG:

  • Values of Authorization, Proxy-Authorization, Cookie, Set-Cookie headers, any headers with key, token, secret, or session in their names.
  • Proxy connection credentials in any form, including strings like user:pass@host.
  • Request and response bodies. Completely. If you need to inspect a body for debugging, do it on the application side, in temporary storage, with automatic deletion.
  • The full query string with parameters. Parameters regularly contain emails, phone numbers, one-time link tokens.
  • The full User-Agent combined with the client IP — that's a fingerprint that can identify a person.
  • Plaintext IP addresses of your service's end users, if the proxy layer serves more than internal processes.

Four Anonymization Techniques

Banning is easy. The hard part is preserving analytical value. Here's the toolkit that makes it possible:

Redaction

Full replacement of a value with a marker like [REDACTED]. Applied to what has no analytical value: authorization headers, passwords. An important nuance — redact at the record-formation stage, inside the proxy process, not by post-processing files. Post-processing means the secret has already hit the disk and entered buffers, backups, and replicas.

Masking

Partial concealment while preserving the shape. Email i***@d*****.com, phone +1 5** *** 12 34. Lets a support operator recognize the data by comparison but not reconstruct it from the log. Applied where a human visual check is needed, and honestly, that's rare in the proxy layer. More often masking is a compromise when the team isn't ready for hashing.

Salted Hashing

The value is replaced with the result of a cryptographic hash function with a secret salt. Equal values produce equal hashes, so grouping and unique counting are preserved, but reconstructing the original without the salt is impossible. This is the primary tool for IP addresses, external platform user IDs, full User-Agents. Key requirement: the salt is stored separately from the logs, in a secrets manager, and rotated periodically. When the salt rotates, the ability to match old and new hashes is lost — a deliberate price for security, and the salt rotation period usually matches the detailed-log retention period.

Tokenization

The value is replaced with a random token, and the token-to-value mapping is stored in a separate hardened store with strict access control. Unlike hashing, it's reversible — but only for those with access to the mapping table. Applied when an incident investigation may require recovering the original value by decision of a security officer. For example, the full request URL when abuse is suspected. The mapping table lives by its own shorter retention period.

Path Templatization

A separate note on URL paths. The path /users/88213/messages/4471 contains two identifiers. Writing it in full preserves indirect personal data. Writing only the domain loses endpoint analytics. The solution: templatization. Numeric segments, UUIDs, long hash-like strings are replaced with placeholders: /users/{id}/messages/{id}. Analytics by request type are fully preserved, identifiers disappear. Implemented with a set of replacement rules applied at the record-formation stage.

Where Exactly the Cutting Happens

The correct architecture assumes three lines of defense:

  1. In the proxy process: the log formatter knows the schema and simply doesn't take forbidden fields. This is the primary line.
  2. In the collection agent: filtering rules that search for secret patterns (Bearer strings followed by a long sequence, JWT-like constructs, card numbers by the Luhn formula) and replace them with markers. This is insurance against a developer's mistake adding a new field.
  3. In storage: a periodic scanner that searches already-stored data for secret patterns and raises an alarm. This is a detector, not a filter: if it finds something, the first two lines were breached and you need to dig in.

Sensitive Data Cutting Checklist

  • Authorization and cookie headers are redacted inside the process, not by post-processing.
  • Proxy credentials never enter the error log, including connection error messages.
  • Request and response bodies are not logged at any level.
  • The query string is dropped or templatized.
  • IP addresses are salted-hashed; the salt is stored separately and rotated.
  • URL paths are templatized by rules.
  • The User-Agent is replaced with a class or lookup-table code.
  • The collection agent has secret-pattern search rules.
  • Storage is periodically scanned for secrets.
  • DEBUG mode is enabled for a specific client_id, for a limited time, with automatic shutdown.

Method 3: Rotation and Tiered Storage

Why Rotation Is Necessary

Log rotation is the regular closing of the current file and opening a new one, followed by compression, moving, or deleting the old ones. Without rotation, the file grows unbounded, can't be read, compressed, or selectively deleted, and a full disk takes down the proxy process itself. Sounds banal, but in our experience, no fewer than a third of availability incidents in homegrown proxy gateways are caused by disk exhaustion from logs.

Rotation Criteria

A file closes on whichever condition comes first:

  • By size — usually from a hundred megabytes to one gigabyte. Smaller means too many files; larger means slow compression and awkward transfer.
  • By time — hourly for high-load gateways, daily for smaller ones. Time-based rotation makes filenames predictable and simplifies searching.
  • By event — process restart, schema version change. A file shouldn't contain records from two different schemas.

Best practice: rotate by time with a size cap. For example, every hour, but no more than five hundred megabytes per file. The filename includes the node, stream, and start timestamp: access-gw-msk-01-2026031408.jsonl.

Lossless Mechanics

A critical detail often missed: rotation must not cause record loss. The classic mistake is renaming the file and creating a new one while the process keeps writing to the old descriptor. Correct approaches: the process opens a new file itself when the condition triggers (built-in rotation); or an external rotator sends the process a signal to reopen the file after renaming; or — preferable in modern architectures — the process writes to stdout and the collection agent picks up the stream and manages the files itself. The third option removes all disk responsibility from the proxy process.

Compression

Structured JSON logs compress superbly — a typical ratio of eight to fifteen thanks to repeated field names and a limited set of values. That means a gigabyte of raw logs turns into seventy to a hundred megabytes of archive. Algorithm choice: fast algorithms with moderate compression for files that will be read often; maximum compression for archival. Don't compress immediately after closing the file — wait one or two rotation periods. Fresh files are often needed for real-time triage.

The Tiered Storage Model

Now we get to the central idea of economical storage. Not all logs are equally in demand. Access to records is extremely unevenly distributed: the vast majority of log queries concern the last day, a noticeable share concern the last week, and only rare queries concern data older than a month. Storing everything in equally expensive and fast storage is wasteful. Hence the tiered model:

Hot Tier

The last one to three days. A full search index, all fields indexed, query answered in seconds. The most expensive tier per gigabyte. This is where operational investigations, debugging, and fresh incident triage live.

Warm Tier

Three to thirty days. Data in a compressed columnar format in object storage, queried via an engine that reads files directly. Query response in tens of seconds, but storage cost an order of magnitude lower. Weekly reports, client complaint investigations, blocker trend analysis live here.

Cold Tier

Thirty days to end of retention. Compressed archives in the cheapest object storage class, possibly with access latency. Search only by exact time and node, then decompress and manually review. Legal requests, audits, and old incident investigations live here.

Aggregates

A separate tier often forgotten. Before deleting detailed records, compute aggregates from them: request counts, response code distribution, average duration — by client, node, domain, hour. Aggregates take a fraction of a percent of detailed log volume and are stored for years. They're what lets you answer "how was it a year ago" without storing a year of detailed records.

Step-by-Step Rotation and Tier Setup

  1. Decide on the proxy process output stream: stdout to the collection agent is preferable to writing directly to a file.
  2. Configure time-based rotation (hourly or daily) with a size cap.
  3. Set compression to start one or two periods after the file closes.
  4. Split streams: access, error, audit, rotation — into separate files or separate indices.
  5. Configure movement to the warm tier by file age, with conversion to columnar format.
  6. Configure movement to the cold tier and deletion when retention is reached.
  7. Implement daily aggregate computation from the hot tier and write them to a separate long-lived store.
  8. Set an alert on node free disk space and on collection-agent send lag: if logs aren't leaving, they're piling up locally.
  9. Quarterly, verify that deletion actually happens: count the oldest file in each tier.

Method 4: Retention Policy

Three Forces That Determine Retention

Log retention is a compromise between three forces pulling in different directions. Engineers want to keep longer: you never know when you'll need to dig into an old incident. Finance wants to keep less: every gigabyte costs money. Security and legal — the most interesting side: they simultaneously demand keeping long enough for investigations and short enough not to accumulate data you'll be liable for if it leaks. Every record with personal data you store longer than necessary is a liability, not an asset.

The Principle: Retention Depends on Sensitivity and Demand

Different streams and different fields live for different durations. That's the key idea of this section. There is no single "log retention period" — there's a retention matrix:

  • Detailed access log: fourteen to ninety days. Demand decays exponentially, sensitivity is moderate even after anonymization.
  • Error log: thirty to a hundred and eighty days. Small volume, high value for recurring-issue analysis.
  • Audit log: one to three years. Who changed configs and issued accesses — data needed in investigations long afterward. Volume is negligible.
  • Rotation log: as long as the access log — they're meaningless without each other.
  • Tokenization table: shorter than the logs themselves — seven to thirty days. After that, tokens in logs become irreversible.
  • Aggregates: two to five years. No personal data, small footprint, valuable for long-term analytics.

How to Calculate Retention for Your Situation

Here's a working framework of five questions:

  1. What's the longest realistic investigation scenario? A client complains about an account block that happened three weeks ago — so detailed logs are needed for at least twenty-one days plus margin.
  2. Are there external obligations? Client contracts may require keeping access logs for a certain period to resolve disputed billing situations. Regulatory requirements for operators in certain data categories may also set minimums.
  3. What data remains sensitive after anonymization? If salted hashing with rotation is applied, sensitivity after rotation drops nearly to zero, and retention can be extended.
  4. How much does storage cost at each tier? Calculate the monthly cost of hot, warm, and cold storage for your volume. Often it turns out cold-storing a year of logs costs less than a week of hot.
  5. What's lost when you delete? If aggregates cover long-term analytics, detailed records can be confidently deleted.

Legal Hold

A mandatory policy element is a deletion suspension mechanism (legal hold). If an incident investigation begins or an official request is received, automatic deletion of the affected period must be frozen until the matter is resolved. Technically, it's a flag on a file or object storage prefix that the deletion process checks. Organizationally, it's a procedure defining who can set and lift this flag. Without such a mechanism, you'll find yourself with the needed data deleted by schedule on the very day it was needed.

Documenting the Policy

A retention policy is a document, not a setting in a config file. Minimum content: a list of log streams and fields; for each — collection purpose, retention at each tier, anonymization method; those responsible for the policy and its execution; the deletion-suspension procedure; the verification procedure that deletion happens; review frequency — usually yearly or on schema change. This document is the first thing an auditor will ask for and the first thing that protects you in an incident: it shows the data was processed thoughtfully.

Retention Matrix Template

Fill in a table for your case with these columns: stream, fields with sensitivity, anonymization method, hot tier (days), warm tier (days), cold tier (days), aggregates (months), retention basis, owner. One row per stream. A complete matrix fits on one page and settles ninety percent of disputes between teams.

Common Mistakes: What Not to Do

Logging Everything "Just in Case"

The most common and most expensive mistake. The argument "better to have it and not use it" doesn't work with logs: unused data costs money, creates leak risk, and turns storage into a place where the needed can't be found among the unneeded. Every field must answer a specific question from your scenario list.

Enabling DEBUG in Production and Forgetting to Turn It Off

Proxy DEBUG mode almost always means dumping headers. Turned on Friday evening to debug one issue, by Monday it has filled the disk with authorization tokens from all clients. Rule: DEBUG is enabled by filter and with an auto-off timer.

Writing Secrets into Error Messages

A developer forms a connection error message and pastes the full connection string into it — login and password included. The error log is stored longer than the access log and viewed by more people. Check all error-message formation sites for interpolation of credentials.

Mixing Streams into One File

Access, error, and audit in one file mean audit records that should live three years get deleted with access records after a month, and access records that should be compressed lie intermingled with rare error records that need human review.

Storing Time in a Local Zone

Pool nodes in different regions, daylight saving transitions in some jurisdictions, comparison with external systems' logs — all of this breaks on local time. UTC only, ISO 8601 only.

Anonymizing by Post-Processing

A script that runs overnight through files cutting out tokens is an admission that tokens lay on disk in the clear all day, and also entered collection agent buffers, replicas, and possibly backups. Anonymization belongs at the record-formation stage.

Not Verifying That Deletion Works

The policy is written, retention is set, automation is configured — and two years later you discover that due to a bucket permissions change, deletion silently failed and storage holds the entire history. Checking the oldest file in each tier is a quarterly ritual.

Storing the Hash Salt Next to the Logs

A salt in a proxy config file sitting in the same repository or on the same disk is no salt at all. Hashed IPs then get recovered by brute force in minutes. The salt belongs in a secrets manager, with restricted access and rotation.

Logging the Full URL

The query string is a concentrate of sensitive data: email in search parameters, one-time link tokens, user IDs. Domain plus a templatized path is enough for any transport analytics.

Tools and Resources

Proxy Servers and Their Logging Capabilities

Most mature proxy servers and reverse proxies support configurable access log formats, including JSON output, and allow excluding or overriding fields via configuration. When choosing, look at three things: the ability to output a structured log without an intermediate text format; the ability to add a client correlation header to the record; the ability to fully prohibit logging of specific headers at the configuration level, not the code level. If your proxy layer is homegrown — not uncommon for mobile proxy management systems — use your language's structured logging library with record-processor support. That's where you implement redaction and hashing.

Collection Agents

The collection agent is a process on the node that reads the proxy's output, applies filters, and sends records to storage. Key selection criteria: support for regex replacement rules for the second line of defense; disk buffering when storage is unavailable, so records aren't lost; low resource consumption, since the agent lives next to the proxy process on the same node; support for routing different streams to different destinations.

Storage and Search Engines

For the hot tier, search indices with field full-text search or columnar analytical databases work — the latter have been displacing classic indices in recent years thanks to better compression and aggregation speed. For warm and cold tiers, object storage with a columnar file format and a query engine reading them directly. Choose object storage with lifecycle policies: automatic movement between storage classes and age-based deletion are configured, not coded.

Anonymization Tools

A separate class of tools — secret scanners, originally built to find keys in code repositories. Their pattern engines apply to logs too: they find tokens, cloud provider keys, JWT-like strings. Configured for periodic runs over storage, such a scanner becomes the third line of defense. For path templatization and hashing there are no specialized tools — it's a few dozen lines in the log processor.

Secrets Management

The hashing salt and tokenization table must live in a secrets manager with access logging. This can be a cloud provider solution or a self-hosted secrets store. The criterion: the ability to grant the proxy process access to the salt without a human having access to its value.

What to Read and What to Follow

Your reference points: industry secure-logging guides from applied security communities, which provide exhaustive lists of data forbidden from logs; regulator documentation on personal data minimization; cloud providers' tiered storage guidance, which includes cost calculations. Separately useful: distributed tracing specification materials — that's where to take your correlation identifier format from.

Cases and Results

Case 1: Marketing Automation Team Cuts Log Volume Sixfold

Initial situation: a gateway for working with ad accounts through a mobile proxy pool, about four million requests per day. Logs written in text format with the full URL, all request headers, and the first kilobyte of response body "for debugging." Average record about two kilobytes, daily volume eight gigabytes, all in a hot search index with ninety-day retention. Index cost — a noticeable line in the infrastructure budget.

What they did: migrated to the twenty-field JSON schema described above. Headers, body, and query string dropped. Paths templatized. Introduced tiers: three days hot, thirty warm, ninety cold. Added hourly aggregates with three-year retention.

Result: average record three hundred twenty bytes, daily volume 1.3 gigabytes, about a hundred megabytes per day after warm-tier compression. Storage cost dropped roughly eighty-five percent. An unexpected side effect: incident triage speed increased because analysts stopped drowning in headers and immediately saw the key fields. And two months later, a secret scanner deployed as the third line found more than twelve thousand active ad account tokens in old archives — they were revoked and reissued.

Case 2: Marketplace Parsing Service Investigates Blocks via the Rotation Log

Situation: a team scrapes public product listings through mobile proxies and faces a wave of session blocks. Metrics show an error uptick but not the cause. The access log exists, the rotation log doesn't.

What they did: added a rotation log with the IP change reason and session duration, linked it to the access log via session_id. Within two days, the data showed it: blocked sessions overwhelmingly had durations under three minutes — rotation fired on an aggressive timer, and the target platform saw the same profile from three different IPs in a short interval. Sessions longer than twenty minutes were blocked three times less often.

Result: after reconfiguring rotation, the share of blocked sessions dropped from eleven to under three percent. Cost: about forty bytes per rotation record, ten thousand records per day — a volume that's invisible next to the access log. The conclusion we consider important: for mobile proxies, the rotation log is not an option, it's a mandatory stream.

Case 3: Platform with External Clients Brings Logs into Personal Data Compliance

Situation: a service provides clients with proxy access through which the clients' end users' requests flow. Logs contain end-user IPs, full User-Agents, full URLs. A legal audit classifies the logs as a personal data database with a one-year retention.

What they did: IPs replaced with salted hashes rotated every thirty days; User-Agent replaced with a lookup-table code; URLs replaced with domain and path template. Compiled a retention matrix with justifications. Introduced a deletion-suspension mechanism. Detailed logs capped at forty-five days, aggregates at three years.

Result: after salt rotation, logs older than thirty days stopped containing data that could identify a user even indirectly and were removed from personal data regulation. The compliance burden shrank while the ability to investigate incidents from the last month was fully preserved. The audit accepted the retention matrix as sufficient evidence of thoughtful processing.

Case 4: The Disk Overflow Incident

Brief but telling. A homegrown proxy manager wrote logs to a file without rotation. In four months the file grew to forty gigabytes, the node's disk filled, the process crashed on a write attempt, and a pool of twenty modems went down for three hours at night. After the incident: output to stdout, a collection agent with buffering, hourly rotation, a free-space alert. No repeats. The fix cost one workday.

FAQ: Hard Questions About Proxy Layer Logs

Should I log successful requests or is logging errors enough?

You should. Successful requests are the basis for billing, for computing error shares (without a denominator, a share is meaningless), for analyzing how platforms treat different carriers, and for proving a specific request didn't go through you. Save on record size, not record count. If volume is truly unacceptable, apply sampling: log all errors and every tenth successful request with a sampling flag, so you can multiply by the factor during aggregation.

How do I investigate an incident if the IP is hashed and the salt has already rotated?

That's a deliberate boundary. After salt rotation, the data becomes irreversible — that's its protective value. Investigations requiring IP recovery must fit within the salt's active period, usually matching detailed retention. If investigation scenarios regularly exceed that period, extend the salt rotation period, but understand that you're thereby extending how long the logs stay sensitive. For genuinely long investigations, use the deletion-suspension mechanism and salts simultaneously.

How do proxy layer logs differ from tracing?

Tracing describes one request's path across multiple services with nested time intervals. A log describes an event in a single component. They complement each other via a shared correlation identifier: from the trace you see the request spent eight hundred milliseconds in the proxy layer; from the log you see why: five hundred on connecting to upstream, two hundred on TLS, one hundred on transfer. Don't try to replace one with the other.

Should I encrypt logs at rest?

Yes, at all tiers. Even anonymized logs contain business information: which domains you request, at what intensity, through which carriers. Storage-side encryption is a standard object storage setting, enabled with a flag. Additionally, restrict hot-tier access by role: analysts see aggregates and anonymized fields, security officers see everything.

What about logs when operating in multiple regions?

Record time in UTC and the node's region as a separate field. Storage itself may be centralized or regional depending on data localization requirements: if users' data from a particular jurisdiction flows through the proxy, logs with their anonymized identifiers may fall under a requirement to be stored in that jurisdiction. Check with your lawyers before designing, not after.

Can I use proxy logs for client billing?

You can and should, with two caveats. First: billing is built on aggregates, not detailed records — sums by client_id per period. Aggregates live longer and contain nothing sensitive. Second: if the log is the basis for settling with clients, it's subject to immutability requirements — records aren't edited after the fact, files carry checksums, and write access belongs only to the collection process.

How do I make sure a new field added by a developer doesn't contain a secret?

Process-wise: any log schema change goes through review with a mandatory sensitivity question for every new field, and the schema version is incremented. Technically: the second line of defense in the collection agent catches secret patterns regardless of field name; the third — the storage scanner — signals if something slipped through. Additionally useful: a test that runs a request with a known fake token through the log formatter and verifies the token didn't appear in the output.

What do I do with logs when a node is removed from the pool?

The node's logs stay in storage until retention expires — they describe events that happened, and deleting them along with the node breaks the integrity of the history. Local buffers on the node must be flushed to storage before it's decommissioned, so the decommission procedure includes waiting for the collection agent's buffer to drain. The node identifier isn't reused for new nodes, otherwise records from two different devices get mixed together.

How do I assess whether the retention policy is adequate?

Quarterly, run three checks. First: take the last five incidents and verify that for each one, the needed data was available in the right tier. Second: count how many log queries in the quarter hit data older than thirty days; if zero — the cold tier can be shortened. Third: verify that the oldest file in each tier is no older than the set period. If all three checks pass, the policy is adequate.

Conclusion: Logs as a Designed System, Not a Byproduct

Let's gather the essentials. Proxy layer logs are not something that "writes itself" — they're a designed system with its own schema, its own boundaries, and its own lifecycle. It answers the questions "who," "through what," and "what was the result," and deliberately doesn't answer "what was transmitted" — that's the application's responsibility.

What to write: UTC timestamp, correlation identifier, client, profile and session IDs, node code, hashed IP, carrier and region, method, domain and path template, response and internal status codes, durations, volumes, attempt number. For mobile proxies — a mandatory rotation log with the IP change reason.

What not to write: authorization headers and cookies, proxy credentials, request and response bodies, full URLs with parameters, full User-Agents paired with IPs, plaintext end-user addresses. Anonymization happens inside the process, with two additional lines of defense in the collection agent and storage scanner.

How to store: separate streams, time-based rotation with a size cap, compression, tiers from hot to cold with aggregates at the exit, a retention matrix by stream, a deletion-suspension mechanism, and regular verification that deletion happens.

Next steps we recommend doing this week. First: pull a hundred random records from your current log and run them through the sensitive-data checklist — the result will probably surprise you. Second: measure the average record size and multiply by daily volume and retention; compare with what you'd get under the schema in this guide. Third: draft a one-page retention matrix and show it to your security and legal teams — it's a conversation better had before an incident than after.

Logs are your infrastructure's memory. Good memory remembers what matters, forgets what doesn't, and never stores what can hurt. Design it that way.