Kubernetes Egress Traffic and Proxies: Pod Env, Sidecar, Egress Gateway, and the Role of NO_PROXY

Introduction: Why "configure it like a regular server" doesn't work in a cluster

On a regular VM, setting up a proxy takes five minutes. You add HTTP_PROXY to your profile, throw in a couple of exclusions, restart the service, and go grab a coffee. In Kubernetes, the same scenario ends differently: half your services suddenly return 502, health checks start killing pods, and the payment service complains it can't reach the pod next door. Meanwhile, the external requests you actually set all this up for? They work just fine.

Why? Because in a cluster, a single process lives in two network worlds at once. The first is the external internet: partner APIs, marketplaces, data sources for scraping, public services. The second is the cluster's internal network: other services via ClusterIP, the Kubernetes API server, DNS, databases, queues, metrics. On a regular server, the second world is nearly empty, so a global proxy variable breaks nothing. In a cluster, the second world is enormous, and any proxy that receives a request to something like payments.default.svc.cluster.local simply can't resolve it. The proxy sits outside and knows nothing about your internal DNS.

Add to that Kubernetes' own quirks. Pod environment variables don't affect kubelet or the container runtime. Images often come with their own config that ignores the environment. SDKs in different languages each have their own idea of what an exception to proxying means. And all of that happens against a backdrop where the pod's egress IP is invisible to the outside world: what goes out is the node address or the cloud NAT gateway.

In this guide, we'll break down Kubernetes egress traffic layer by layer: how a packet physically leaves a pod, where exactly you can slip a proxy in, why NO_PROXY in a cluster goes from a formality to a critical architectural element, how to pass credentials safely, when a sidecar is justified and when it's overkill, and how to debug failures without guessing. We won't rehash basic environment variable syntax — only the cluster-specific stuff. Every manifest snippet is production-ready; feel free to use them as a starting point.

Fundamentals: How pod egress traffic physically works

Before setting up a proxy, it helps to understand the path a packet takes without one. Otherwise, troubleshooting turns into guesswork.

From container to node

Each pod gets its own IP from the pod CIDR, say 10.244.3.17. Inside the pod, all containers share one network namespace: one IP, one set of interfaces, a common localhost. This is the key fact for the sidecar approach we'll get back to. A packet from a container exits through the pod's virtual interface onto the node, where the CNI plugin takes over. Then there are two scenarios.

• Destination inside the cluster. If the app calls a service ClusterIP (say 10.96.14.2), kube-proxy or the CNI's eBPF dataplane rewrites the destination address to the IP of a specific backend pod. Traffic goes to the right node over the overlay or directly via L3 and never leaves the cluster.
• Destination outside. If the address belongs to neither the pod CIDR nor the service CIDR, the packet takes the node's default route. On the way out of the node, SNAT almost always kicks in: the source address 10.244.3.17 is replaced with the node's IP. In clouds, a NAT gateway may sit on top of that, in which case the outside world sees its address.

Here's the first practical takeaway: an external service never sees the pod's IP. It sees the node IP or the cloud NAT. If you have node autoscaling, egress IPs change along with the nodes. That's why IP-based proxy authorization without a dedicated egress node works poorly in a cluster — more on that below.

DNS inside the cluster

Pods use cluster DNS (usually CoreDNS). The pod's resolv.conf lists search domains: namespace.svc.cluster.local, svc.cluster.local, cluster.local. That's why the name payments resolves to payments.default.svc.cluster.local. An external proxy has no clue about any of this. It'll get a request to the host payments, try to resolve it via public DNS, and return an error. This is the fundamental reason NO_PROXY matters more in a cluster than on any standalone server.

What "setting up a proxy" in a cluster actually means

Engineers mean three different things by this phrase, and confusing them is dangerous:

1. Make a specific application go outside through a proxy server (say, through dedicated Proxeon addresses) without touching other services.
2. Provide a stable, predictable egress IP for the whole cluster or part of it — a task for corporate networking, compliance, and auditing.
3. Centralize control over external traffic: logging, rate limits, a single exit point.

The first task is solved at the pod level. The second and third are solved at the cluster level. Mixing levels causes most of the problems we'll discuss.

Three levels where you can place a proxy: container, sidecar, egress gateway

Picture a cluster as an office building. You can hand each employee their own proxy (container), seat an assistant next to every workstation (sidecar), or put one guard post at the building exit (egress gateway). Each has its own cost.

Level 1: The container itself

The app receives the proxy address through environment variables, a config file, or launch flags, and routes its own traffic through it. This is the simplest and most common approach.

Pros:

• Zero extra infrastructure: changes only in the Deployment manifest.
• Granularity: one service needs a proxy, its neighbor doesn't.
• The proxy "understands" the protocol: an HTTP proxy sees the destination host and can log and apply rules.

Cons:

• Only works if the code or library respects the settings. Java, some Node.js clients, and certain gRPC implementations need separate configuration.
• Total dependence on a correct NO_PROXY: one mistake and intra-cluster communication is broken.
• Proxy credentials end up in the process environment and are available to anyone who can exec into the container.
• Rotating the proxy or changing its address requires recreating pods.

Level 2: The sidecar container

A second container runs next to the app in the same pod — a local forward proxy. The app sends traffic to localhost:3128 (or it's intercepted by iptables rules), and the sidecar decides where and how to forward it: through which upstream proxy, with what auth, with what timeout.

Pros:

• The app knows nothing about external credentials: they live only in the sidecar.
• A single point of logic: rotation, retries, address pool selection, metrics — all without touching app code.
• Transparent mode via iptables works even with libraries that ignore HTTP_PROXY.
• Updating proxy logic means updating one sidecar image.

Cons:

• An extra container in every pod: memory, CPU, one more object to monitor.
• Transparent interception requires NET_ADMIN in an init container, which isn't always allowed by security policies.
• Harder to debug: two places where things can break.

Level 3: Cluster egress gateway

A dedicated set of pods (or nodes) becomes the single exit point. Other pods either explicitly go through the gateway's internal Service or are forced to do so by network policies and routing. The gateway then goes out either directly or through an external proxy.

Pros:

• One stable egress IP (or a small set) for the whole cluster. Perfect for IP-based auth on the proxy provider's side and for partner allowlists.
• Centralized logs, rate limits, auditing.
• Apps store no credentials: they only see the gateway's internal address.
• External address rotation happens in one place.

Cons:

• A single point of failure: the gateway needs to be made fault-tolerant (multiple replicas, PodDisruptionBudget, anti-affinity).
• An extra network hop and a potential bandwidth bottleneck.
• Requires well-thought-out NetworkPolicy, otherwise there's a "hole" with direct egress bypassing the gateway.
• Granularity is harder: if different teams need different egress addresses, you'll have to spin up multiple gateways.

How to choose

A practical rule we use when designing:

• One or two apps, code under your control, standard HTTP clients — container level.
• Many services with different stacks, requirements for rotation and retries, unwillingness to hand out credentials — sidecar.
• Compliance requirements, IP-based auth, auditing of all external traffic, dozens of teams — egress gateway, often combined with a sidecar for fine-grained logic.

Note that these levels aren't mutually exclusive. A typical mature setup is an egress gateway as a mandatory boundary plus environment variables or a sidecar where specifics are needed.

Environment variables in the manifest: HTTP_PROXY, HTTPS_PROXY, NO_PROXY and why a wrong NO_PROXY breaks the cluster

You already know the variable syntax. What interests us is what happens to them inside a pod and why the standard NO_PROXY value of "localhost,127.0.0.1" is catastrophically insufficient here.

Anatomy of a failure

Let's walk through the chain. The orders service calls payments at http://payments:8080/charge. HTTP_PROXY is set in the environment. The HTTP client looks at the destination host, doesn't find it in NO_PROXY, and instead of a TCP connection to payments, it opens a connection to the proxy and sends it the request GET http://payments:8080/charge. The proxy tries to resolve the name payments through its own resolver. No such name exists. The proxy returns 502 or 503. Your service gets an error from the proxy, but in the logs it looks like an error from payments. The team runs off to fix payments, which is perfectly healthy.

There's a second, sneakier option. If the service calls a ClusterIP directly at an address like 10.96.14.2, the proxy gets a request to a private address. Some proxies drop such requests immediately; others try to connect to 10.96.14.2 on their own network. In the worst case, some other host with that address exists there.

A third option concerns Kubernetes itself. Client libraries that talk to the API server (operators, controllers, apps with leader election) use the address kubernetes.default.svc or the IP in the KUBERNETES_SERVICE_HOST variable. Without an exception, they'll also go through the proxy and lose contact with the control plane.

What must be in NO_PROXY for a pod

The minimum set we recommend as a starting point. Plug in your own CIDRs — you can find them in the cluster config or in service annotations:

• localhost, 127.0.0.1, ::1 — local addresses, including communication with the sidecar.
• .svc, .svc.cluster.local, .cluster.local — all service names. A leading dot means "any subdomain."
• kubernetes.default.svc, kubernetes.default.svc.cluster.local — explicitly, for API clients.
• Service CIDR, e.g., 10.96.0.0/12 — direct calls to ClusterIP.
• Pod CIDR, e.g., 10.244.0.0/16 — headless services and pod-to-pod calls.
• Node CIDR — if the app calls node ports (metrics, hostNetwork services).
• 169.254.169.254 — cloud metadata, if used.
• Internal company domains, e.g., .corp.local, if there's a direct route to them from the cluster.
• Internal DB and queue addresses if they're outside the cluster but on the private network.

A trap not everyone knows: not every client supports CIDR

This is where the real cluster-specific pain begins. The NO_PROXY format is not standardized. Every library parses it differently:

• Go (net/http) understands CIDR notation, leading-dot suffixes, and individual hosts. This is the best case, and most Kubernetes tools are written in Go.
• curl supports CIDR only from version 7.86 onward. In older images based on outdated distros, the entry 10.96.0.0/12 will simply be ignored.
• Python (urllib, requests) doesn't understand CIDR at all. Comparison is by string suffix. The entry 10.96.0.0/12 will never work; meanwhile 10.96. will only match hosts ending in that string, so it doesn't help either.
• Node.js ignores environment variables entirely in its built-in HTTP client. You need ProxyAgent in undici or a third-party agent, and it's up to that agent how to interpret NO_PROXY.
• Java uses its own http.nonProxyHosts variable with wildcard syntax like *.svc.cluster.local and a pipe separator.

The practical takeaway: if you have a zoo of languages in the cluster, the only reliable way to exclude internal traffic by CIDR is not to rely on the client's NO_PROXY, but to solve it at the sidecar or gateway level, where the parser is under your control. For DNS names, dotted suffixes work almost everywhere, so apps should call services by name, not by ClusterIP.

Variable case

Another one that regularly eats hours of debugging. curl reads http_proxy only in lowercase: it ignores HTTP_PROXY on purpose, due to a historical CGI vulnerability. But HTTPS_PROXY and NO_PROXY are accepted by curl in both cases. Go, Python, and .NET read both forms. That's why in our manifests we always set both — it's cheap and eliminates a whole class of problems.

A working snippet: a ConfigMap with NO_PROXY and a Deployment

Put NO_PROXY in a namespace-level ConfigMap. That way you change the list in one place instead of twenty Deployments:

apiVersion: v1
kind: ConfigMap
metadata:
 name: egress-proxy-common
 namespace: crawlers
data:
 NO_PROXY: "localhost,127.0.0.1,::1,.svc,.svc.cluster.local,.cluster.local,kubernetes.default.svc,10.96.0.0/12,10.244.0.0/16,192.168.10.0/24,169.254.169.254,.corp.local"
 no_proxy: "localhost,127.0.0.1,::1,.svc,.svc.cluster.local,.cluster.local,kubernetes.default.svc,10.96.0.0/12,10.244.0.0/16,192.168.10.0/24,169.254.169.254,.corp.local"

And the Deployment that uses it. The proxy address here has no credentials yet — we'll cover credential storage in a dedicated section:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: price-collector
 namespace: crawlers
spec:
 replicas: 3
 selector:
matchLabels:
 app: price-collector
 template:
metadata:
 labels:
app: price-collector
spec:
 containers:
 - name: app
image: registry.internal/price-collector:2.7.0
envFrom:
- configMapRef:
name: egress-proxy-common
env:
- name: HTTP_PROXY
 value: "http://proxy-host.example:8080"
- name: http_proxy
 value: "http://proxy-host.example:8080"
- name: HTTPS_PROXY
 value: "http://proxy-host.example:8080"
- name: https_proxy
 value: "http://proxy-host.example:8080"
readinessProbe:
 httpGet:
path: /healthz
port: 8080

Note the scheme in HTTPS_PROXY: most often it's http://, because the client connects to the proxy itself over plain TCP and only then establishes a CONNECT tunnel for HTTPS requests. The value https:// here means TLS to the proxy itself, which far from all clients and proxies support. Check your Proxeon dashboard for what connection type your plan provides.

Make sure the probe doesn't go through the proxy

Liveness and readiness probes are executed by kubelet, not the app, so they don't see environment variables and don't get proxied. But if your app inside the /healthz handler itself calls dependencies by internal addresses, those calls will hit HTTP_PROXY. That's exactly what the classic symptom "after adding the proxy, pods stopped passing readiness" looks like.

What does NOT pick up these variables: kubelet, images with their own config, SDKs

Pod environment variables live inside the app process. Anything outside of it, or anything that deliberately ignores the environment, needs separate setup. Let's look at three groups.

Kubelet and container runtime

The most common question when moving a cluster behind a proxy: "why aren't images pulling, we set HTTP_PROXY in the Deployment." The answer: images aren't pulled by the pod, they're pulled by the container runtime (containerd or CRI-O) on the node, on kubelet's orders. These are system services running under systemd, and they only read their own environment.

Configuration is done via a systemd drop-in file on each node. For containerd it looks like this (file /etc/systemd/system/containerd.service.d/http-proxy.conf):

[Service]
Environment="HTTP_PROXY=http://proxy-host.example:8080"
Environment="HTTPS_PROXY=http://proxy-host.example:8080"
Environment="NO_PROXY=localhost,127.0.0.1,10.96.0.0/12,10.244.0.0/16,192.168.10.0/24,.svc,.cluster.local,registry.internal"

Then systemctl daemon-reload and systemctl restart containerd. NO_PROXY must include the internal registry, otherwise the runtime will pull your own images through the external proxy. A separate drop-in for kubelet is needed less often — only if it itself calls cloud APIs or external webhooks.

In managed clusters you may not have node access, and then runtime configuration is done through the cloud provider's mechanisms or via a DaemonSet with a privileged container that rewrites the host config. The second option is a compromise worth coordinating with the security team.

Images with their own configuration

Many off-the-shelf images ship a config file that overrides or ignores the environment:

• JVM apps. Java doesn't read HTTP_PROXY. You need system properties. It's convenient to pass them through JAVA_TOOL_OPTIONS, which the JVM picks up without touching launch scripts:

env:
- name: JAVA_TOOL_OPTIONS
 value: "-Dhttp.proxyHost=proxy-host.example -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy-host.example -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=localhost|127.0.0.1|*.svc|*.svc.cluster.local|*.cluster.local|10.*|192.168.*"

• Images with their own HTTP client config (some monitoring agents, connectors, ETL tools) read the proxy from their own YAML or TOML. They see the environment, but don't use it. Read the image's docs and look for keys like proxy_url.
• Images with a hardcoded DNS resolver inside the app break the whole idea of name-based NO_PROXY: the app resolves the name itself, gets an IP, and then compares the IP against your list of domain suffixes.
• Entrypoint scripts that override variables. This is rarer, but worth checking: kubectl exec pod -- cat /proc/1/environ | tr null-byte to newline shows the actual environment of the main process.

Individual SDKs and protocols

The list is long here, so we'll cover the most common:

• gRPC. Many implementations support HTTP_PROXY only through CONNECT and with caveats; some ignore NO_PROXY. For gRPC traffic inside the cluster, this is doubly dangerous.
• Node.js. The built-in fetch and the http/https module don't read environment variables. An explicit agent is needed. axios reads HTTP_PROXY through its dependency, but native fetch doesn't. In a single app, two clients can behave differently.
• Cloud SDKs. Most respect HTTP_PROXY, but calls to the metadata endpoint (169.254.169.254) must be in the exceptions, otherwise fetching temporary credentials will fail.
• WebSocket and raw TCP. Clients for databases, message brokers, and Redis use their own protocols over TCP and don't go through an HTTP proxy at all. If such traffic must go through a proxy, you need SOCKS mode or transparent interception at the sidecar level.
• Custom DNS clients in apps (resolving through a separate library) don't interact with the proxy and go directly over UDP 53. This is fine, but you need to remember it when writing NetworkPolicy.

The takeaway of this section: before rolling out a proxy on a service, map its outbound connections by protocol and library. Half an hour of inventory saves a day of debugging.

Secrets: Proxy credentials via Secret, not in the manifest

A proxy address with a login and password in the body of a Deployment is a leak waiting to happen. Manifests live in git, end up in CI logs, and show up in kubectl get deploy -o yaml for anyone with read access to the namespace. Let's do it right.

Option 1: The full proxy URL in a Secret

The simplest scheme: a Secret stores a ready-made string with credentials, and the app receives it as an environment variable. We create the secret outside git (via CI variables, an external secrets manager, or manually):

kubectl -n crawlers create secret generic egress-proxy-creds --from-literal=HTTP_PROXY="http://user123:s3cretPass@proxy-host.example:8080" --from-literal=HTTPS_PROXY="http://user123:s3cretPass@proxy-host.example:8080" --from-literal=http_proxy="http://user123:s3cretPass@proxy-host.example:8080" --from-literal=https_proxy="http://user123:s3cretPass@proxy-host.example:8080"

In the Deployment, replace explicit envs with a reference to the Secret and the ConfigMap with NO_PROXY:

containers:
- name: app
 image: registry.internal/price-collector:2.7.0
 envFrom:
 - configMapRef:
 name: egress-proxy-common
 - secretRef:
 name: egress-proxy-creds

An important detail: if the login or password contains special characters (@, :, /, %), they must be URL-encoded. A password with an @ character isn't parsed correctly and the client tries to connect to the wrong host. In the Proxeon dashboard, it's convenient to choose a password without special characters when generating credentials, or copy the ready-made URL.

Option 2: Separate fields assembled into a URL inside the container

When login and password are stored separately (for example, synced from an external secrets store), the URL is assembled in the entrypoint or in the app itself:

env:
- name: PROXY_USER
 valueFrom:
secretKeyRef:
 name: egress-proxy-creds
 key: username
- name: PROXY_PASS
 valueFrom:
secretKeyRef:
 name: egress-proxy-creds
 key: password
- name: PROXY_HOST
 value: "proxy-host.example:8080"

This option is more convenient for password rotation: you change one Secret key instead of four ready-made strings.

Option 3: A file instead of a variable

Process environment variables are readable from /proc and often end up in crash dumps and error reports. If your security policy is strict, mount the Secret as a file with 0400 permissions and read it in the app or sidecar:

volumes:
- name: proxy-creds
 secret:
secretName: egress-proxy-creds
defaultMode: 0400
containers:
- name: app
 volumeMounts:
 - name: proxy-creds
mountPath: /var/run/secrets/proxy
readOnly: true

Option 4: Remove credentials entirely

If you have an egress gateway with a stable egress IP, you can switch proxy authorization from login/password to an IP allowlist. In that case, nothing secret is stored in the cluster at all: pods go to the gateway's internal Service, the gateway exits from a fixed address, and the provider passes traffic by address. Proxeon supports both authorization modes, and for a cluster with a dedicated egress, we usually recommend IP mode as more secure and simpler to operate.

Secret hygiene

• Enable Secret encryption at rest in etcd — it's off by default in most self-managed installs.
• Restrict RBAC: only service accounts that genuinely need to read namespace secrets should have that permission. An app that gets a secret via envFrom shouldn't be able to read the Secret object through the API.
• Don't log the environment. Check that the framework doesn't dump env on startup in debug mode.
• Agree on rotation. Changing the proxy password should be a routine operation: update the Secret, do a rollout restart, disable the old password.
• Remember that when using envFrom, updating a Secret doesn't change the environment of already running pods. You need a rollout restart. When mounted as a file, kubelet updates the contents within a minute or two, but the app has to be able to re-read the file.

The sidecar approach: when it's justified and what it gives you beyond proxying

A sidecar in the egress context is a local proxy container in the pod that accepts the app's traffic and forwards it through an external proxy. It turns the question "does the library respect HTTP_PROXY" into "can the sidecar intercept traffic," and the second question is easier to answer.

When a sidecar is justified

• The app is written in a stack where proxy management is inconvenient (a JVM with a dozen different clients, Node.js with mixed fetch and axios, legacy C++).
• You need address rotation on schedule or on errors without touching code: the sidecar holds a pool of several Proxeon proxies and switches on its own.
• You want to move credentials out of the app: the sidecar is the only one that knows the password, and the app calls localhost without auth.
• You need a unified policy of retries, timeouts, and rate limits for external traffic for the service.
• You need non-HTTP traffic: SOCKS mode or transparent TCP interception.

When a sidecar is overkill

• A single Go or Python service with one HTTP client. Environment variables will do.
• Tight resource limits: thousands of small pods, where an extra 30–50 MB per sidecar multiplies into tons of memory.
• You already have an egress gateway with the required logic, and duplicating it in every pod is pointless.

Native sidecar containers

Since Kubernetes 1.29, and stable by 1.33, there's a mechanism for native sidecars: an init container with restartPolicy: Always. Such a container starts before the main one, lives for the whole pod lifetime, and shuts down correctly after the main one. This solves an old pain: previously, the app could start before the proxy and get connection refused on its first requests, and a Job with a sidecar would never complete because the sidecar kept running.

A working snippet: a pod with a sidecar proxy

Here the sidecar is a small forward proxy (say, the egress-sidecar image) listening on 127.0.0.1:3128, reading upstream credentials from a Secret, and knowing which addresses to send directly:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: feed-importer
 namespace: integrations
spec:
 replicas: 2
 selector:
matchLabels:
 app: feed-importer
 template:
metadata:
 labels:
app: feed-importer
spec:
 initContainers:
 - name: egress-sidecar
image: registry.internal/egress-sidecar:1.3.0
restartPolicy: Always
env:
- name: LISTEN_ADDR
 value: "127.0.0.1:3128"
- name: DIRECT_CIDRS
 value: "10.96.0.0/12,10.244.0.0/16,192.168.10.0/24"
- name: DIRECT_SUFFIXES
 value: ".svc,.cluster.local,.corp.local"
envFrom:
- secretRef:
name: egress-upstream-creds
resources:
 requests:
cpu: 20m
memory: 32Mi
 limits:
memory: 96Mi
readinessProbe:
 tcpSocket:
port: 3128
 containers:
 - name: app
image: registry.internal/feed-importer:4.1.0
envFrom:
- configMapRef:
name: egress-proxy-common
env:
- name: HTTP_PROXY
 value: "http://127.0.0.1:3128"
- name: HTTPS_PROXY
 value: "http://127.0.0.1:3128"
- name: http_proxy
 value: "http://127.0.0.1:3128"
- name: https_proxy
 value: "http://127.0.0.1:3128"

Note that the app still needs NO_PROXY. The sidecar can send internal traffic directly, but why send it through an extra hop on localhost? On top of that, CIDR parsing in the sidecar is under your control, but in the app it isn't. Double protection is appropriate here.

Transparent mode via iptables

If the app ignores the environment entirely, add an init container that redirects all outbound TCP traffic in the pod, except internal networks and the sidecar's own traffic, to the sidecar's port. The sidecar must work in transparent proxy mode and restore the original destination. The init container needs NET_ADMIN capability, and the sidecar must run under a separate UID so its own traffic doesn't loop back:

initContainers:
- name: egress-redirect
 image: registry.internal/egress-init:1.0.0
 securityContext:
capabilities:
 add: ["NET_ADMIN"]
 command: ["sh", "-c"]
 args:
 - |
iptables -t nat -N EGRESS_REDIRECT
iptables -t nat -A EGRESS_REDIRECT -d 10.96.0.0/12 -j RETURN
iptables -t nat -A EGRESS_REDIRECT -d 10.244.0.0/16 -j RETURN
iptables -t nat -A EGRESS_REDIRECT -d 127.0.0.0/8 -j RETURN
iptables -t nat -A EGRESS_REDIRECT -m owner --uid-owner 1337 -j RETURN
iptables -t nat -A EGRESS_REDIRECT -p tcp -j REDIRECT --to-ports 15001
iptables -t nat -A OUTPUT -p tcp -j EGRESS_REDIRECT

Here 1337 is the UID the sidecar runs under, and 15001 is its transparent port. This is the same principle service meshes use to intercept traffic. If you already have a mesh in the cluster, its egress capabilities will likely be enough and you won't need a separate sidecar.

What a sidecar gives you beyond proxying

• Egress metrics by destination host, response code, and time: the app knows nothing about them, but the sidecar sees everything.
• Circuit breaker for external APIs: after a series of errors, the sidecar stops hammering the partner and returns quick failures.
• Address pool and rotation: a list of Proxeon proxies in the config, switching on round-robin or on a 407/403 error.
• Local DNS cache for external names, reducing load on CoreDNS.
• Rate limiting per-destination to respect partner API limits and avoid their throttling.

Cluster egress gateway: a single exit point

An egress gateway solves a problem of a different scale: not "how this service goes through a proxy," but "through what addresses the cluster goes out into the world." Let's look at a practical implementation without tying it to a specific CNI.

Architecture

1. A group of nodes with fixed external IPs is allocated (in a cloud, with a pinned public address or behind a dedicated NAT gateway). Nodes get the label node-role: egress and a taint so unrelated pods don't land on them.
2. A forward proxy Deployment with several replicas and anti-affinity is deployed on those nodes. A ClusterIP Service sits in front of it.
3. App pods get HTTP_PROXY pointing to that Service. Their NO_PROXY contains the internal networks.
4. NetworkPolicy forbids any app pod egress outside the cluster, except DNS and traffic to the gateway Service.
5. The gateway itself goes out either directly (then its IP is the cluster's egress address) or through an external Proxeon proxy with IP-based auth.

A working snippet: the gateway Deployment and NetworkPolicy

apiVersion: apps/v1
kind: Deployment
metadata:
 name: egress-gateway
 namespace: egress
spec:
 replicas: 3
 selector:
matchLabels:
 app: egress-gateway
 template:
metadata:
 labels:
app: egress-gateway
spec:
 nodeSelector:
node-role: egress
 tolerations:
 - key: dedicated
value: egress
effect: NoSchedule
 affinity:
podAntiAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 matchLabels:
app: egress-gateway
topologyKey: kubernetes.io/hostname
 containers:
 - name: proxy
image: registry.internal/egress-gateway:2.0.1
ports:
- containerPort: 3128
envFrom:
- secretRef:
name: egress-upstream-creds
---
apiVersion: v1
kind: Service
metadata:
 name: egress-gateway
 namespace: egress
spec:
 selector:
app: egress-gateway
 ports:
 - port: 3128
targetPort: 3128

The policy for the app namespace that closes direct internet egress:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: egress-only-via-gateway
 namespace: crawlers
spec:
 podSelector: {}
 policyTypes: ["Egress"]
 egress:
 - to:
- namespaceSelector:
matchLabels:
 kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
 port: 53
- protocol: TCP
 port: 53
 - to:
- namespaceSelector:
matchLabels:
 kubernetes.io/metadata.name: egress
 podSelector:
matchLabels:
 app: egress-gateway
ports:
- protocol: TCP
 port: 3128
 - to:
- ipBlock:
cidr: 10.244.0.0/16
- ipBlock:
cidr: 10.96.0.0/12

Such a policy turns the gateway from a recommendation into an obligation: a pod with a forgotten HTTP_PROXY won't get out bypassing it, it'll just time out. This gets discovered in a test environment in minutes instead of in production over weeks.

Fault tolerance and capacity

• At least three replicas on different nodes and a PodDisruptionBudget with minAvailable: 2.
• Connection monitoring: number of open connections, CONNECT setup time, share of 5xx from upstream.
• A cap on concurrent connections from a single client pod, so one failing service doesn't clog the gateway.
• Track conntrack on gateway nodes: with tens of thousands of connections per second, the table overflows and you get mysterious packet loss. Raise nf_conntrack_max in advance.

Diagnostics in the cluster: how to check egress IP from a pod and where to look for failures

Proxy problems in a cluster are insidious because symptoms appear far from the cause. Below is a methodology that localizes a failure in a few steps.

Step 1: Confirm the environment actually applied

kubectl -n crawlers exec deploy/price-collector -- env | grep -i proxy

Look for both case variants, NO_PROXY completeness, and no stray spaces. If the app starts via an entrypoint script, also check the main process's environment via /proc/1/environ — it may differ from what exec shows when launching a new shell.

Step 2: Check the egress IP

Deploy a simple echo service in the cluster or on a company external host that returns the client address and headers. Then from the pod:

kubectl -n crawlers exec deploy/price-collector -- curl -s https://echo.yourcompany.example/ip
kubectl -n crawlers exec deploy/price-collector -- curl -s --noproxy '*' https://echo.yourcompany.example/ip

The first command follows the environment's rules, the second forcibly bypasses the proxy. If the addresses match, the proxy isn't used. If the first returns the proxy provider's IP and the second returns the node IP or cloud NAT, the scheme works. This is also the only reliable way to confirm you're seeing exactly the address Proxeon's dashboard gave you, not the node's address.

Step 3: Look at the conversation with the proxy from the client's perspective

kubectl -n crawlers exec deploy/price-collector -- curl -sv https://api.partner.example/status 2>&1 | head -30

Look for the CONNECT line and the proxy's response code to it. Typical signals:

• 407 Proxy Authentication Required — credentials didn't arrive or are wrong. Check URL encoding of the password and that the proxy expects Basic auth.
• 403 from the proxy on CONNECT — the target host or port is blocked by proxy policy, or your IP isn't on the allowlist for IP-based auth.
• 502/503 on CONNECT — the proxy couldn't reach the target. If the target is internal, that's a diagnosis of a NO_PROXY hole.
• Connection refused to the proxy itself — wrong address, port, or NetworkPolicy blocking traffic to the gateway.
• Timeout with no response — usually a network policy or missing route. Check that DNS resolves at all.

Step 4: Check intra-cluster calls

kubectl -n crawlers exec deploy/price-collector -- curl -sv http://payments.default.svc.cluster.local:8080/healthz 2>&1 | grep -iE "connected|proxy|HTTP/"

If the output shows a connection to the proxy instead of a direct connection to ClusterIP, NO_PROXY didn't cover that address. Repeat the same for the short name payments and for the service IP — all three should be direct.

Step 5: An ephemeral container for images without tools

Minimal images have neither curl nor nslookup. Use kubectl debug, which attaches a debug container into the pod's network namespace:

kubectl -n crawlers debug -it pod/price-collector-7d9f4-x2k8q --image=registry.internal/netdebug:latest --target=app -- sh

Inside you have the same IPs, the same resolv.conf, and the same sidecar on localhost. Only the app's environment isn't inherited here: check variables via the main process's /proc/PID/environ.

Step 6: Logs and events

• Sidecar or gateway logs: upstream response codes, CONNECT times, which hosts were called. This is where you see an attempt to send an internal address outside.
• Pod events: kubectl describe pod will show ImagePullBackOff — a sign that the container runtime isn't configured for the proxy or the registry isn't in its NO_PROXY.
• CoreDNS logs in debug mode: a spike of NXDOMAIN for external names means the app is resolving external hosts itself instead of handing them to the proxy.
• conntrack metrics on egress nodes: nf_conntrack_count approaching the limit is a precursor to packet loss.

Step 7: Separate the proxy problem from the target problem

The same 403 can come from the proxy or from the target site. A Via header or provider-specific response headers help distinguish, as does comparing with a --noproxy request. If it's 403 through the proxy and 200 direct, the issue is the proxy config or the target restricting traffic from that pool's addresses. If it's 403 in both cases, the problem is in the request itself.

Typical mistakes

Let's gather the rakes people step on most often. We've covered many already; here's a summary for quick checks.

1. NO_PROXY copied from a server. The value localhost,127.0.0.1 in a cluster will definitely break service communication. You need .svc, .cluster.local, service and pod CIDRs.
2. Relying on CIDR in NO_PROXY with Python or old curl. The library silently ignores the entry and traffic to ClusterIP goes through the proxy. Use DNS names and suffixes.
3. Only uppercase variables. curl doesn't read HTTP_PROXY in uppercase. Set both forms.
4. Credentials in the manifest. The password ends up in git and in kubectl get -o yaml output. Only Secrets, and ideally IP-based auth through a gateway.
5. A password with special characters without URL encoding. The client parses the URL incorrectly and you get connection refused to a non-existent host.
6. Expecting kubelet and containerd to pick up pod env. Images aren't pulling, everyone goes to inspect the Deployment, but you actually need a systemd drop-in on the nodes.
7. A forgotten registry in the runtime's NO_PROXY. The internal registry becomes unreachable because containerd went for it through the external proxy.
8. Java with HTTP_PROXY. The JVM doesn't read these variables. You need system properties via JAVA_TOOL_OPTIONS.
9. A sidecar as a regular container, not a native init with restartPolicy: Always. The app starts before the proxy and fails on its first requests, and the Job never completes.
10. An egress gateway without NetworkPolicy. The gateway exists, but half the pods bypass it, and you learn about it when a partner sends a list of unknown IPs.
11. Updated the Secret without a rollout restart. Old pods keep running with the old password you've already disabled in the dashboard.
12. Ignoring cloud metadata. 169.254.169.254 isn't in NO_PROXY — the cloud SDK can't fetch temporary keys and the app fails with mysterious auth errors.
13. Overloading one proxy address with all replicas. Thirty replicas hammer one egress address and the partner API responds with rate limits. Distribute load across a pool of addresses via a sidecar or gateway.

Tools and resources

The set worth keeping at hand when working with egress traffic in a cluster.

Built-in Kubernetes tools

• kubectl exec and kubectl debug — checking environment and network connections inside a pod, including ephemeral containers for minimal images.
• kubectl describe pod and events — ImagePullBackOff, probe failures, secret mount errors.
• NetworkPolicy — enforcing egress through the gateway and testing "who bypasses it" hypotheses.
• Native sidecar containers (restartPolicy: Always in initContainers) — correct lifecycle for a proxy helper.

Network diagnostics

• A debug image with curl, dig, tcpdump, ss, iptables — build your own based on a minimal distro and keep it in the internal registry.
• Your own echo service returning the client IP and headers — the most honest way to see the egress address.
• tcpdump on the node with a filter by pod IP — when you need to see exactly which address a connection is being opened to.
• CoreDNS logs in debug mode — for understanding who resolves what.

Managing secrets and configuration

• An external secrets manager synced into cluster Secrets — so proxy credentials don't live in git and can be rotated centrally.
• Kustomize or Helm with a shared ConfigMap for NO_PROXY — a single source of truth for all Deployments in the namespace.
• Admission policies checking that NO_PROXY is present in every pod with HTTP_PROXY — automatic protection against "copied from a server" copy-paste.

Proxy provider

On the proxy side, three things matter: support for IP-based authorization (for the gateway scheme), stable addresses with a clear retention period, and an API for managing access so password rotation can be automated from CI. Proxeon has these in the dashboard and through the API, which lets you embed the provider into your GitOps process instead of manually copying connection strings.

Case studies and results

Case 1: An ETL platform and "broken" internal services

A data team added HTTP_PROXY to twenty Deployments so marketplace catalog scrapers would go through dedicated addresses. They copied NO_PROXY from the server docs: localhost and 127.0.0.1. Within an hour, the 5xx rate between internal services jumped from 0.1% to 34%, and readiness probes on three services started flapping. After adding .svc, .cluster.local, and both CIDRs to a shared ConfigMap and doing a rollout restart, the errors cleared within five minutes. A side benefit: moving NO_PROXY into a ConfigMap cut the time for the next list change from half a day (editing twenty manifests) to one commit.

Case 2: JVM integrations and a sidecar

A financial integrator on Java with seven different HTTP clients inside one app (some through libraries that don't read proxy system properties). An attempt to configure everything via JAVA_TOOL_OPTIONS covered five clients out of seven. The solution was a native sidecar with transparent interception via iptables. All seven clients went through the proxy with no code changes, credentials disappeared from the app, and egress metrics appeared for the first time in the service's history. The cost: 40 MB of memory per pod, which across twelve replicas turned out to be negligible.

Case 3: An egress gateway and IP-based auth

A cluster of sixty nodes with autoscaling was calling several partner APIs that required IP allowlisting. Every node scaling event changed the set of egress addresses and partners blocked traffic. They deployed an egress gateway on three dedicated nodes with pinned addresses, introduced NetworkPolicy across all app namespaces, and switched Proxeon proxy auth to IP mode. Result: the "the partner blocked us again" incidents dropped from three or four a month to zero in a quarter, and all Secrets with proxy passwords disappeared from the cluster. A side effect: centralized gateway logs revealed two services calling external APIs that nobody knew about.

Case 4: containerd and the mysterious ImagePullBackOff

After moving the cluster into a network with a mandatory proxy, pods with HTTP_PROXY set worked fine, but new nodes couldn't pull a single image. The cause was a missing containerd drop-in. After adding the file with the proxy and the internal registry in NO_PROXY, the problem was solved. The lesson became a check in the node bootstrap script: before joining the cluster, a node verifies that containerd can pull a test image from both the internal and the external registry.

Rollout checklist

Go through this list before rolling out a proxy to a production service. Every item is someone's nighttime incident.

Preparation

1. You've mapped the service's outbound connections: external hosts, internal services, databases, queues, cloud endpoints, protocols.
2. You know the cluster's service CIDR, pod CIDR, node CIDR, and company internal domains.
3. You've checked which HTTP clients the app uses and how each interprets NO_PROXY (CIDR, suffixes, case).
4. You've chosen the level: container, sidecar, or gateway. The decision is documented.

Configuration

1. NO_PROXY is in the namespace ConfigMap and contains: localhost, 127.0.0.1, .svc, .svc.cluster.local, .cluster.local, kubernetes.default.svc, service CIDR, pod CIDR, 169.254.169.254, internal domains.
2. Variables are set in both cases.
3. Credentials are only in a Secret; the password is URL-encoded; RBAC on secrets is restricted.
4. For JVM, JAVA_TOOL_OPTIONS with nonProxyHosts is set.
5. If a sidecar — it's native (initContainers with restartPolicy: Always), with a readinessProbe and resource limits.
6. If a gateway — at least three replicas, anti-affinity, PDB, NetworkPolicy in app namespaces.
7. The container runtime on nodes is configured separately; the internal registry is in its NO_PROXY.

Pre-prod validation

1. env inside the pod shows the expected values.
2. A request to the echo service returns the proxy address; with --noproxy — the node address.
3. Requests to internal services by short name, full name, and ClusterIP go directly (visible in curl -v).
4. Readiness and liveness pass consistently for at least ten minutes.
5. The Kubernetes API client (if any) can see the control plane.
6. The sidecar or gateway logs contain no calls to internal addresses.

Rollout and after

1. A phased rollout with maxUnavailable no higher than 25%; monitoring of 5xx between services and 407/403 errors from the proxy.
2. A documented password rotation procedure: update the Secret, rollout restart, disable the old access.
3. Alerts are set for growth in proxy error rates and for conntrack approaching the limit on egress nodes.
4. A week later — a review of gateway logs: no unexpected external destinations appeared.

FAQ

Can I set HTTP_PROXY globally for the whole cluster instead of in each Deployment?

There's no direct mechanism: environment variables are part of the pod spec. Practical options are a shared ConfigMap with envFrom in all manifests via Helm or Kustomize, or a mutating admission controller that automatically injects variables into pods with a certain annotation. The second method is more powerful but requires care: it must also add NO_PROXY, otherwise the automation will break services en masse.

Why does curl in the pod go through the proxy, but a Python app doesn't, even though the environment is the same?

Most likely it's case or NO_PROXY parsing. curl reads http_proxy only in lowercase, Python in both. On the other hand, Python doesn't understand CIDR in NO_PROXY, while a recent curl does. Check both hypotheses: set variables in both cases and use domain suffixes instead of CIDR.

Is NO_PROXY even needed if all internal traffic goes through a sidecar with direct rules?

Technically you could skip it, but we recommend keeping it. First, an extra hop through localhost for thousands of internal requests means non-zero latency and load on the sidecar. Second, if the sidecar crashes or restarts, an app with a correct NO_PROXY keeps talking to internal services, but without it, it loses everything.

How do I confirm a partner request went out from the proxy address, not the node address?

The only reliable way is a control request to an echo service that returns the client IP, from the same pod and with the same environment. Compare the response to the address shown in the Proxeon dashboard for your access and to the response with --noproxy. The partner's own logs help too, but you usually don't have them handy.

Should I encrypt the connection to the proxy (https:// in HTTPS_PROXY)?

For HTTPS traffic to target sites, the content is already end-to-end encrypted inside the CONNECT tunnel, and the proxy only sees the destination host. TLS to the proxy protects the auth credentials and host names on the pod-to-proxy leg. If that leg goes over an untrusted network, TLS to the proxy is justified, but make sure both the client and the proxy support it: some libraries ignore the https:// scheme in HTTPS_PROXY. Inside the cluster, between pod and egress gateway, network policies are usually enough.

What about database and broker traffic that needs to exit the cluster?

It won't go through an HTTP proxy — it's not HTTP. Options: SOCKS5 proxy mode, if your provider and client library support it; transparent interception at the sidecar level with TCP tunneling; or an egress gateway that works at the routing level rather than the protocol level. The choice depends on whether you need a specific egress IP or just a centralized exit.

Can I use the same proxy for containerd and for applications?

Technically yes, but they're usually different tasks. The runtime pulls images from public registries and needs a corporate perimeter proxy. Applications call partner APIs and need dedicated addresses with specific properties. Mixing them means image pulls consume traffic allowances meant for application tasks. Keep them separate.

How do I rotate the proxy password safely without downtime?

Create a second access with the provider, update the Secret with the new credentials, do a rollout restart with phased replica updates, confirm from the logs that there are no 407s, and only then disable the old access. If you use IP-based auth through a gateway, password rotation disappears from your to-do list entirely.

How do I restrict the proxy to only specific external domains?

NO_PROXY works the opposite way — it describes where not to go through the proxy, and you can't build an allowlist from it. That logic goes in the sidecar or gateway: rules like "domains from list A — through the Proxeon address pool, everything else — direct or blocked." Plus NetworkPolicy so direct internet egress physically doesn't exist.

Conclusion

Egress traffic in Kubernetes differs from server-based traffic not in details, but in structure: every pod has two worlds — internal and external — and the proxy should only see the second. Everything else follows from that. NO_PROXY goes from a formality to an architectural element that must describe the entire internal cluster network and account for the fact that different libraries parse it differently. Environment variables only work inside the process — kubelet, the container runtime, and stubborn SDKs need separate attention. Credentials shouldn't end up in manifests, and ideally should disappear from the cluster entirely thanks to a gateway with IP-based auth.

The three levels of proxy integration — container, sidecar, egress gateway — aren't competitors, they're tools for different jobs. The container level is fast and granular. A sidecar removes dependence on library behavior and adds metrics, rotation, and retries. A gateway gives the cluster a predictable face in the outside world and turns proxied egress from a recommendation into a rule backed by network policies.

What to do next? Start with inventory: map your services' outbound connections and see how many are internal. Then move NO_PROXY into a shared ConfigMap and validate it in a test environment using the diagnostics section's methodology. If the cluster has more than one stack or requirements for stable addresses — design an egress gateway and connect dedicated Proxeon addresses to it with IP-based auth. And keep the rollout checklist nearby: it pays for itself on the very first incident that didn't happen.