The Rooms the Fix Can't Reach

You did everything the last post asked. The Zscaler root is in your OS trust store, NODE_EXTRA_CA_CERTS is set in your shell profile, az finally logs in, git clone stopped complaining — every tool on the machine has been taught, one dialect at a time, to trust the corporate root honestly. You’ve earned the competent feeling. Then you type docker build, and it dies at the first RUN npm install with the exact error from Part 2 — SELF_SIGNED_CERT_IN_CHAIN — on the exact machine where npm install at the shell now works fine. Nothing regressed. Your shell profile is untouched; run the command outside the build and it still succeeds. The fix simply didn’t cross a line you didn’t know was there, and once you see where that line falls, half the mysteries left in this series resolve at once. Because what you fixed in Part 2 wasn’t trust — it was one machine’s trust. And a container is not that machine. It’s a brand-new one, built seconds ago from a base image that has never in its life heard of your employer, carrying its own trust store with nothing in it but the public roots someone compiled in at build time. You didn’t teach it anything; you were never talking to it. This post is a tour of the rooms on the wrong side of that line — containers, WSL distros, CI runners — each a fresh machine that never got the memo, and each failing for the one reason you’re now equipped to recognise on sight.

So you teach it — except “it” is now a machine that gets rebuilt from scratch on every docker build, so the teaching has to live in the Dockerfile itself, baked into the image’s construction rather than typed into a shell that the container never sees. Here’s the reusable spine, for a Debian or Ubuntu base:

# Teach the machine who to trust, before anything tries to reach the wire
COPY zscaler-root.crt /usr/local/share/ca-certificates/zscaler-root.crt
RUN update-ca-certificates                                          # OS store: curl, apt, .NET, git
ENV NODE_EXTRA_CA_CERTS=/usr/local/share/ca-certificates/zscaler-root.crt   # Node
RUN npm config set cafile "$NODE_EXTRA_CA_CERTS"                    # npm, which sometimes still wants telling

Look at what that is: the same certificate, installed three ways into one small image. Not because I’m being thorough — because a fresh container trusts nothing, and every tool inside it reads a different store, exactly as Part 2 warned. update-ca-certificates folds the root into the OS bundle, which satisfies curl, apt, and .NET in one stroke. NODE_EXTRA_CA_CERTS teaches Node, which never consults that bundle. npm config set cafile teaches npm a second time, because npm occasionally insists on hearing it for its own registry traffic. You hand the same memo to three different clerks because none of them talk to each other — the entire thesis of this series, reproduced in four lines inside a machine that didn’t exist a minute ago.

[!NOTE] That spine assumes a Debian or Ubuntu base. Alpine — the default under a lot of Node and Go images — uses musl, keeps its trust store elsewhere, and often ships without the tooling at all, so the same Dockerfile fails in a fresh way: RUN apk add --no-cache ca-certificates first, then COPY your root into /usr/local/share/ca-certificates/ and run update-ca-certificates. Same three moves, different clerk’s dialect — the tax you pay for the smaller image is remembering it isn’t Debian.

There’s an order hiding in those four lines, and getting it wrong is its own genre of failure. Every line in a Dockerfile that reaches the internet — apt-get update, RUN npm install, a curl that bootstraps a runtime — is a cliff: a TLS handshake that dies the instant it runs if the cert isn’t already in place. My own image bootstraps .NET with curl -fsSL https://dot.net/v1/dotnet-install.sh, which means even fetching the installer script is a handshake that fails before the script has run a single line. So the cert install can’t sit wherever it reads tidily; it has to come first, above the first instruction that touches the wire. The Dockerfile stops being a script you skim top to bottom and becomes a dependency graph of trust: nothing that reaches the network may appear above the line that teaches the machine whose signatures to believe.

And the line recurses. A multi-stage build is several fresh machines stacked in one file: each FROM opens a new stage with its own filesystem, so the root you carefully installed in the builder stage doesn’t exist in the final one unless you copy it across. This bites hardest with distroless or scratch final images — the ones with no shell, no package manager, and no update-ca-certificates left to run. There the move is to lift the finished bundle out of the builder, COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/, and make sure the app actually reads it — a Go binary honours SSL_CERT_FILE, .NET just needs the bundle present on disk. It’s the host-to-container jump again, one layer smaller: every boundary that mints a new machine mints a new empty trust store, and the cert has to be walked across each one by hand.

Containers at least look like what they are — a fresh machine you built on purpose. WSL wears the host’s face. A WSL2 distro is a whole separate Linux machine running inside your Windows one — its own kernel, its own filesystem, and, the part that catches everyone, its own trust store that never saw the Zscaler root your Windows side trusts. Fixing Windows does precisely nothing for it, exactly like the container, only stranger: the container was visibly elsewhere, while WSL sits in the same taskbar, opens your files, and feels like one more shell on the machine you already fixed. So you run the container move under a new name — export the root from the Windows Cert:\LocalMachine\Root store, drop the .crt into the distro’s /usr/local/share/ca-certificates/, and sudo update-ca-certificates.

I’ll be straight about this room, though, because the rest of this series has been first-hand and this part isn’t. WSL is the one place here I can’t hand you a clean walkthrough I earned. The last time Zscaler and WSL collided for me, what’s left in memory is friction, not triumph — I genuinely don’t remember whether I fixed it properly or just worked around it until it stopped mattering. (I did later move to a Mac, but for the Unix environment, not because the proxy ran me off Windows — it didn’t.) So take what follows as documented ground walked carefully, not scars I can show you.

Two more WSL gotchas worth naming so you don’t chase them as ghosts, both about reaching the network rather than trusting it. First, DNS: WSL regenerates its own /etc/resolv.conf on each boot, and under corporate split-tunnel DNS that generated resolver is often wrong, so name resolution fails in ways that masquerade as connectivity or even certificate problems. Pin it — set generateResolvConf = false in /etc/wsl.conf and supply a static resolver you trust. Second, networking mode: the default NAT gives the distro its own network stack, one more layer between it and whatever Zscaler is doing; on Windows 11, networkingMode = mirrored instead shares the Windows network and loopback, so the distro inherits the host’s proxy and VPN behaviour more gracefully and a good deal of behind-the-proxy friction simply never arises. If your build supports it, mirrored is the calmer default behind Zscaler.

[!WARNING] The gotcha that names a certificate and means a clock. Close the laptop, come back later, and your next TLS handshake inside WSL fails with certificate is not yet valid. Look at the cert, by all means — but this time the look clears it of all charges: the certificate is fine; its validity window simply starts after the moment your drifted clock believes it is. When Windows sleeps and wakes, the WSL VM’s clock can fall behind, and a perfectly good cert reads as not-yet-valid against a wrong “now.” sudo hwclock -s resyncs the clock from the hardware and the error evaporates. It’s the one failure in this series that mentions a certificate and has nothing to do with trust — your clock time-travelled.

The last room is the one you’re least likely to be standing in when it bites, which is what makes it so disorienting. CI runners are fresh machines too, but they come in two kinds that fail in opposite directions. A cloud-hosted runner egresses straight to the internet, outside Zscaler entirely — it never meets an inspected certificate, so your build passes there without a single cert line in sight. A self-hosted runner inside the corporate network sits behind Zscaler like everything else, and needs the root baked into its image exactly the way the container did. Put those two together and you get the signature ghost of proxied CI: “works in CI, fails locally,” or its mirror, “works locally, fails in CI” — one pipeline, passing on one machine and failing on another, for no reason anywhere in the code, because the difference isn’t in the code. It’s which side of the proxy the runner happens to sit on. Say that asymmetry out loud and you stop hunting a bug in your build script that was never there: the script was fine, the machine’s vantage point wasn’t.

Step back from all three rooms and notice how little you actually had to learn to walk through them. Not a Docker fix and a WSL fix and a CI fix, filed separately — one idea, applied three times. Every machine that touches the wire carries its own trust store, and every boundary that mints a new machine — a docker build, a WSL distro, a runner spun up in a pipeline — mints a new store that starts empty and has never heard of your employer. The host fix doesn’t propagate because there was never a single “the host” to fix; there was only ever a succession of machines, each needing the same memo handed to it directly. Once you see that, the specific commands stop being things to memorise. update-ca-certificates, NODE_EXTRA_CA_CERTS, the Windows-store export, the cert baked into a runner image — they’re one sentence in four accents.

There’s a comment in my Dockerfile that reads: “The Zscaler root CA is installed above so npm can validate inspected TLS.” I wrote it for whoever opens that file next — reliably a future version of me who’s forgotten — and it turns out to be the whole point of these three posts. This series is that one comment, expanded: the memo nobody ever handed me, finally written down somewhere the next machine’s owner will actually read it. And if you keep just one thing from all of it, keep the reflex, not the commands — the commands age out with the tools and the reflex doesn’t. There was never an aha here, no moment the chain lit up and I gasped; just the same failure, in a new tool and on a new machine each time, turning up often enough that suspicion wore a groove. So when something that should just work doesn’t, and the error is strange or missing altogether — look at the cert first. Every fix in these three posts is downstream of that one look.

This series has no talk or book beneath it — it’s the memo I never got, reconstructed the slow way across a decade of the same certificate breaking the same handshake in a new tool, on a new machine, each time. The Docker scars are first-hand: the belt-and-suspenders Dockerfile is lifted from my own, stripped of its internal freight down to the four lines that matter. WSL and CI I’ve flagged in the text as documented ground rather than lived — I won’t sell you a triumph I don’t have. Every command here was run against public hosts and public base images; nothing is specific to one employer’s tenant.

⊏≣⊐ Ōla — Kerala’s palm-leaf manuscripts, text incised into dried talipot leaf with an iron stylus and rubbed with soot to show; the leaf perishes within a lifetime, so each had to be recopied by hand, generation to generation, or the words were simply lost.