<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[The Esc Newsletter]]></title><description><![CDATA[Software Engineering, SRE, AI & Philosophy]]></description><link>https://newsletter.esc.sh</link><image><url>https://newsletter.esc.sh/img/substack.png</url><title>The Esc Newsletter</title><link>https://newsletter.esc.sh</link></image><generator>Substack</generator><lastBuildDate>Mon, 16 Nov 2026 00:06:02 GMT</lastBuildDate><atom:link href="https://newsletter.esc.sh/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[Mansoor]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[writeahead@substack.com]]></webMaster><itunes:owner><itunes:email><![CDATA[writeahead@substack.com]]></itunes:email><itunes:name><![CDATA[Mansoor]]></itunes:name></itunes:owner><itunes:author><![CDATA[Mansoor]]></itunes:author><googleplay:owner><![CDATA[writeahead@substack.com]]></googleplay:owner><googleplay:email><![CDATA[writeahead@substack.com]]></googleplay:email><googleplay:author><![CDATA[Mansoor]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[Why I switched from Claude Code to a Custom Coding Agent (Pi)]]></title><description><![CDATA[Claude Code is built for everyone. That's the problem. I replaced it with a custom coding agent that follows my rules, challenges my assumptions, and doesn't bury my instructions under 20k tokens of product logic.]]></description><link>https://newsletter.esc.sh/p/claude-code-to-pi</link><guid isPermaLink="false">https://newsletter.esc.sh/p/claude-code-to-pi</guid><dc:creator><![CDATA[Mansoor]]></dc:creator><pubDate>Sat, 25 Apr 2026 23:36:49 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/50ad6e74-6f9e-453d-a7d1-ee2f8247907c_2000x993.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!EuCW!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4c4294d-cf32-4791-83d5-e4fc979d6410_2000x993.png 424w, https://substackcdn.com/image/fetch/$s_!EuCW!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4c4294d-cf32-4791-83d5-e4fc979d6410_2000x993.png 848w, https://substackcdn.com/image/fetch/$s_!EuCW!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4c4294d-cf32-4791-83d5-e4fc979d6410_2000x993.png 1272w, https://substackcdn.com/image/fetch/$s_!EuCW!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4c4294d-cf32-4791-83d5-e4fc979d6410_2000x993.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!EuCW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4c4294d-cf32-4791-83d5-e4fc979d6410_2000x993.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f4c4294d-cf32-4791-83d5-e4fc979d6410_2000x993.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!EuCW!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4c4294d-cf32-4791-83d5-e4fc979d6410_2000x993.png 424w, https://substackcdn.com/image/fetch/$s_!EuCW!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4c4294d-cf32-4791-83d5-e4fc979d6410_2000x993.png 848w, https://substackcdn.com/image/fetch/$s_!EuCW!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4c4294d-cf32-4791-83d5-e4fc979d6410_2000x993.png 1272w, https://substackcdn.com/image/fetch/$s_!EuCW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4c4294d-cf32-4791-83d5-e4fc979d6410_2000x993.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><p>I had been a Claude Code user ever since it was released. I used it daily for my work as well as for personal hobby projects. It served me much better than the competition such as Cursor and Codex. But as I started relying on it more heavily, I noticed something felt off. I was fighting Claude Code more and more &#8212; it wasn't following my instructions properly anymore.</p><p>I'd be brainstorming something with half-baked ideas and instead of pushing back or questioning my framing, it would be so eager to jump in and code with my weak assumptions. The "You're absolutely right!" response is famous for a reason &#8212; it's great that they got rid of that phrase, but the behavior isn't that different. I didn't want a "yes-man."</p><p>I had my custom <code>CLAUDE.md</code> files with all the rules that I wanted it to follow, but they were mere suggestions rather than rules to Claude. I knew the issue was not with the model itself but the harness &#8212; Claude Code.</p><p>I recently came across a <a href="https://lucumr.pocoo.org/2026/1/31/pi/">blog post by Armin Ronacher</a> (creator of Flask) where he talked about "Pi" the coding agent. So I decided to give it a try and it completely changed my experience (with the same underlying model &#8211; Opus 4.6)</p><h2>The Problem with "One Size Fits All" Agents</h2><p>When an agent is designed to be used by both completely new coders and experienced engineers, it has to make some compromises, the biggest of them is the bloated system prompt that tries to do everything</p><h3>Bloated System Prompt</h3><p>Every AI coding tool sends a set of hidden instructions to the model before your message ever reaches it &#8212; this is called the system prompt. It tells the model how to behave, what tools it has access to, and what rules to follow.</p><p>The problem is, <strong>these system prompts are massive</strong>. They're packed with instructions for every feature the product offers &#8212; web search, file handling, image generation, citation formatting, safety guardrails, copyright rules, tool definitions for dozens of integrations &#8212; regardless of whether you'll ever use any of it. All of this <strong>competes for the model's limited attention</strong>. Your carefully written <code>CLAUDE.md</code> instructions aren't competing with a few sensible defaults &#8212; they're buried under thousands of tokens of product logic that has nothing to do with your task.</p><blockquote><p>&#128161; Ever wonder why Claude ignores your carefully written <code>CLAUDE.md</code> rules? The model's attention is finite. Your instructions aren't competing with a few sensible defaults &#8212; they're competing with thousands of tokens of product logic you didn't write and don't need. And they're losing.</p></blockquote><p>Check the screenshot below. Anthropic recently cut down on the system prompt, even with that there is ~20k tokens that are already in front of your first message to the model</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!sDIw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf4d077c-f2cd-4b7e-b56c-d49e3375a5a0_1516x616.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!sDIw!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf4d077c-f2cd-4b7e-b56c-d49e3375a5a0_1516x616.png 424w, https://substackcdn.com/image/fetch/$s_!sDIw!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf4d077c-f2cd-4b7e-b56c-d49e3375a5a0_1516x616.png 848w, https://substackcdn.com/image/fetch/$s_!sDIw!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf4d077c-f2cd-4b7e-b56c-d49e3375a5a0_1516x616.png 1272w, https://substackcdn.com/image/fetch/$s_!sDIw!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf4d077c-f2cd-4b7e-b56c-d49e3375a5a0_1516x616.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!sDIw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf4d077c-f2cd-4b7e-b56c-d49e3375a5a0_1516x616.png" width="1516" height="616" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/af4d077c-f2cd-4b7e-b56c-d49e3375a5a0_1516x616.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:616,&quot;width&quot;:1516,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!sDIw!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf4d077c-f2cd-4b7e-b56c-d49e3375a5a0_1516x616.png 424w, https://substackcdn.com/image/fetch/$s_!sDIw!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf4d077c-f2cd-4b7e-b56c-d49e3375a5a0_1516x616.png 848w, https://substackcdn.com/image/fetch/$s_!sDIw!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf4d077c-f2cd-4b7e-b56c-d49e3375a5a0_1516x616.png 1272w, https://substackcdn.com/image/fetch/$s_!sDIw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf4d077c-f2cd-4b7e-b56c-d49e3375a5a0_1516x616.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Claude Code Context</figcaption></figure></div><p>Take a look at this <a href="https://github.com/Piebald-AI/claude-code-system-prompts">repo</a> to get an idea of how big it can be depending on what you are doing.</p><p>In contrast, check Pi's context</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!PL17!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85e494fd-996c-47ec-8bfd-f78531922dbd_1258x736.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!PL17!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85e494fd-996c-47ec-8bfd-f78531922dbd_1258x736.png 424w, https://substackcdn.com/image/fetch/$s_!PL17!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85e494fd-996c-47ec-8bfd-f78531922dbd_1258x736.png 848w, https://substackcdn.com/image/fetch/$s_!PL17!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85e494fd-996c-47ec-8bfd-f78531922dbd_1258x736.png 1272w, https://substackcdn.com/image/fetch/$s_!PL17!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85e494fd-996c-47ec-8bfd-f78531922dbd_1258x736.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!PL17!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85e494fd-996c-47ec-8bfd-f78531922dbd_1258x736.png" width="1258" height="736" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/85e494fd-996c-47ec-8bfd-f78531922dbd_1258x736.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:736,&quot;width&quot;:1258,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!PL17!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85e494fd-996c-47ec-8bfd-f78531922dbd_1258x736.png 424w, https://substackcdn.com/image/fetch/$s_!PL17!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85e494fd-996c-47ec-8bfd-f78531922dbd_1258x736.png 848w, https://substackcdn.com/image/fetch/$s_!PL17!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85e494fd-996c-47ec-8bfd-f78531922dbd_1258x736.png 1272w, https://substackcdn.com/image/fetch/$s_!PL17!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85e494fd-996c-47ec-8bfd-f78531922dbd_1258x736.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">pi context usage</figcaption></figure></div><h2>Solution: Switching to a custom coding agent (Pi)</h2><p>To fix this, I moved to a custom coding agent setup built on <a href="https://pi.dev">Pi</a>. (Note: Some of the extensions in my setup are shamelessly copied from Armin Ronacher's excellent <a href="https://github.com/mitsuhiko/agent-stuff">agent-stuff repo</a>).</p><p>By stripping away the massive default prompt of Claude Code, I could define exactly how the agent should behave.</p><h3>Why Pi?</h3><p>Pi is an open-source coding agent built on a simple premise: ship the minimum viable agent and let the user build the rest. It comes with four tools &#8212; read, write, edit, and bash &#8212; a system prompt under 1,000 tokens, and no MCP, no built-in permissions, no product logic. That's it.</p><p>Everything else is added through extensions &#8212; TypeScript files you drop into a folder that Pi picks up on startup. Extensions can register new tools, intercept tool calls, gate permissions, modify the UI, or do anything else you can think of. If you want planning workflows, you write an extension. If you want sandboxing, you write an extension.</p><p>This is what makes it fundamentally different from Claude Code or Cursor. It's not a product with opinions about how you should work &#8212; it's a framework for building the agent you actually want.</p><p>Here is a great read by the author himself - <a href="https://mariozechner.at/posts/2025-11-30-pi-coding-agent/">What I learned building an opinionated and minimal coding agent</a></p><h3><strong>How I use Pi</strong></h3><h3>1 - Minimal, Opinionated System Prompt</h3><p>Pi by default comes with a tiny system prompt at around 500 tokens. I modified it to add a few more things that fit my workflow. You can see my exact system prompt <a href="https://github.com/MansoorMajeed/Clawd/blob/main/extensions/workflow-guard.ts#L3">here</a></p><p>My first few lines are indicative of the biggest issue I had with Claude Code.</p><pre><code>You are a thinking partner. Your approach is supervised autonomy &#8212; you discuss, question, and challenge assumptions by default. You do NOT jump to writing code, editing files, or executing commands unless explicitly asked. The user stays in the loop; you stay close enough to course-correct.


## Default mode: Discussion

Your natural state is conversation. When the user brings a problem:
1. Discuss it. Ask questions. Share your understanding. Challenge assumptions. Think out loud.
2. Don't touch anything. No file edits, no code, no commands &#8212; just talk.</code></pre><h3>2 - Scratch Workspace and <code>n2c</code></h3><p>For non-trivial tasks, I enforce planning. I maintain a <code>.scratch</code> folder in each repo that is gitignored. This acts as a dedicated workspace for plans, research, session logs, reviews etc. If anything relevant that should be checked into the repo, I will move it to <code>docs</code> folder.</p><p>Here is the part of the system prompt</p><pre><code>## Scratch area

`.scratch/` is a gitignored directory for all ephemeral agent work, organized by type:
- `research/` &#8212; distilled research (\`YYYY-MM-DD-&lt;slug&gt;.md\`)
- `plans/` &#8212; change plans with \`n2c:\` annotation loop (\`YYYY-MM-DD-&lt;slug&gt;.md\`)
- `reviews/` &#8212; code review findings (\`YYYY-MM-DD-&lt;branch&gt;.md\`)
- `sessions/` &#8212; session state for \`/continue\` handoffs</code></pre><p><strong>My flow looks like this</strong></p><ol><li><p>We start by discussing a problem that needs to be solved. The initial discussion may need fetching additional resources, documentation etc (which all go into the <code>.scratch</code> folder)</p></li><li><p>Once we both have a clear idea of what needs to be done, the agent will write a plan in the scratch folder.</p></li><li><p>I will read the plan (yes, actually read it!) and annotate with <code>n2c</code> (which stands for note to claude &#8211; though maybe I should change it to <code>n2p</code> ;) )</p><ol><li><p>I may annotate like <code>n2c: This assumption is wrong</code> or <code>n2c: No, this will not work due to X</code> or <code>n2c: This is a great idea, can we talk more on this?</code></p></li></ol></li><li><p>Then agent is instructed to re-read the plan, and <strong>then we will discuss the annotations one by one</strong></p></li><li><p>Once we both are agreed on a plan, it will write the final plan which can be executed</p></li></ol><h3>3 - Keeping a clean context</h3><p>While discussing something you may reach a point where you need clarity on something or discuss something in depth without it affecting the main context. I have two extensions that I use a lot (copied from Armin)</p><h4>Split-Fork and btw</h4><ul><li><p><strong>Split-fork:</strong> opens a new tmux/zellij pane to the right with the exact same context and you can do whatever want with it without affecting the main context. You can find the source <a href="https://github.com/mitsuhiko/agent-stuff/blob/main/extensions/split-fork.ts">here</a></p></li><li><p><strong>btw</strong>: Similar to split-fork, but optimized for quick, inline questions without derailing the agent's current train of thought. <a href="https://github.com/mitsuhiko/agent-stuff/blob/main/extensions/btw.ts">Source</a></p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!O20b!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff23024cc-b94a-4b8b-9cd8-becc0a557b85_2000x1308.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!O20b!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff23024cc-b94a-4b8b-9cd8-becc0a557b85_2000x1308.png 424w, https://substackcdn.com/image/fetch/$s_!O20b!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff23024cc-b94a-4b8b-9cd8-becc0a557b85_2000x1308.png 848w, https://substackcdn.com/image/fetch/$s_!O20b!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff23024cc-b94a-4b8b-9cd8-becc0a557b85_2000x1308.png 1272w, https://substackcdn.com/image/fetch/$s_!O20b!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff23024cc-b94a-4b8b-9cd8-becc0a557b85_2000x1308.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!O20b!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff23024cc-b94a-4b8b-9cd8-becc0a557b85_2000x1308.png" width="2000" height="1308" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f23024cc-b94a-4b8b-9cd8-becc0a557b85_2000x1308.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1308,&quot;width&quot;:2000,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!O20b!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff23024cc-b94a-4b8b-9cd8-becc0a557b85_2000x1308.png 424w, https://substackcdn.com/image/fetch/$s_!O20b!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff23024cc-b94a-4b8b-9cd8-becc0a557b85_2000x1308.png 848w, https://substackcdn.com/image/fetch/$s_!O20b!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff23024cc-b94a-4b8b-9cd8-becc0a557b85_2000x1308.png 1272w, https://substackcdn.com/image/fetch/$s_!O20b!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff23024cc-b94a-4b8b-9cd8-becc0a557b85_2000x1308.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">split-fork and btw</figcaption></figure></div><h3>4 - When the context starts to get full</h3><p>Even with models that support 1M token context windows, <strong>I never push past 200k</strong>. The reason is simple &#8212; model performance degrades noticeably as context fills up. It starts losing track of earlier decisions, forgets constraints you set at the beginning of the session, and gives increasingly generic responses. This is a good read on "context rot" : <a href="https://www.trychroma.com/research/context-rot">https://www.trychroma.com/research/context-rot</a></p><p>With Pi, I use the <code>continue</code> extension (<a href="https://github.com/MansoorMajeed/Clawd/blob/main/extensions/continue.ts">source</a>) to handle this cleanly. When the context reaches around 100&#8211;150k tokens and I want to keep going in the same direction, I type <code>/continue</code>. The agent summarizes what's relevant in the current context, writes it into a file in the <code>.scratch/sessions/</code> folder, clears the context, and starts a fresh chat with that file loaded in. Same trajectory, clean slate. No copy-pasting, no trying to manually recap where we were.</p><h2><strong>Organizations Should Lean Into Custom Coding Harnesses</strong></h2><p>This isn't just an individual developer problem &#8212; it's an organizational one as well.</p><p>When Anthropic released Opus 4.7, while it came with impressive benchmark claims, real-world performance turned out to be a major disappointment.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!-adb!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5901af2c-cd6b-47b5-8c90-eabc90fcf92f_1616x1074.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!-adb!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5901af2c-cd6b-47b5-8c90-eabc90fcf92f_1616x1074.png 424w, https://substackcdn.com/image/fetch/$s_!-adb!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5901af2c-cd6b-47b5-8c90-eabc90fcf92f_1616x1074.png 848w, https://substackcdn.com/image/fetch/$s_!-adb!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5901af2c-cd6b-47b5-8c90-eabc90fcf92f_1616x1074.png 1272w, https://substackcdn.com/image/fetch/$s_!-adb!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5901af2c-cd6b-47b5-8c90-eabc90fcf92f_1616x1074.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!-adb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5901af2c-cd6b-47b5-8c90-eabc90fcf92f_1616x1074.png" width="1616" height="1074" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5901af2c-cd6b-47b5-8c90-eabc90fcf92f_1616x1074.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1074,&quot;width&quot;:1616,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!-adb!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5901af2c-cd6b-47b5-8c90-eabc90fcf92f_1616x1074.png 424w, https://substackcdn.com/image/fetch/$s_!-adb!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5901af2c-cd6b-47b5-8c90-eabc90fcf92f_1616x1074.png 848w, https://substackcdn.com/image/fetch/$s_!-adb!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5901af2c-cd6b-47b5-8c90-eabc90fcf92f_1616x1074.png 1272w, https://substackcdn.com/image/fetch/$s_!-adb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5901af2c-cd6b-47b5-8c90-eabc90fcf92f_1616x1074.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Reddit Post - Opus 4.7</figcaption></figure></div><p>As they detailed in their <a href="https://www.anthropic.com/engineering/april-23-postmortem">April 23 postmortem</a>, three separate product-layer changes &#8212; a reasoning effort downgrade, a session-clearing bug that kept stripping thinking history, and a <strong>system prompt tweak</strong> that hurt coding quality &#8212; compounded into weeks of erratic behavior and wasted tokens. <strong>The model itself wasn't the problem. The harness was</strong>. And tellingly, the API was unaffected the entire time.</p><p>This should be a wake-up call. As more teams adopt coding agents into their workflows, it's imperative that each organization owns its own coding harness and simply uses frontier models as the backend. When you own the harness, you control the context limits, the system prompt, the tooling, and the feedback loops. You can enforce code quality standards, expected behaviors, and domain-specific safeguards &#8212; rather than relying on an opaque tool <strong>built for everyone that can regress overnight</strong>.</p><h2>A Halfway Measure: Overriding Claude Code's System Prompt</h2><blockquote><p>&#128161; I haven't investigated this approach deeply enough to vouch for it as a daily workflow</p></blockquote><p>If you're not ready to build your own harness, Claude Code does let you override its base system prompt entirely using a CLI flag:</p><pre><code>claude --system-prompt "Your custom system prompt here..."</code></pre><p>This replaces the default behavioral instructions while keeping the tool definitions intact &#8212; the model can still read files, write code, and run bash commands. What you're removing is the playbook that tells it <em>how</em> to use those tools: when to ask permission, how to structure edits, what conventions to follow.</p><p><strong>That's both the appeal and the catch</strong>. For real work, you'd need to write a proper system prompt that covers how you want the agent to approach tasks &#8212; your coding standards, when to ask vs. act, how to handle destructive commands, what patterns to follow. The <a href="https://github.com/Piebald-AI/claude-code-system-prompts">Piebald-AI/claude-code-system-prompts</a> repo tracks what the default prompt covers, which is a useful starting point for writing your own leaner version.</p><p>I haven't investigated this approach deeply enough to vouch for it as a daily workflow &#8212; it could end up being more overhead than it's worth. <strong>It's the kind of thing that's one yak-shave away from "I should just build my own harness," which is exactly what Pi already is.</strong></p><p>Still, if you want to feel the difference between a bloated default prompt and a focused one before committing to a different tool entirely, it's a worthwhile experiment.</p><h2>Caveats with Pi</h2><p><strong>1 - Security &amp; Sandboxing:</strong> By default, Pi has unrestricted bash access &#8212; it can execute any command without guardrails. There is a <a href="https://github.com/MansoorMajeed/Clawd/tree/main/extensions/permission-guard">permission-guard</a> extension that <strong>attempts</strong> to apply some safeguards, but it is <strong>not foolproof</strong>. You should treat this the same way you'd treat running Claude Code with <code>--dangerously-skip-permissions</code>. Personally, I run it in a dedicated VM with backups.</p><p>If you want something lighter, there are community extensions like <a href="https://github.com/carderne/pi-sandbox">pi-sandbox</a> that enforce OS-level filesystem and network restrictions &#8212; I haven't tried it myself, but it's worth investigating. Either way, don't run this on a machine where a bad <code>rm -rf</code> would ruin your day.</p><p><strong>2 - Billing:</strong> You <strong>cannot use a Claude Pro or Max subscription with Pi</strong> &#8212; Anthropic recently blocked third-party agents from using subscription billing. You'll need API keys with usage-based billing (or you can use the extra-usage with Claude Code). <strong>OpenAI's Codex subscriptions do appear to work directly with Pi</strong>, so if you're using OpenAI models that's a simpler path. But for Claude models, budget accordingly.</p><h2>Sold! Where to get started?</h2><p>You should not use my workflow &#8211; unless you think and work exactly like me (which probably is not the case). But what you should do is copy whatever extension or ideas you like from me (or others) and build your own workflow. That is where the real value is. You know your strengths and weakness, build a workflow around that.</p><ul><li><p>You can start at <a href="https://pi.dev/">pi.dev</a></p></li><li><p>Use these repos for ideas</p></li></ul><blockquote><h4><a href="https://github.com/MansoorMajeed/Clawd">GitHub - MansoorMajeed/Clawd: Custom Coding Agent Built on Pi</a></h4></blockquote><blockquote><h4><a href="https://github.com/mitsuhiko/agent-stuff">GitHub - mitsuhiko/agent-stuff: These are commands I use with agents, mostly Claude</a></h4></blockquote><h2>Closing thoughts</h2><p>I have been using Pi almost exclusively for work and in my experience, it has been absolutely amazing. I never have to fight with it. It does exactly what I ask it to do. It feels like a really smart tool rather than a yes-man</p><p>To be clear &#8212; this is not an anti-Claude post. I'm still using Anthropic's models. Opus 4.6 is excellent (can't say the same about 4.7 ). The problem was never the model; it was the layers of product logic sitting between me and it.</p><p>Yes, it's more work upfront than just using Claude Code out of the box. You're taking on the responsibility of writing your own system prompt, managing your own context, and running it safely. But the time you stop spending fighting your tools pays for itself quickly &#8212; and you end up with something that actually fits how you think and work.</p><blockquote><p>&#128161; Building a frontier model is hard. Building a harness is not. Most of the problems you're hitting aren't the model &#8212; they're the harness. And that harness was built for everyone, not for your company or your Engineers</p></blockquote><p>For organizations, the stakes are higher. Coding agents are here to stay, and the models will keep improving and changing. But what a company needs is the model &#8212; not a harness built for every developer on the planet. Each org's platform team should build a sensible harness that encodes their security policies, code quality standards, review processes, and domain-specific guardrails, and then let their engineers work the way they work &#8212; not the way Anthropic or OpenAI decided they should.</p><p>And as teams start depending on these tools for their daily output, operational basics become critical: pinning a model version, testing a new release internally before rolling it out, falling back to a different model with the same harness when something regresses. Without owning the harness, you have none of this. You're just hoping the next vendor update doesn't pull the rug out from under your entire engineering team overnight &#8212; and as the April 23 postmortem showed, that's not a hypothetical.</p>]]></content:encoded></item><item><title><![CDATA[A Better Way to Learn SRE & DevOps - An Interactive RoadMap]]></title><description><![CDATA[Learning DevOps / SRE is overwhelming because traditional roadmaps just throw 200 tools at you without explaining why they exist. I built an interactive, progressive roadmap to fix that. Stop memorizing tools and start building mental models.]]></description><link>https://newsletter.esc.sh/p/sre-devops-roadmap</link><guid isPermaLink="false">https://newsletter.esc.sh/p/sre-devops-roadmap</guid><dc:creator><![CDATA[Mansoor]]></dc:creator><pubDate>Sat, 07 Mar 2026 21:33:06 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/03343e67-3e35-4577-a3c7-f99a478acfec_2000x1087.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!DXHa!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06d0bdba-272a-468c-960c-3ea0413c2afd_2000x1087.jpeg 424w, https://substackcdn.com/image/fetch/$s_!DXHa!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06d0bdba-272a-468c-960c-3ea0413c2afd_2000x1087.jpeg 848w, https://substackcdn.com/image/fetch/$s_!DXHa!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06d0bdba-272a-468c-960c-3ea0413c2afd_2000x1087.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!DXHa!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06d0bdba-272a-468c-960c-3ea0413c2afd_2000x1087.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!DXHa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06d0bdba-272a-468c-960c-3ea0413c2afd_2000x1087.jpeg" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/06d0bdba-272a-468c-960c-3ea0413c2afd_2000x1087.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!DXHa!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06d0bdba-272a-468c-960c-3ea0413c2afd_2000x1087.jpeg 424w, https://substackcdn.com/image/fetch/$s_!DXHa!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06d0bdba-272a-468c-960c-3ea0413c2afd_2000x1087.jpeg 848w, https://substackcdn.com/image/fetch/$s_!DXHa!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06d0bdba-272a-468c-960c-3ea0413c2afd_2000x1087.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!DXHa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06d0bdba-272a-468c-960c-3ea0413c2afd_2000x1087.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><h2>TL;DR</h2><ul><li><p>I made a progressive, interactive roadmap for DevOps/SRE/Infra : <a href="https://roadmap.esc.sh/">https://roadmap.esc.sh/</a></p></li><li><p>It is Open Source : <a href="https://github.com/MansoorMajeed/infra-roadmap">https://github.com/MansoorMajeed/infra-roadmap</a></p></li><li><p>Planning to make free, long-form videos (no promises though) : Leave a comment here on topics / suggestions, and subscribe to the <a href="https://www.youtube.com/@esc-sh">YouTube Channel </a>I am planning to revive.</p></li></ul><div><hr></div><h2>Intro</h2><p>Let's get one thing out of the way: for the sake of this article, we are going to ignore the classic gatekeeping phrase, "SRE/DevOps is not a junior role." As much as some might agree with the history of that statement, the reality of the modern industry is very different. There <em>are</em> junior SRE/DevOps roles out there, and there are plenty of <strong>new engineers who are genuinely fascinated</strong> by the field, but they are absolutely overwhelmed by it.</p><blockquote><p>&#128161; <strong>Disclaimer</strong>: The following is a highly opinionated take. While I stand by it and strongly believe it is a great way to learn, I also understand that everyone's brain is wired differently and people have their own learning styles.</p></blockquote><h2>DevOps / SRE Field is Too Wide</h2><p>When a new engineer learns frontend development, the path is reasonably clear (one could argue that modern JS frameworks make this more complicated, I agree, but you could still get started fairly easy) : You learn HTML, CSS, JavaScript, a framework like React. You write some code, you immediately see what you built. The feedback loop is immediate and rewarding</p><p>Now compare that to SRE/DevOps. Feedback loop is non-existent most of the time, especially for new engineers. You are expected to understand OS internals, networking, pipelines &amp; an entire dictionary of jargons. To make matters worse, what a "DevOps Engineer" actually does at Company A might look completely different from Company B.</p><p>Where does one even start? The industry's usual advice sounds something like this:</p><blockquote><p>&#128161; "It is just so easy! Why don't you just learn everything about Linux, networking, software engineering, CI/CD, infrastructure as code, containers, Kubernetes, IAM, AWS, GCP, how the internet works... and while you are at it, also learn to implement SLAs, SLIs, SLOs, SAP, SSO, SEC, SSLC, and what have you. Shouldn't take more than a weekend."</p></blockquote><h2>Problem With Normal Roadmaps</h2><p>I am not dissing on roadmap.sh, I actually like it for what it is. But lets take a look at the DevOps roadmap from the perspective of a junior engineer &#8211; it is essentially a giant web of alien words.</p><blockquote><p>&#128161; In my opinion, the fundamental problem with standard roadmaps is that they show you <strong>what</strong> exists, but give zero context on <strong>why</strong> it exists or what actual problem we are trying to solve. This inadvertently trains beginners to focus on learning (or worse, memorizing) tools rather than understanding concepts.</p></blockquote><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!CMDI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43a4798f-d4bf-4334-a157-b160bb675d21_2408x1954.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!CMDI!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43a4798f-d4bf-4334-a157-b160bb675d21_2408x1954.png 424w, https://substackcdn.com/image/fetch/$s_!CMDI!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43a4798f-d4bf-4334-a157-b160bb675d21_2408x1954.png 848w, https://substackcdn.com/image/fetch/$s_!CMDI!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43a4798f-d4bf-4334-a157-b160bb675d21_2408x1954.png 1272w, https://substackcdn.com/image/fetch/$s_!CMDI!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43a4798f-d4bf-4334-a157-b160bb675d21_2408x1954.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!CMDI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43a4798f-d4bf-4334-a157-b160bb675d21_2408x1954.png" width="2000" height="1623" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/43a4798f-d4bf-4334-a157-b160bb675d21_2408x1954.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1623,&quot;width&quot;:2000,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!CMDI!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43a4798f-d4bf-4334-a157-b160bb675d21_2408x1954.png 424w, https://substackcdn.com/image/fetch/$s_!CMDI!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43a4798f-d4bf-4334-a157-b160bb675d21_2408x1954.png 848w, https://substackcdn.com/image/fetch/$s_!CMDI!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43a4798f-d4bf-4334-a157-b160bb675d21_2408x1954.png 1272w, https://substackcdn.com/image/fetch/$s_!CMDI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43a4798f-d4bf-4334-a157-b160bb675d21_2408x1954.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h2>A better Way to Learn</h2><p>If we want to build better infrastructure engineers, we need to completely change how we teach the subject.</p><h3>1 - Understand the End Goal: Running Software at Scale</h3><p>I think this is the biggest missing piece for most new engineers. Before we dive deep into Kubernetes, <strong>we need to understand what we are even doing all of this </strong><em><strong>for</strong></em>. To do that, you need a conceptual understanding of the Software Development Life Cycle (which is just a fancy way of saying how web apps and services are designed, built, and run).</p><p>Keep this end goal in mind at all times, and things will magically start to make sense. It is crucial to understand how software is written, tested, packaged, delivered, and executed. You don't necessarily need an in-depth mastery of every single aspect, but you absolutely need a high-level understanding of the full flow.</p><h3>2 - Always ask the "Why" - Learn Concepts Over Tools</h3><p>The actual problems of running software are basically permanent. Tools are just temporary answers to those permanent problems. Instead of blindly learning a tool, figure out what problem it was built to solve, and approach it from that perspective.</p><p>For example: instead of just memorizing <code>kubectl</code> commands, truly understand <em>what Kubernetes solves</em> first. Once you understand the problem space (container orchestration, self-healing, scaling), the tool itself makes sense. Plus, it makes it infinitely easier to learn the <em>next</em> tool that does a similar thing.</p><p>Without the foundational "why" behind it, you will just be repeating the learning process over and over. You will inevitably forget tools (that is totally expected) but you will never forget the concepts if you truly understand the core problem.</p><h3>3 - Make Connections to What You Know</h3><p>This ties heavily into the last two points. When you have to learn something new, figure out how it connects to what you already know&#8212;the concepts, the problems, the potential solutions, and exactly where this new tool fits into the big picture.</p><p>When you do this, your brain automatically categorizes the information and makes the connections. As you learn more, you will start connecting more dots, eventually building a robust mental model rather than just a list of tools and commands.</p><h3>4 - Use AI to fill in the Gaps</h3><p>In my opinion, this is the absolute best use of AI for learning. Pick your favorite LLM and ask it questions based on what you already know. Ask it to conceptually explain your knowledge gaps, and have it explain exactly how a new topic fits into your existing mental model.</p><h2>I built an interactive Roadmap tool</h2><p>I wanted a tool that actually reflected this learning philosophy. Since I couldn't find one that didn't immediately overwhelm the user, I built one: <strong><a href="https://roadmap.esc.sh/">roadmap.esc.sh</a></strong>.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!0qmY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0325e88-a825-416c-a07c-55fd70992468_2000x1087.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!0qmY!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0325e88-a825-416c-a07c-55fd70992468_2000x1087.jpeg 424w, https://substackcdn.com/image/fetch/$s_!0qmY!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0325e88-a825-416c-a07c-55fd70992468_2000x1087.jpeg 848w, https://substackcdn.com/image/fetch/$s_!0qmY!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0325e88-a825-416c-a07c-55fd70992468_2000x1087.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!0qmY!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0325e88-a825-416c-a07c-55fd70992468_2000x1087.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!0qmY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0325e88-a825-416c-a07c-55fd70992468_2000x1087.jpeg" width="2000" height="1087" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f0325e88-a825-416c-a07c-55fd70992468_2000x1087.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1087,&quot;width&quot;:2000,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!0qmY!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0325e88-a825-416c-a07c-55fd70992468_2000x1087.jpeg 424w, https://substackcdn.com/image/fetch/$s_!0qmY!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0325e88-a825-416c-a07c-55fd70992468_2000x1087.jpeg 848w, https://substackcdn.com/image/fetch/$s_!0qmY!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0325e88-a825-416c-a07c-55fd70992468_2000x1087.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!0qmY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0325e88-a825-416c-a07c-55fd70992468_2000x1087.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><blockquote><p>&#128161; It is an open-source, interactive and progressive roadmap for DevOps/SRE and Infrastructure in general that is designed to guide you through concepts logically through the "why" questions</p></blockquote><p>You can start at any "Zone". Click on the <strong>"Where do I start"</strong> if you are not sure</p><h3>Starting from Foundations</h3><p>Instead of jumping straight into SRE principles, let us start with baby steps and start from the very beginning.</p><ul><li><p>Each Node gives a short overview, you can save the progress as well. Of course the goal is not to give you all the information you need straight fro here, but it acts as a starting point with links to other resources for you to deepen your understanding.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!E0ns!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bf15b9b-14bf-4da8-993a-bb4f44e1eedb_2000x1499.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!E0ns!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bf15b9b-14bf-4da8-993a-bb4f44e1eedb_2000x1499.png 424w, https://substackcdn.com/image/fetch/$s_!E0ns!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bf15b9b-14bf-4da8-993a-bb4f44e1eedb_2000x1499.png 848w, https://substackcdn.com/image/fetch/$s_!E0ns!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bf15b9b-14bf-4da8-993a-bb4f44e1eedb_2000x1499.png 1272w, https://substackcdn.com/image/fetch/$s_!E0ns!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bf15b9b-14bf-4da8-993a-bb4f44e1eedb_2000x1499.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!E0ns!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bf15b9b-14bf-4da8-993a-bb4f44e1eedb_2000x1499.png" width="2000" height="1499" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4bf15b9b-14bf-4da8-993a-bb4f44e1eedb_2000x1499.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1499,&quot;width&quot;:2000,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!E0ns!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bf15b9b-14bf-4da8-993a-bb4f44e1eedb_2000x1499.png 424w, https://substackcdn.com/image/fetch/$s_!E0ns!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bf15b9b-14bf-4da8-993a-bb4f44e1eedb_2000x1499.png 848w, https://substackcdn.com/image/fetch/$s_!E0ns!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bf15b9b-14bf-4da8-993a-bb4f44e1eedb_2000x1499.png 1272w, https://substackcdn.com/image/fetch/$s_!E0ns!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bf15b9b-14bf-4da8-993a-bb4f44e1eedb_2000x1499.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3>Connecting the Dots</h3><p>The map is expanded progressively as you click the <strong>+ </strong>on the nodes that you find most interesting. It reveals a bunch of path that you can take</p><h3>The "Why" Nodes - Your Questions and Thoughts</h3><p>These are the questions or thoughts you may have when reading a concept or a tool. These questions allow you to pick the path that you find most natural to you.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!9jId!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5940bf54-c941-4fa9-a3bd-b61c50dfe180_2000x1499.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!9jId!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5940bf54-c941-4fa9-a3bd-b61c50dfe180_2000x1499.png 424w, https://substackcdn.com/image/fetch/$s_!9jId!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5940bf54-c941-4fa9-a3bd-b61c50dfe180_2000x1499.png 848w, https://substackcdn.com/image/fetch/$s_!9jId!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5940bf54-c941-4fa9-a3bd-b61c50dfe180_2000x1499.png 1272w, https://substackcdn.com/image/fetch/$s_!9jId!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5940bf54-c941-4fa9-a3bd-b61c50dfe180_2000x1499.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!9jId!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5940bf54-c941-4fa9-a3bd-b61c50dfe180_2000x1499.png" width="2000" height="1499" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5940bf54-c941-4fa9-a3bd-b61c50dfe180_2000x1499.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1499,&quot;width&quot;:2000,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!9jId!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5940bf54-c941-4fa9-a3bd-b61c50dfe180_2000x1499.png 424w, https://substackcdn.com/image/fetch/$s_!9jId!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5940bf54-c941-4fa9-a3bd-b61c50dfe180_2000x1499.png 848w, https://substackcdn.com/image/fetch/$s_!9jId!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5940bf54-c941-4fa9-a3bd-b61c50dfe180_2000x1499.png 1272w, https://substackcdn.com/image/fetch/$s_!9jId!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5940bf54-c941-4fa9-a3bd-b61c50dfe180_2000x1499.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h2>Next Steps : Long Form Video Series for SRE</h2><p>As I mentioned earlier, the fundamental problem with standard roadmaps and a lot of tutorials is that they show you <strong>what</strong> exists, but give zero context on <strong>why</strong> it exists or what actual problem we are trying to solve. This inadvertently trains beginners to focus on memorizing tools rather than understanding concepts.</p><p>I am thinking of making a free, long-form video series based on the exact flow of this roadmap. The idea is to keep the focus on the <strong>why</strong></p><p>We would start with the absolute basics and progressively introduce modern infrastructure tools to scale it to millions of users. Obviously, that is a highly ambitious goal and will take a ton of effort to produce, but I think it's the best way to actually see these concepts in action.</p><p>If you would be interested in watching a series like this, let me know! <strong>Drop a comment</strong> below with specific topics or problems you&#8217;d like to see covered, and go ahead and subscribe to my <a href="https://www.youtube.com/@esc-sh">YouTube channel</a>. <em>And yes, I know I haven't posted a DevOps video in 5 years &#128516; but we are fixing that! &#8211; Maybe (But no promises)</em></p>]]></content:encoded></item><item><title><![CDATA[Using Claude-Code on the go]]></title><description><![CDATA[In this blog post I explore how we can access claude-code (running on your laptop) from a mobile device while you are out and about -- all using free tools like SSH, Tmux and Tailscale]]></description><link>https://newsletter.esc.sh/p/using-claude-code-on-the-go</link><guid isPermaLink="false">https://newsletter.esc.sh/p/using-claude-code-on-the-go</guid><dc:creator><![CDATA[Mansoor]]></dc:creator><pubDate>Sat, 23 Aug 2025 15:22:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/928ad45e-9f7d-4cd9-9056-f60b311e167c_2000x1091.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!bjj5!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f86af81-fc14-4e88-84cd-1fc57958ae21_2000x1091.jpeg 424w, https://substackcdn.com/image/fetch/$s_!bjj5!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f86af81-fc14-4e88-84cd-1fc57958ae21_2000x1091.jpeg 848w, https://substackcdn.com/image/fetch/$s_!bjj5!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f86af81-fc14-4e88-84cd-1fc57958ae21_2000x1091.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!bjj5!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f86af81-fc14-4e88-84cd-1fc57958ae21_2000x1091.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!bjj5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f86af81-fc14-4e88-84cd-1fc57958ae21_2000x1091.jpeg" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2f86af81-fc14-4e88-84cd-1fc57958ae21_2000x1091.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!bjj5!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f86af81-fc14-4e88-84cd-1fc57958ae21_2000x1091.jpeg 424w, https://substackcdn.com/image/fetch/$s_!bjj5!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f86af81-fc14-4e88-84cd-1fc57958ae21_2000x1091.jpeg 848w, https://substackcdn.com/image/fetch/$s_!bjj5!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f86af81-fc14-4e88-84cd-1fc57958ae21_2000x1091.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!bjj5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f86af81-fc14-4e88-84cd-1fc57958ae21_2000x1091.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div><figcaption class="image-caption">you can never have too many monitors</figcaption></figure></div><h1>Introduction</h1><h2>What is this about?</h2><p>You're at home, and claude-code is busy building your next billion-dollar project. But you've got to run and don't want to lug your laptop with you. The problem? When Claude finishes an item on its to-do list, you need to be there to press a key to let it continue.</p><p>What if you could do that from your phone, without paying for anything extra? That's what this post is about.</p><h2>But why would you do that?</h2><p>Great question. Before you go any further, you should really ask yourself: do you <em>really</em> want to do this? If you're taking a walk, then take the walk. Don't bring Claude with you.</p><p>This guide is for that <em>one time</em> you're out and it would be a massive help to quickly review Claude's work, press <code>1</code>, <code>2</code>, or <code>Esc</code>, and then immediately put your phone away. If that's the case, read on.</p><h2>How does it work?</h2><p>It's quite simple. We'll enable SSH on your workstation, run <code>claude-code</code> inside <code>tmux</code> (to keep it alive), and connect to your workstation from your phone via an SSH client. To make it all work seamlessly while your phone is on 5G and your laptop is on your home network, we'll use a bit of magic called Tailscale.</p><h3>Is it free?</h3><p>Yes (except for claude-code, everything else is free)</p><h3>Is it safe?</h3><p>Yes. No one can see your traffic. Take a look at <a href="https://tailscale.com/blog/how-tailscale-works">how tailscale works</a> to get an idea, if you are interested. TL;DR is that Tailscale builds a private tunnel between your devices, no traffic flows through Tailscale's servers (it is peer to peer)</p><p>Is Tailscale safe? Yes; You can enable <a href="https://tailscale.com/kb/1226/tailnet-lock">Tailnet lock</a> to be extra secure</p><h2>Is there an easier way?</h2><p>Maybe. There are tools like <a href="https://github.com/amantus-ai/vibetunnel">VibeTunnel</a> . I have not tried it. I would rather rely on tried and tested solutions like SSH. Feel free to choose whatever works for you.</p><h1>How to set it up</h1><h2>Step 1 - Enable SSH on your workstation</h2><p>This of course depends on your operating system.</p><blockquote><p>&#128161; <strong>CAUTION: </strong>Enabling SSH on your device means any other device on the same network as you would be able to login to your device if they know your password. Make sure you have a strong password. Ideally, use key based authentication instead of password</p></blockquote><h3>MacOS</h3><p>In MacOS, you can enable SSH by going to settings -&gt; General -&gt; Sharing -&gt; Remote Login. Check Apple docs for more details</p><blockquote><h4><a href="https://support.apple.com/lt-lt/guide/mac-help/mchlp1066/mac">Allow a remote computer to access your Mac</a></h4></blockquote><p>It should look something like this once enabled</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!1QcF!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2900eaf2-6bdf-4b4e-8845-1b4be615db1d_470x214.png 424w, https://substackcdn.com/image/fetch/$s_!1QcF!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2900eaf2-6bdf-4b4e-8845-1b4be615db1d_470x214.png 848w, https://substackcdn.com/image/fetch/$s_!1QcF!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2900eaf2-6bdf-4b4e-8845-1b4be615db1d_470x214.png 1272w, https://substackcdn.com/image/fetch/$s_!1QcF!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2900eaf2-6bdf-4b4e-8845-1b4be615db1d_470x214.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!1QcF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2900eaf2-6bdf-4b4e-8845-1b4be615db1d_470x214.png" width="470" height="214" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2900eaf2-6bdf-4b4e-8845-1b4be615db1d_470x214.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:214,&quot;width&quot;:470,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!1QcF!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2900eaf2-6bdf-4b4e-8845-1b4be615db1d_470x214.png 424w, https://substackcdn.com/image/fetch/$s_!1QcF!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2900eaf2-6bdf-4b4e-8845-1b4be615db1d_470x214.png 848w, https://substackcdn.com/image/fetch/$s_!1QcF!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2900eaf2-6bdf-4b4e-8845-1b4be615db1d_470x214.png 1272w, https://substackcdn.com/image/fetch/$s_!1QcF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2900eaf2-6bdf-4b4e-8845-1b4be615db1d_470x214.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><h3>Linux</h3><p>Depending on your Linux distro, install and enable SSH server.</p><ul><li><p>Ubuntu : <a href="https://documentation.ubuntu.com/server/how-to/security/openssh-server/">HERE</a></p></li><li><p>Fedora : <a href="https://docs.fedoraproject.org/en-US/fedora/f40/system-administrators-guide/infrastructure-services/OpenSSH/">HERE</a></p></li><li><p>Arch BTW : <a href="https://wiki.archlinux.org/title/OpenSSH">HERE</a></p></li></ul><h3>Windows</h3><p>If you plan to use <code>claude-code</code> with WSL2, you can enable SSH on WSL2 and basically have it work like Linux. Check <a href="https://gist.github.com/MansoorMajeed/1af34b0900c174bf247cd1769efa458c">HERE</a></p><h3>Step 1.1 - Disable Password Login (Windows WSL / Linux)</h3><blockquote><p>&#128161; Disabling password based SSH auth on <strong>MacOS </strong>requires messing with PAM configuration, I have not tried doing so, try it at your own risk.</p></blockquote><p>In your <code>/etc/ssh/sshd_config</code> ensure you have <code>PasswordAuthentication no</code> to prevent password based authentication</p><h2>Step 2 - Install and setup Tmux</h2><p>Tmux is a terminal multiplexer &#8211; which is a fancy way of saying that you can run a terminal session with any number of tabs/windows on a remote host and detach and attach to it any time.</p><p>While not 100% required, Tmux makes it so much more convenient. You can easily switch between your phone and your workstation &#8211; continuity is preserved. Additionally, even if you lose internet access on your phone, you can simply reconnect to the Tmux session and resume</p><h3>Installing Tmux</h3><ul><li><p>Ubuntu <code>sudo apt install tmux</code></p></li><li><p>Fedora <code>sudo dnf install tmux</code></p></li><li><p>Arch BTW <code>sudo pacman -S tmux</code></p></li></ul><h3>Starting Tmux</h3><p>Open a terminal and simply type</p><pre><code>tmux new -s claude-demo</code></pre><p>This will create a new tmux session with the name <code>claude-demo</code></p><p>All of tmux commands are done in a funny way. That is, you press <code>Ctrl+b</code> then let go and then immediately press one of the keys. For example</p><ul><li><p><code>Ctrl+b</code>, let go completely, press <code>c</code> : Open a new tab</p></li><li><p><code>Ctrl+b</code>, release, press <code>1</code> : switch to tab 1</p></li><li><p><code>Ctrl+b</code>, release, <code>d</code> : detach from Tmux. You will be back on your terminal before you started the tmux session. Now, even if you exit or close your terminal, <code>Tmux</code> is still running in the background, waiting for you to re-attach to it</p></li><li><p><code>tmux ls</code> : List Tmux sessions</p></li><li><p><code>tmux a -t claude-demo</code> : Attach again to the running session</p></li></ul><p>Great, now you have a Tmux session running on your workstation</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!wTZj!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7db56056-9bc0-4854-b0aa-c9b11b214783_1264x827.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!wTZj!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7db56056-9bc0-4854-b0aa-c9b11b214783_1264x827.png 424w, https://substackcdn.com/image/fetch/$s_!wTZj!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7db56056-9bc0-4854-b0aa-c9b11b214783_1264x827.png 848w, https://substackcdn.com/image/fetch/$s_!wTZj!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7db56056-9bc0-4854-b0aa-c9b11b214783_1264x827.png 1272w, https://substackcdn.com/image/fetch/$s_!wTZj!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7db56056-9bc0-4854-b0aa-c9b11b214783_1264x827.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!wTZj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7db56056-9bc0-4854-b0aa-c9b11b214783_1264x827.png" width="1264" height="827" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7db56056-9bc0-4854-b0aa-c9b11b214783_1264x827.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:827,&quot;width&quot;:1264,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!wTZj!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7db56056-9bc0-4854-b0aa-c9b11b214783_1264x827.png 424w, https://substackcdn.com/image/fetch/$s_!wTZj!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7db56056-9bc0-4854-b0aa-c9b11b214783_1264x827.png 848w, https://substackcdn.com/image/fetch/$s_!wTZj!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7db56056-9bc0-4854-b0aa-c9b11b214783_1264x827.png 1272w, https://substackcdn.com/image/fetch/$s_!wTZj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7db56056-9bc0-4854-b0aa-c9b11b214783_1264x827.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Terminal running Claude on Tmux</figcaption></figure></div><h2>Step 3 - SSH Client for your Phone</h2><h3>Android</h3><p>There are several SSH client apps available for free. I use <code>Termius</code>. <code>Termux</code> is another option. You can choose whichever you like</p><h3>iOS</h3><p><code>Termius</code> is popular among IOS users too. Or you can find another one you like better</p><h2>Step 4 - Generating SSH keys for your phone</h2><p>Depending on the app you use, these steps might vary. You could generate them on your workstation and copy the private keys to your phone's ssh client too. This is fully up to you.</p><p>I will show you how to generate ssh keys for your Phone using <code>Termius</code></p><ul><li><p>Open Termius</p></li><li><p>Go to <code>Vaults</code> -&gt; <code>Keychain</code> -&gt; <code>+</code> button</p></li><li><p>Choose <code>New SSH key...</code></p></li><li><p>Choose <code>Generate new SSH key</code></p><ul><li><p>Give it a name you like, it does not matter</p></li><li><p>Type: <code>ED25519</code></p></li><li><p>Optionally choose a passphrase.</p></li><li><p><code>Generate</code></p></li></ul></li><li><p>Click <code>Export or share a public key</code></p></li></ul><p>Now, you should have your ssh public key available. We need to get this key to your workstation so you can allow this key to access SSH. This public key is completely safe to be shown in public, so you can easily copy it to a note app and transfer it to your workstation</p><h2>Step 5 - Allowing your phone's SSH key on your workstation</h2><h3>WSL/Linux</h3><p>Edit or create the file <code>~/.ssh/authorized_keys</code> and add your public key there</p><h3>MacOS</h3><h4>Security considerations for MacOS</h4><p>Enabling public key only SSH on MacOS requires us to edit the pam configuration files. This can mess up things if not done properly. Because of this, the easiest solution is to keep using password for ssh. However this puts us in a risky situation &#8211; anyone on the same network could attempt to brute force your password. With that, I recommend you try some or all of the following</p><ul><li><p>Use a strong password for your Mac user</p></li><li><p>Disable remote login (SSH) while you are on insecure wifi networks</p></li><li><p>Use MacOS firewall to allow SSH only from the network interface used by Tailscale and drop everything else.</p></li></ul><h2>Step 6 - Install Tailscale</h2><p>For your workstation, go <a href="https://tailscale.com/download">HERE</a>, download and install Tailscale. For your mobile devices, you can download <code>Tailscale</code> from your App store / Play store. Login to both devices using the same account.</p><p>Once you have logged in and connected, you should be able to see your devices in the Tailscale admin page.</p><blockquote><p>&#128161; Note the names of the machines you have -- by default it uses the hostname</p></blockquote><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!94uB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa51678e4-50f9-4d44-848f-5867db481c8b_600x906.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!94uB!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa51678e4-50f9-4d44-848f-5867db481c8b_600x906.png 424w, https://substackcdn.com/image/fetch/$s_!94uB!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa51678e4-50f9-4d44-848f-5867db481c8b_600x906.png 848w, https://substackcdn.com/image/fetch/$s_!94uB!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa51678e4-50f9-4d44-848f-5867db481c8b_600x906.png 1272w, https://substackcdn.com/image/fetch/$s_!94uB!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa51678e4-50f9-4d44-848f-5867db481c8b_600x906.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!94uB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa51678e4-50f9-4d44-848f-5867db481c8b_600x906.png" width="600" height="906" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a51678e4-50f9-4d44-848f-5867db481c8b_600x906.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:906,&quot;width&quot;:600,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!94uB!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa51678e4-50f9-4d44-848f-5867db481c8b_600x906.png 424w, https://substackcdn.com/image/fetch/$s_!94uB!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa51678e4-50f9-4d44-848f-5867db481c8b_600x906.png 848w, https://substackcdn.com/image/fetch/$s_!94uB!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa51678e4-50f9-4d44-848f-5867db481c8b_600x906.png 1272w, https://substackcdn.com/image/fetch/$s_!94uB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa51678e4-50f9-4d44-848f-5867db481c8b_600x906.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Tailscale mobile app</figcaption></figure></div><p>Here you can see my devices, <code>helium</code> is the Windows PC I will use this for post. I have WSL2 running on Windows.</p><h2>Step 7 - Adding SSH host to your mobile device</h2><p>We are almost there. Let's add SSH connection to your workstation from your mobile device.</p><p>Open the SSH client app (Termius in my case)</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!WALi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c75569a-a35f-43f8-9326-17481e16e9cf_500x1099.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!WALi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c75569a-a35f-43f8-9326-17481e16e9cf_500x1099.png 424w, https://substackcdn.com/image/fetch/$s_!WALi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c75569a-a35f-43f8-9326-17481e16e9cf_500x1099.png 848w, https://substackcdn.com/image/fetch/$s_!WALi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c75569a-a35f-43f8-9326-17481e16e9cf_500x1099.png 1272w, https://substackcdn.com/image/fetch/$s_!WALi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c75569a-a35f-43f8-9326-17481e16e9cf_500x1099.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!WALi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c75569a-a35f-43f8-9326-17481e16e9cf_500x1099.png" width="500" height="1099" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4c75569a-a35f-43f8-9326-17481e16e9cf_500x1099.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1099,&quot;width&quot;:500,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!WALi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c75569a-a35f-43f8-9326-17481e16e9cf_500x1099.png 424w, https://substackcdn.com/image/fetch/$s_!WALi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c75569a-a35f-43f8-9326-17481e16e9cf_500x1099.png 848w, https://substackcdn.com/image/fetch/$s_!WALi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c75569a-a35f-43f8-9326-17481e16e9cf_500x1099.png 1272w, https://substackcdn.com/image/fetch/$s_!WALi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c75569a-a35f-43f8-9326-17481e16e9cf_500x1099.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Add new host to Termius</figcaption></figure></div><ol><li><p>Alias : You can set it to whatever you like</p></li><li><p>Hostname: This should match the Tailscale name (hostname) of your workstation</p></li><li><p>Port : In Linux and MacOS it <strong>would be most probably <code>22</code></strong>, in my case it is <code>2222</code> because I am connecting to a WSL2 instance where SSH is configured to run on <code>2222</code></p></li><li><p>Username: Replace it with your username for your workstation</p></li><li><p>Password: Leave it empty for Linux and Windows/WSL, enter your password for MacOS</p></li><li><p>For Linux and Windows/WSL, choose the SSH key you created in Step 4</p></li><li><p>Save it</p></li></ol><h2>Step 8 - Connecting and Vibing</h2><p>You are ready to make your billion dollar SaaS.</p><p>Re-attach using <code>tmux a -t claude-demo</code> (or create a new session using <code>tmux new -s claude-demo</code>) &#8211; You can do this from your phone or workstation, it does not matter</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!r-wP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff22e016d-dbad-4b50-995b-a6e6d822475b_1223x904.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!r-wP!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff22e016d-dbad-4b50-995b-a6e6d822475b_1223x904.png 424w, https://substackcdn.com/image/fetch/$s_!r-wP!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff22e016d-dbad-4b50-995b-a6e6d822475b_1223x904.png 848w, https://substackcdn.com/image/fetch/$s_!r-wP!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff22e016d-dbad-4b50-995b-a6e6d822475b_1223x904.png 1272w, https://substackcdn.com/image/fetch/$s_!r-wP!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff22e016d-dbad-4b50-995b-a6e6d822475b_1223x904.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!r-wP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff22e016d-dbad-4b50-995b-a6e6d822475b_1223x904.png" width="1223" height="904" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f22e016d-dbad-4b50-995b-a6e6d822475b_1223x904.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:904,&quot;width&quot;:1223,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!r-wP!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff22e016d-dbad-4b50-995b-a6e6d822475b_1223x904.png 424w, https://substackcdn.com/image/fetch/$s_!r-wP!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff22e016d-dbad-4b50-995b-a6e6d822475b_1223x904.png 848w, https://substackcdn.com/image/fetch/$s_!r-wP!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff22e016d-dbad-4b50-995b-a6e6d822475b_1223x904.png 1272w, https://substackcdn.com/image/fetch/$s_!r-wP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff22e016d-dbad-4b50-995b-a6e6d822475b_1223x904.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Claude is not working for me. Anyone else have this issue?</figcaption></figure></div><p>Now that I have Claude-Code running on my PC, I can access it from my phone. Running <code>tmux ls</code> from phone, I can see the <code>claude-demo</code> tmux session</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!JXy8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8254a4c3-1cec-433c-9952-779b752b5bb3_1080x541.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!JXy8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8254a4c3-1cec-433c-9952-779b752b5bb3_1080x541.png 424w, https://substackcdn.com/image/fetch/$s_!JXy8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8254a4c3-1cec-433c-9952-779b752b5bb3_1080x541.png 848w, https://substackcdn.com/image/fetch/$s_!JXy8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8254a4c3-1cec-433c-9952-779b752b5bb3_1080x541.png 1272w, https://substackcdn.com/image/fetch/$s_!JXy8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8254a4c3-1cec-433c-9952-779b752b5bb3_1080x541.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!JXy8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8254a4c3-1cec-433c-9952-779b752b5bb3_1080x541.png" width="1080" height="541" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8254a4c3-1cec-433c-9952-779b752b5bb3_1080x541.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:541,&quot;width&quot;:1080,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!JXy8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8254a4c3-1cec-433c-9952-779b752b5bb3_1080x541.png 424w, https://substackcdn.com/image/fetch/$s_!JXy8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8254a4c3-1cec-433c-9952-779b752b5bb3_1080x541.png 848w, https://substackcdn.com/image/fetch/$s_!JXy8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8254a4c3-1cec-433c-9952-779b752b5bb3_1080x541.png 1272w, https://substackcdn.com/image/fetch/$s_!JXy8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8254a4c3-1cec-433c-9952-779b752b5bb3_1080x541.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>I can attach to it and resume work by simply doing <code>tmux a -t claude-demo</code></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!yb8W!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10ae7618-db1c-43ca-bde6-c3380681133e_1080x1601.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!yb8W!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10ae7618-db1c-43ca-bde6-c3380681133e_1080x1601.png 424w, https://substackcdn.com/image/fetch/$s_!yb8W!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10ae7618-db1c-43ca-bde6-c3380681133e_1080x1601.png 848w, https://substackcdn.com/image/fetch/$s_!yb8W!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10ae7618-db1c-43ca-bde6-c3380681133e_1080x1601.png 1272w, https://substackcdn.com/image/fetch/$s_!yb8W!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10ae7618-db1c-43ca-bde6-c3380681133e_1080x1601.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!yb8W!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10ae7618-db1c-43ca-bde6-c3380681133e_1080x1601.png" width="1080" height="1601" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/10ae7618-db1c-43ca-bde6-c3380681133e_1080x1601.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1601,&quot;width&quot;:1080,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!yb8W!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10ae7618-db1c-43ca-bde6-c3380681133e_1080x1601.png 424w, https://substackcdn.com/image/fetch/$s_!yb8W!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10ae7618-db1c-43ca-bde6-c3380681133e_1080x1601.png 848w, https://substackcdn.com/image/fetch/$s_!yb8W!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10ae7618-db1c-43ca-bde6-c3380681133e_1080x1601.png 1272w, https://substackcdn.com/image/fetch/$s_!yb8W!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10ae7618-db1c-43ca-bde6-c3380681133e_1080x1601.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Claude code accessed from phone</figcaption></figure></div><h2>Step 8 - (Optional) Reading and Editing code on your phone</h2><p>If you want to quickly view some code from your workstation, you can achieve this very easily using <code>code-server</code>, which is a VS code server running on your workstation that can be accessed from your phone's browser</p><h3>Installing Code Server</h3><p>Check <a href="https://github.com/coder/code-server">HERE</a> on how to install this on your operating system.</p><p>But TL;DR is running</p><pre><code>curl -fsSL https://code-server.dev/install.sh | sh</code></pre><h3>Running Code Server</h3><p>Once installed, follow the docs to set it up to start using systemd (if on Linux/WSL2). Otherwise you can manually start it by running <code>code-server</code> on your workstation</p><p>When you run it, it will show you the config file <code>code-server</code> will be using, something like <code>/home/your username/.config/code-server/config.yaml</code></p><p>Edit this file</p><pre><code>vim /home/mansoor/.config/code-server/config.yaml</code></pre><p>And change the <code>bind-addr</code> to <code>0.0.0.0:8080</code> so that code-server will listen on all network interfaces (You can change the port number if you would like something else). The same file will also show the password you will use to login to it, take note of it. You</p><blockquote><p>&#128161; This makes your code-server be accessible over plain text http on your local network. If you use it on insecure wifi networks, you should set up https. Check docs <a href="https://coder.com/docs/code-server/guide">HERE</a></p></blockquote><p>After changing the <code>bind-addr</code>, restart <code>code-server</code></p><h3>Accessing Code Server</h3><p>From your phone's browser, access <code>your-workstation-hostname:8080</code></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ZEPh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97b4cae8-f7b1-4806-8836-1ee1de59078f_1079x1618.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ZEPh!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97b4cae8-f7b1-4806-8836-1ee1de59078f_1079x1618.png 424w, https://substackcdn.com/image/fetch/$s_!ZEPh!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97b4cae8-f7b1-4806-8836-1ee1de59078f_1079x1618.png 848w, https://substackcdn.com/image/fetch/$s_!ZEPh!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97b4cae8-f7b1-4806-8836-1ee1de59078f_1079x1618.png 1272w, https://substackcdn.com/image/fetch/$s_!ZEPh!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97b4cae8-f7b1-4806-8836-1ee1de59078f_1079x1618.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ZEPh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97b4cae8-f7b1-4806-8836-1ee1de59078f_1079x1618.png" width="1079" height="1618" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/97b4cae8-f7b1-4806-8836-1ee1de59078f_1079x1618.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1618,&quot;width&quot;:1079,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!ZEPh!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97b4cae8-f7b1-4806-8836-1ee1de59078f_1079x1618.png 424w, https://substackcdn.com/image/fetch/$s_!ZEPh!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97b4cae8-f7b1-4806-8836-1ee1de59078f_1079x1618.png 848w, https://substackcdn.com/image/fetch/$s_!ZEPh!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97b4cae8-f7b1-4806-8836-1ee1de59078f_1079x1618.png 1272w, https://substackcdn.com/image/fetch/$s_!ZEPh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97b4cae8-f7b1-4806-8836-1ee1de59078f_1079x1618.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">code-server on phone</figcaption></figure></div>]]></content:encoded></item><item><title><![CDATA[How Kubernetes Runs Containers : A Practical Deep Dive]]></title><description><![CDATA[Taking a deep dive into how Kubernetes runs containers as Linux processes]]></description><link>https://newsletter.esc.sh/p/kubernetes-containers-linux-processes</link><guid isPermaLink="false">https://newsletter.esc.sh/p/kubernetes-containers-linux-processes</guid><dc:creator><![CDATA[Mansoor]]></dc:creator><pubDate>Sat, 26 Apr 2025 22:53:10 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/cdbf6def-ff02-45cb-ada6-dbd7a9c59f55_1452x779.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!SjOa!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41f03a68-6c01-46bc-a2d9-02e8ff9f9a9a_1452x779.png 424w, https://substackcdn.com/image/fetch/$s_!SjOa!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41f03a68-6c01-46bc-a2d9-02e8ff9f9a9a_1452x779.png 848w, https://substackcdn.com/image/fetch/$s_!SjOa!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41f03a68-6c01-46bc-a2d9-02e8ff9f9a9a_1452x779.png 1272w, https://substackcdn.com/image/fetch/$s_!SjOa!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41f03a68-6c01-46bc-a2d9-02e8ff9f9a9a_1452x779.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!SjOa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41f03a68-6c01-46bc-a2d9-02e8ff9f9a9a_1452x779.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/41f03a68-6c01-46bc-a2d9-02e8ff9f9a9a_1452x779.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!SjOa!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41f03a68-6c01-46bc-a2d9-02e8ff9f9a9a_1452x779.png 424w, https://substackcdn.com/image/fetch/$s_!SjOa!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41f03a68-6c01-46bc-a2d9-02e8ff9f9a9a_1452x779.png 848w, https://substackcdn.com/image/fetch/$s_!SjOa!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41f03a68-6c01-46bc-a2d9-02e8ff9f9a9a_1452x779.png 1272w, https://substackcdn.com/image/fetch/$s_!SjOa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41f03a68-6c01-46bc-a2d9-02e8ff9f9a9a_1452x779.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><h2>Introduction</h2><p>One might ask, with all the advancements in container orchestration, why even bother looking under the hood? Well, depending on whether you're running a managed or self-managed cluster, the answer could vary &#8212; sometimes it matters a lot. Beyond that, understanding how Kubernetes actually works under the hood is crucial when you're managing clusters at scale. It helps with troubleshooting, performance tuning, and designing more resilient systems.</p><p>And last but not least, it's just fun!</p><h2>A Dumb Container Orchestration Model</h2><p>First of all, let us take a look at how a very primitive container orchestration system might work. This will give us an idea of how everything fit together.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!i9Pd!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6af65d2-d9f2-4456-80bc-af28d8498ff2_1840x986.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!i9Pd!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6af65d2-d9f2-4456-80bc-af28d8498ff2_1840x986.png 424w, https://substackcdn.com/image/fetch/$s_!i9Pd!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6af65d2-d9f2-4456-80bc-af28d8498ff2_1840x986.png 848w, https://substackcdn.com/image/fetch/$s_!i9Pd!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6af65d2-d9f2-4456-80bc-af28d8498ff2_1840x986.png 1272w, https://substackcdn.com/image/fetch/$s_!i9Pd!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6af65d2-d9f2-4456-80bc-af28d8498ff2_1840x986.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!i9Pd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6af65d2-d9f2-4456-80bc-af28d8498ff2_1840x986.png" width="1840" height="986" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a6af65d2-d9f2-4456-80bc-af28d8498ff2_1840x986.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:986,&quot;width&quot;:1840,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!i9Pd!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6af65d2-d9f2-4456-80bc-af28d8498ff2_1840x986.png 424w, https://substackcdn.com/image/fetch/$s_!i9Pd!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6af65d2-d9f2-4456-80bc-af28d8498ff2_1840x986.png 848w, https://substackcdn.com/image/fetch/$s_!i9Pd!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6af65d2-d9f2-4456-80bc-af28d8498ff2_1840x986.png 1272w, https://substackcdn.com/image/fetch/$s_!i9Pd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6af65d2-d9f2-4456-80bc-af28d8498ff2_1840x986.png 1456w" sizes="100vw"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>We have a <strong>single Linux machine</strong> (no clustering, sorry)</p><ul><li><p>An <strong>operator</strong> (that is you), defines a simple spec (maybe a text file? a yaml file maybe?)</p><ul><li><p>This defines what you want to run. Here, we run an app called <code>nginx</code></p></li></ul></li><li><p>A <strong>service</strong> (or controller) exposes an API where the user (you) can send this spec &#8212; either using a tool or making an API call.</p><ul><li><p>It then <strong>translates</strong> that request into something the <strong>container manager</strong> (example Docker) understands</p></li></ul></li><li><p>The container manager (docker /containerd) will then use <strong>Linux's features</strong> (for isolation and resource limits) to run these applications as <strong>Linux processes</strong></p></li></ul><p>Of course, this is a really dumbed-down view &#8211; it&#8217;s only meant to give you a rough idea of how container orchestration might work. A real-world production-grade orchestrator like <strong>Kubernetes</strong> is a lot more complex &#8212; but not because the <strong>core idea is different</strong>,<br>just because Kubernetes needs to handle <strong>a lot more things</strong>: clustering, networking, scheduling, health checks, scaling, and more.</p><p>Maybe take a look at the <a href="https://kubernetes.io/docs/concepts/architecture/">Kubernetes architecture</a> to compare the mental model</p><h2>Quick refresher - How Linux Runs Processes</h2><p>Before we dive into Kubernetes, let's take a quick look at how Linux itself runs processes. At the end of the day, a container is just a Linux process(s) under the hood.</p><p>When you start a program on Linux, the OS creates a <strong>process. </strong>This process has a unique ID called <strong>PID (Process ID). </strong>Each process has its own memory, file descriptors, etc.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!H8ND!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec8a55e1-06f8-4a3d-be13-0a35c765e9ce_996x286.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!H8ND!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec8a55e1-06f8-4a3d-be13-0a35c765e9ce_996x286.png 424w, https://substackcdn.com/image/fetch/$s_!H8ND!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec8a55e1-06f8-4a3d-be13-0a35c765e9ce_996x286.png 848w, https://substackcdn.com/image/fetch/$s_!H8ND!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec8a55e1-06f8-4a3d-be13-0a35c765e9ce_996x286.png 1272w, https://substackcdn.com/image/fetch/$s_!H8ND!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec8a55e1-06f8-4a3d-be13-0a35c765e9ce_996x286.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!H8ND!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec8a55e1-06f8-4a3d-be13-0a35c765e9ce_996x286.png" width="996" height="286" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ec8a55e1-06f8-4a3d-be13-0a35c765e9ce_996x286.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:286,&quot;width&quot;:996,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!H8ND!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec8a55e1-06f8-4a3d-be13-0a35c765e9ce_996x286.png 424w, https://substackcdn.com/image/fetch/$s_!H8ND!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec8a55e1-06f8-4a3d-be13-0a35c765e9ce_996x286.png 848w, https://substackcdn.com/image/fetch/$s_!H8ND!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec8a55e1-06f8-4a3d-be13-0a35c765e9ce_996x286.png 1272w, https://substackcdn.com/image/fetch/$s_!H8ND!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec8a55e1-06f8-4a3d-be13-0a35c765e9ce_996x286.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">ps aux - showing processes in Linux</figcaption></figure></div><p>Linux organizes these processes into a <strong>tree </strong>where each process has a parent &#8211; except the very first one (called <code>init</code> or <code>systemd</code>). For example, this is how the tree looks like, using the <code>pstree</code> command</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!S41B!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd2f406d-0286-4f71-a734-37ed46d242ae_1147x504.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!S41B!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd2f406d-0286-4f71-a734-37ed46d242ae_1147x504.png 424w, https://substackcdn.com/image/fetch/$s_!S41B!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd2f406d-0286-4f71-a734-37ed46d242ae_1147x504.png 848w, https://substackcdn.com/image/fetch/$s_!S41B!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd2f406d-0286-4f71-a734-37ed46d242ae_1147x504.png 1272w, https://substackcdn.com/image/fetch/$s_!S41B!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd2f406d-0286-4f71-a734-37ed46d242ae_1147x504.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!S41B!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd2f406d-0286-4f71-a734-37ed46d242ae_1147x504.png" width="1147" height="504" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fd2f406d-0286-4f71-a734-37ed46d242ae_1147x504.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:504,&quot;width&quot;:1147,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!S41B!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd2f406d-0286-4f71-a734-37ed46d242ae_1147x504.png 424w, https://substackcdn.com/image/fetch/$s_!S41B!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd2f406d-0286-4f71-a734-37ed46d242ae_1147x504.png 848w, https://substackcdn.com/image/fetch/$s_!S41B!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd2f406d-0286-4f71-a734-37ed46d242ae_1147x504.png 1272w, https://substackcdn.com/image/fetch/$s_!S41B!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd2f406d-0286-4f71-a734-37ed46d242ae_1147x504.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">pstree output</figcaption></figure></div><h3>Isolating different processes (Namespaces and cgroups)</h3><p>By default, processes share resources such as file systems, network interfaces, etc. But we need some way to isolate them. But why?</p><p>Without isolation, processes can see and interfere with each other. A misbehaving or compromised app process could inspect other processes, access shared files, or disrupt the network. Additionally, we don't want one bad process to starve others of precious resources such as CPU, memory, disk, or network.</p><p>For this, Linux uses two major features:</p><ul><li><p><strong>Namespaces</strong>: These isolate what a process can see &#8212; network interfaces, mounted filesystems, process IDs, etc.</p></li><li><p><strong>cgroups (control groups)</strong>: These control how much CPU, memory, or I/O a group of processes can use.</p></li></ul><p>Together, namespaces and cgroups let us sandbox processes in a lightweight way &#8212; in contrast to running a full virtual machine for each application.</p><h2>Deep Dive into Kubernetes internals</h2><p>Alright, now that we have refreshed our memory on how Linux processes work, let's take a look at Kubernetes. For this exercise, I want to run a Kubernetes cluster, deploy a pod, and take a look at how it runs as processes.</p><p><br>I will use <strong>k3s</strong> to create a cluster on a Linux VM.</p><h3>Installing Kubernetes (k3s)</h3><p>Installing k3s is fairly easy. Follow the <a href="https://docs.k3s.io/quick-start">quick start guide</a></p><pre><code>curl -sfL https://get.k3s.io | sh -</code></pre><blockquote><p>&#128161; Obligatory notice to NOT copy paste commands from the internet</p></blockquote><h3>Let's look around &#8211; What's running after k3s install</h3><p>Run <code>ps</code> and <code>grep</code> for the k3s related process names</p><pre><code>ps aux | grep -E 'k3s|containerd|etcd|flannel|kube'</code></pre><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!0GIE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa481fda-5304-4371-b855-b63e80cffff9_1556x662.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!0GIE!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa481fda-5304-4371-b855-b63e80cffff9_1556x662.png 424w, https://substackcdn.com/image/fetch/$s_!0GIE!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa481fda-5304-4371-b855-b63e80cffff9_1556x662.png 848w, https://substackcdn.com/image/fetch/$s_!0GIE!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa481fda-5304-4371-b855-b63e80cffff9_1556x662.png 1272w, https://substackcdn.com/image/fetch/$s_!0GIE!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa481fda-5304-4371-b855-b63e80cffff9_1556x662.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!0GIE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa481fda-5304-4371-b855-b63e80cffff9_1556x662.png" width="1556" height="662" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/aa481fda-5304-4371-b855-b63e80cffff9_1556x662.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:662,&quot;width&quot;:1556,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!0GIE!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa481fda-5304-4371-b855-b63e80cffff9_1556x662.png 424w, https://substackcdn.com/image/fetch/$s_!0GIE!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa481fda-5304-4371-b855-b63e80cffff9_1556x662.png 848w, https://substackcdn.com/image/fetch/$s_!0GIE!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa481fda-5304-4371-b855-b63e80cffff9_1556x662.png 1272w, https://substackcdn.com/image/fetch/$s_!0GIE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa481fda-5304-4371-b855-b63e80cffff9_1556x662.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">k3s related processes</figcaption></figure></div><p>That's a lot of processes. But we don't need to worry about them all. Only few are important in this context</p><ul><li><p><strong>k3s server : </strong>This is a lightweight Kubernetes server that bundles a few components together</p><ul><li><p><strong>It combines the API server, kube-proxy, kubelet, scheduler, and a bunch of other services.</strong></p></li><li><p>You can read more about the architecture in the k3s documentation.</p></li></ul><ul><li><p>This also contains an <strong>embedded etcd</strong> server as the database</p></li></ul></li><li><p><strong>containerd</strong> : This is the container runtime which is actually responsible for managing containers</p></li></ul><p>We can ignore the rest for now</p><h3>Let's take a look at the current namespaces</h3><blockquote><p>&#128161; Note that the <code>namespace</code> here means the Linux namespace and is different from the Kubernetes namespace</p></blockquote><p>We can use the command <code>lsns</code> to list the current namespaces</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!SS0s!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2597bc4d-ee7f-40b9-af15-eb96a818e0e1_1072x395.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!SS0s!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2597bc4d-ee7f-40b9-af15-eb96a818e0e1_1072x395.png 424w, https://substackcdn.com/image/fetch/$s_!SS0s!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2597bc4d-ee7f-40b9-af15-eb96a818e0e1_1072x395.png 848w, https://substackcdn.com/image/fetch/$s_!SS0s!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2597bc4d-ee7f-40b9-af15-eb96a818e0e1_1072x395.png 1272w, https://substackcdn.com/image/fetch/$s_!SS0s!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2597bc4d-ee7f-40b9-af15-eb96a818e0e1_1072x395.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!SS0s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2597bc4d-ee7f-40b9-af15-eb96a818e0e1_1072x395.png" width="1072" height="395" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2597bc4d-ee7f-40b9-af15-eb96a818e0e1_1072x395.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:395,&quot;width&quot;:1072,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!SS0s!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2597bc4d-ee7f-40b9-af15-eb96a818e0e1_1072x395.png 424w, https://substackcdn.com/image/fetch/$s_!SS0s!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2597bc4d-ee7f-40b9-af15-eb96a818e0e1_1072x395.png 848w, https://substackcdn.com/image/fetch/$s_!SS0s!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2597bc4d-ee7f-40b9-af15-eb96a818e0e1_1072x395.png 1272w, https://substackcdn.com/image/fetch/$s_!SS0s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2597bc4d-ee7f-40b9-af15-eb96a818e0e1_1072x395.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">lsns output</figcaption></figure></div><p>At this point, most of these namespaces are system-wide, created by <code>init</code> when the machine is booted. We can see what type of namespace each one is - pid, network, mount, etc.</p><blockquote><p>&#128161; We have not created our pod yet, we will revisit this output and look for our namespace once we launch our test pod!</p></blockquote><h3>Let's take a look at cgroups</h3><p>We can see the current cgroups at <code>/sys/fs/cgroup</code></p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!0uM3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7de0fdd-dd48-4020-bc24-fac4c7052bf8_1507x166.png 424w, https://substackcdn.com/image/fetch/$s_!0uM3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7de0fdd-dd48-4020-bc24-fac4c7052bf8_1507x166.png 848w, https://substackcdn.com/image/fetch/$s_!0uM3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7de0fdd-dd48-4020-bc24-fac4c7052bf8_1507x166.png 1272w, https://substackcdn.com/image/fetch/$s_!0uM3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7de0fdd-dd48-4020-bc24-fac4c7052bf8_1507x166.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!0uM3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7de0fdd-dd48-4020-bc24-fac4c7052bf8_1507x166.png" width="1507" height="166" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b7de0fdd-dd48-4020-bc24-fac4c7052bf8_1507x166.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:166,&quot;width&quot;:1507,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!0uM3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7de0fdd-dd48-4020-bc24-fac4c7052bf8_1507x166.png 424w, https://substackcdn.com/image/fetch/$s_!0uM3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7de0fdd-dd48-4020-bc24-fac4c7052bf8_1507x166.png 848w, https://substackcdn.com/image/fetch/$s_!0uM3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7de0fdd-dd48-4020-bc24-fac4c7052bf8_1507x166.png 1272w, https://substackcdn.com/image/fetch/$s_!0uM3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7de0fdd-dd48-4020-bc24-fac4c7052bf8_1507x166.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>This won't make much sense right now, but I promise we will come back to it soon!</p><h3>Deploying a simple pod</h3><p>Now that we have seen what's running on the system, let's deploy a simple pod. We will use <code>Nginx</code> for this example. I will create the <code>pod.yaml</code></p><pre><code>apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:stable-alpine
    ports:
    - containerPort: 80</code></pre><p>And create it using <code>kubectl apply -f pod.yaml</code></p><p>And our pod is running under the <code>default</code> Kubernetes namespace</p><pre><code>root@playground:~# kubectl get pod
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          6s
root@playground:~#</code></pre><p>We can get some details about our new pod using <code>kubectl describe pod nginx</code></p><blockquote><p>&#128161; I have removed some lines that are not needed in this context</p></blockquote><pre><code>root@playground:~# kubectl describe pod nginx
Name:             nginx
Namespace:        default
Node:             playground/192.168.61.101
Status:           Running
IP:               10.42.0.9
Containers:
  nginx:
    Container ID:   containerd://1da4ef4758e3404c6be5d7222115c64d7007cec7c67b8c0fb84ba391bf347c38
    Image:          nginx:stable-alpine
    Port:           80/TCP
    Host Port:      0/TCP</code></pre><ul><li><p>We can see few important things such as the IP, Container ID etc</p></li></ul><h3>Looking at pstree</h3><p>Now that we have our pod running, let us run <code>pstree</code> to see all the processes in our Node where k3s is running</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!L6JT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b3200fb-e957-4368-a34d-5aecaccbdfc2_892x748.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!L6JT!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b3200fb-e957-4368-a34d-5aecaccbdfc2_892x748.png 424w, https://substackcdn.com/image/fetch/$s_!L6JT!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b3200fb-e957-4368-a34d-5aecaccbdfc2_892x748.png 848w, https://substackcdn.com/image/fetch/$s_!L6JT!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b3200fb-e957-4368-a34d-5aecaccbdfc2_892x748.png 1272w, https://substackcdn.com/image/fetch/$s_!L6JT!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b3200fb-e957-4368-a34d-5aecaccbdfc2_892x748.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!L6JT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b3200fb-e957-4368-a34d-5aecaccbdfc2_892x748.png" width="892" height="748" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9b3200fb-e957-4368-a34d-5aecaccbdfc2_892x748.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:748,&quot;width&quot;:892,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!L6JT!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b3200fb-e957-4368-a34d-5aecaccbdfc2_892x748.png 424w, https://substackcdn.com/image/fetch/$s_!L6JT!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b3200fb-e957-4368-a34d-5aecaccbdfc2_892x748.png 848w, https://substackcdn.com/image/fetch/$s_!L6JT!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b3200fb-e957-4368-a34d-5aecaccbdfc2_892x748.png 1272w, https://substackcdn.com/image/fetch/$s_!L6JT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b3200fb-e957-4368-a34d-5aecaccbdfc2_892x748.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">pstree output</figcaption></figure></div><p>You can already see the <code>nginx</code> process running under the <code>containerd-shim</code> process</p><ul><li><p><strong>containerd-shim</strong> is a small helper process that manages the lifecycle of a container.</p></li><li><p>Under each shim, you&#8217;ll see the actual application process &#8212; in this case, the <strong>nginx</strong> master and its worker processes.</p></li><li><p>You&#8217;ll also notice a <strong>pause</strong> process alongside nginx. This is a special container started by Kubernetes to manage the pod&#8217;s network namespace. Every pod has one pause container acting as the &#8220;parent&#8221; for networking.</p></li></ul><h3>Let's look at the running containers</h3><p>Since k3s uses containerd to run the containers, We can use <code>crictl</code> to look at the running containers (similar to <code>docker ps</code>)</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!bXAA!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48480563-0a43-4fa4-9d17-1535dcd9b116_1771x243.png 424w, https://substackcdn.com/image/fetch/$s_!bXAA!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48480563-0a43-4fa4-9d17-1535dcd9b116_1771x243.png 848w, https://substackcdn.com/image/fetch/$s_!bXAA!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48480563-0a43-4fa4-9d17-1535dcd9b116_1771x243.png 1272w, https://substackcdn.com/image/fetch/$s_!bXAA!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48480563-0a43-4fa4-9d17-1535dcd9b116_1771x243.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!bXAA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48480563-0a43-4fa4-9d17-1535dcd9b116_1771x243.png" width="1771" height="243" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/48480563-0a43-4fa4-9d17-1535dcd9b116_1771x243.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:243,&quot;width&quot;:1771,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!bXAA!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48480563-0a43-4fa4-9d17-1535dcd9b116_1771x243.png 424w, https://substackcdn.com/image/fetch/$s_!bXAA!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48480563-0a43-4fa4-9d17-1535dcd9b116_1771x243.png 848w, https://substackcdn.com/image/fetch/$s_!bXAA!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48480563-0a43-4fa4-9d17-1535dcd9b116_1771x243.png 1272w, https://substackcdn.com/image/fetch/$s_!bXAA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48480563-0a43-4fa4-9d17-1535dcd9b116_1771x243.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">crictl output</figcaption></figure></div><p>Here we can see a bunch of containers that comes with k3s, and we also see our <code>nginx</code> container at the top. Perfect! Nginx is running with container id <code>fdb8e461f5ce0</code></p><p>We can run <code>crictl inspect fdb8e461f5ce0</code> to see a very detailed view of the container. It is way too much information, so I am not going to show the full output</p><p>Let's find the linux process ID of our container</p><pre><code>root@playground:~# crictl inspect fdb8e461f5ce0 | grep pid
            "pid": 1
    "pid": 5896,
            "type": "pid"
root@playground:~#</code></pre><p>The process id is <code>5896</code>, which we can actually see directly as well</p><pre><code>root@playground:~# ps aux|grep 5896
root         253  0.0  0.2  26380  5896 ?        Ss   17:11   0:00 /lib/systemd/systemd-udevd
root        5896  0.0  0.2   9516  5472 ?        Ss   17:59   0:00 nginx: master process nginx -g daemon off;
root        6383  0.0  0.1   6332  2068 pts/1    S+   18:09   0:00 grep 5896
root@playground:~#</code></pre><h3>Inspecting Namespaces of Nginx container</h3><p>We can bring <code>lsns</code> back again, this time we pass the process ID and see all the namespaces for the Nginx process</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!7oe5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c3a3de-7841-4863-860e-413ffc59bd96_842x246.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!7oe5!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c3a3de-7841-4863-860e-413ffc59bd96_842x246.png 424w, https://substackcdn.com/image/fetch/$s_!7oe5!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c3a3de-7841-4863-860e-413ffc59bd96_842x246.png 848w, https://substackcdn.com/image/fetch/$s_!7oe5!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c3a3de-7841-4863-860e-413ffc59bd96_842x246.png 1272w, https://substackcdn.com/image/fetch/$s_!7oe5!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c3a3de-7841-4863-860e-413ffc59bd96_842x246.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!7oe5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c3a3de-7841-4863-860e-413ffc59bd96_842x246.png" width="842" height="246" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/62c3a3de-7841-4863-860e-413ffc59bd96_842x246.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:246,&quot;width&quot;:842,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!7oe5!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c3a3de-7841-4863-860e-413ffc59bd96_842x246.png 424w, https://substackcdn.com/image/fetch/$s_!7oe5!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c3a3de-7841-4863-860e-413ffc59bd96_842x246.png 848w, https://substackcdn.com/image/fetch/$s_!7oe5!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c3a3de-7841-4863-860e-413ffc59bd96_842x246.png 1272w, https://substackcdn.com/image/fetch/$s_!7oe5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c3a3de-7841-4863-860e-413ffc59bd96_842x246.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>There are some really interesting observations here</p><ul><li><p><strong>Network, UTS (host and domain name), IPC</strong> namespaces are created at the <strong>pod level</strong></p><ul><li><p>These namespaces are <strong>owned by the pause container</strong> &#8212; the small helper container that Kubernetes starts for every pod.</p></li><li><p>This is very important &#8211; It&#8217;s <strong>why multiple containers inside a single pod share the same network stack, hostname, and IPC space</strong>.</p></li></ul></li></ul><blockquote><p>&#128161; If you <code>exec</code> into a pod, you&#8217;ll find that <strong>all containers share the same process space, network interfaces, and domain names</strong>.<br>They look like they&#8217;re inside the same machine &#8212; because, namespace-wise, <strong>they actually are</strong>.</p></blockquote><h3>Inspecting the cgroups of the Nginx container</h3><p>So, as we talked about before, cgroups are how Linux manages resource usage like CPU, memory and I/O for groups of processes.</p><p>Since we know the PID of the Nginx master process, let us take a look at the cgroup assigments</p><pre><code>root@playground:~# cat /proc/5896/cgroup
0::/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6584cbdb_8529_4e28_8418_511e9056869e.slice/cri-containerd-fdb8e461f5ce08656bc92d114a76326a38b7f79c869c0ebf38e82e52d106254a.scope
root@playground:~#</code></pre><ul><li><p><code>/kubepods.slice/</code> - The Kubernetes pods are grouped under the <code>kubepods</code> slice</p></li><li><p><code>/kubepods-besteffort.slice/</code> &#8212; This pod belongs to the <strong>besteffort</strong> quality-of-service class.</p></li><li><p><code>/kubepods-besteffort-pod6584cbdb_8529_4e28_8418_511e9056869e.slice/</code> &#8212; This specific pod&#8217;s cgroup, identified by its <strong>Pod UID</strong> (underscores <code>_</code> replace dashes <code>-</code>).</p></li><li><p><code>/cri-containerd-&lt;container-id&gt;.scope</code> &#8212; The <strong>actual container process</strong> inside the pod, managed by containerd.</p></li></ul><p>We can even take a look at the current usage directly. For example, the memory usage</p><pre><code>root@playground:~# cat /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6584cbdb_8529_4e28_8418_511e9056869e.slice/cri-containerd-fdb8e461f5ce08656bc92d114a76326a38b7f79c869c0ebf38e82e52d106254a.scope/memory.current
3158016
root@playground:~#</code></pre><h3>Let's add some resource limits</h3><p>Now, let us add some CPU / memory requests and limits. Modify the pod yaml</p><pre><code>apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:stable-alpine
    resources:
      requests:
        memory: "64Mi"
        cpu: "50m"
      limits:
        memory: "128Mi"
        cpu: "100m"
    ports:
    - containerPort: 80</code></pre><blockquote><p>&#128161; We will have to delete and re-create the pod since we are changing the resources!</p></blockquote><p>Let's find the process ID again as we did before</p><pre><code>root@playground:~# crictl inspect c32fcc962d2bd | grep pid
            "pid": 1
    "pid": 7398,
            "type": "pid"
root@playground:~#</code></pre><h3>Checking cgroup after CPU and Memory resource update</h3><p>If take a look at the cgroup directory again</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!UXMc!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c81bf06-f076-411a-9753-f733457e9756_1541x323.png 424w, https://substackcdn.com/image/fetch/$s_!UXMc!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c81bf06-f076-411a-9753-f733457e9756_1541x323.png 848w, https://substackcdn.com/image/fetch/$s_!UXMc!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c81bf06-f076-411a-9753-f733457e9756_1541x323.png 1272w, https://substackcdn.com/image/fetch/$s_!UXMc!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c81bf06-f076-411a-9753-f733457e9756_1541x323.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!UXMc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c81bf06-f076-411a-9753-f733457e9756_1541x323.png" width="1541" height="323" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2c81bf06-f076-411a-9753-f733457e9756_1541x323.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:323,&quot;width&quot;:1541,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!UXMc!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c81bf06-f076-411a-9753-f733457e9756_1541x323.png 424w, https://substackcdn.com/image/fetch/$s_!UXMc!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c81bf06-f076-411a-9753-f733457e9756_1541x323.png 848w, https://substackcdn.com/image/fetch/$s_!UXMc!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c81bf06-f076-411a-9753-f733457e9756_1541x323.png 1272w, https://substackcdn.com/image/fetch/$s_!UXMc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c81bf06-f076-411a-9753-f733457e9756_1541x323.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>there are a ton of stuff in there. Let us take a look at the <code>cpu.max</code> and <code>cpu.stats</code></p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!__wj!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbbd4199-d945-46c7-898a-41dbeae5ef12_1541x364.png 424w, https://substackcdn.com/image/fetch/$s_!__wj!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbbd4199-d945-46c7-898a-41dbeae5ef12_1541x364.png 848w, https://substackcdn.com/image/fetch/$s_!__wj!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbbd4199-d945-46c7-898a-41dbeae5ef12_1541x364.png 1272w, https://substackcdn.com/image/fetch/$s_!__wj!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbbd4199-d945-46c7-898a-41dbeae5ef12_1541x364.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!__wj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbbd4199-d945-46c7-898a-41dbeae5ef12_1541x364.png" width="1541" height="364" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/dbbd4199-d945-46c7-898a-41dbeae5ef12_1541x364.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:364,&quot;width&quot;:1541,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!__wj!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbbd4199-d945-46c7-898a-41dbeae5ef12_1541x364.png 424w, https://substackcdn.com/image/fetch/$s_!__wj!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbbd4199-d945-46c7-898a-41dbeae5ef12_1541x364.png 848w, https://substackcdn.com/image/fetch/$s_!__wj!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbbd4199-d945-46c7-898a-41dbeae5ef12_1541x364.png 1272w, https://substackcdn.com/image/fetch/$s_!__wj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbbd4199-d945-46c7-898a-41dbeae5ef12_1541x364.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><ul><li><p><code>cpu.max</code> : The first number is the quota, second is the period</p><ul><li><p>it says 10,000 100,000</p></li><li><p><strong>Quota</strong> : 10,000 microseconds (10ms) : Nginx is allowed to run 10ms</p></li><li><p><strong>Period</strong> : 100,000 microseconds (100ms) : the total window for measuring CPU time.</p></li></ul></li></ul><blockquote><p>&#128161; So, this essentially means, <strong>Nginx is allowed to consume 10ms of CPU time every 100ms.</strong></p></blockquote><p>A full CPU core (1000m in Kubernetes terminology) represents 100% of CPU time.<br>Since we set a limit of <strong>100m</strong> for our Nginx pod, Kubernetes translates that into <strong>10% of a single CPU core</strong> &#8212; which matches exactly what we are seeing in the <code>cpu.max</code> file</p><ul><li><p><code>cpu.stats</code>: This is showing the current CPU stats for that container</p><ul><li><p>Most of the metrics are self explanatory here</p></li><li><p><strong>throttled_usec 243979 : </strong>Microseconds the container was CPU throttled (0.24 seconds)</p></li></ul></li></ul><h3>Bonus : Entering the namespace of Nginx container</h3><p>When we do <code>kubectl exec</code> into a pod, we are entering the namespace of that container (by default <code>kubectl exec</code> defaults into the first container &#8211; you can specify which container you want to <code>exec</code> into using <code>-c container_name</code>)</p><p>Similarly, we can actually enter the Linux namespace for the container and take a look around and see the things from the perspective of the container process itself</p><p>Let's find the process ID again and enter that namespace using <code>nsenter</code></p><pre><code>nsenter -t &lt;pid&gt; -a /bin/sh</code></pre><p>This should give us a shell inside the namespace. Let's take a look around</p><p><strong>Listing processes</strong></p><pre><code>/ # ps aux
PID   USER     TIME  COMMAND
    1 root      0:00 nginx: master process nginx -g daemon off;
   30 nginx     0:00 nginx: worker process
   31 nginx     0:00 nginx: worker process
   33 root      0:00 /bin/sh
   38 root      0:00 ps aux
/ #</code></pre><ul><li><p>The <strong>nginx master process</strong> is <strong>PID 1</strong> inside the container.</p></li><li><p>We only see a few processes: nginx, the shell we just entered, and the <code>ps aux</code> we just ran.</p></li><li><p>No host processes are visible &#8212; full <strong>process isolation</strong> thanks to the PID namespace!</p></li></ul><p><strong>ip and hostname</strong></p><pre><code>/ # ip a
1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if15: &lt;BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN&gt; mtu 1450 qdisc noqueue state UP qlen 1000
    link/ether 4e:df:af:b2:e1:ff brd ff:ff:ff:ff:ff:ff
    inet 10.42.0.11/24 brd 10.42.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::4cdf:afff:feb2:e1ff/64 scope link
       valid_lft forever preferred_lft forever
/ #
/ #
/ # hostname
nginx
/ #
</code></pre><p>Again, we see the isolated network namespace for the container with its own IP addresses and the hostname! How cool is that?</p><h2>Wrapping up!</h2><p>So, in this post, we took a bit of a deep dive into what actually happens when Kubernetes runs a pod. We&#8217;ve seen that at the end of the day, <strong>containers are just processes</strong> &#8212; with <strong>cgroups</strong> and <strong>namespaces</strong> applied for isolation and control.</p><p>Kubernetes might feel like magic sometimes, but it&#8217;s really built on <strong>solid Linux fundamentals</strong>. And understanding these fundamentals helps us <strong>scale </strong>and troubleshoot better and faster</p><p>Anyway, I had a ton of fun sneaking around in the terminal looking at all this stuff.<br>I hope this was at least a little bit helpful for you too! Thanks for reading. Leave a comment if you found this helpful</p>]]></content:encoded></item><item><title><![CDATA[Golang Performance Penalty in Kubernetes]]></title><description><![CDATA[Go apps in Kubernetes can suffer performance hits if GOMAXPROCS doesn&#8217;t match CPU limits. This post shows why it happens, how to fix it, and why it&#8217;s basically free performance. Benchmarks, dashboards, and a one-line solution included.]]></description><link>https://newsletter.esc.sh/p/golang-performance-penalty-in-kubernetes</link><guid isPermaLink="false">https://newsletter.esc.sh/p/golang-performance-penalty-in-kubernetes</guid><dc:creator><![CDATA[Mansoor]]></dc:creator><pubDate>Sat, 05 Apr 2025 00:01:27 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/3f6c7e41-c454-4ce2-a0dd-b06979bce966_2039x1134.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!oCu-!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4541311-2e98-4141-91d3-7e6d1a8e158d_2039x1134.png 424w, https://substackcdn.com/image/fetch/$s_!oCu-!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4541311-2e98-4141-91d3-7e6d1a8e158d_2039x1134.png 848w, https://substackcdn.com/image/fetch/$s_!oCu-!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4541311-2e98-4141-91d3-7e6d1a8e158d_2039x1134.png 1272w, https://substackcdn.com/image/fetch/$s_!oCu-!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4541311-2e98-4141-91d3-7e6d1a8e158d_2039x1134.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!oCu-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4541311-2e98-4141-91d3-7e6d1a8e158d_2039x1134.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b4541311-2e98-4141-91d3-7e6d1a8e158d_2039x1134.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!oCu-!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4541311-2e98-4141-91d3-7e6d1a8e158d_2039x1134.png 424w, https://substackcdn.com/image/fetch/$s_!oCu-!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4541311-2e98-4141-91d3-7e6d1a8e158d_2039x1134.png 848w, https://substackcdn.com/image/fetch/$s_!oCu-!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4541311-2e98-4141-91d3-7e6d1a8e158d_2039x1134.png 1272w, https://substackcdn.com/image/fetch/$s_!oCu-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4541311-2e98-4141-91d3-7e6d1a8e158d_2039x1134.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><h2>Motivation</h2><p>Recently I had to deploy a Golang application in Kubernetes, it was a very lightweight sidecar and I was surprised to see that it was performing <em>a little bit</em> slower than I anticipated. I am not going to talk in detail about this application and the nature of the traffic, but I was able to fix this performance issue by setting the environment variable <code>GOMAXPROCS</code> to match the Kubernetes deployment resource limit.</p><p>So, I am writing this blog post to explore the impact of <code>GOMAXPROCS</code> in Kubernetes and why this is a blindspot where quite a lot of performance goes to waste. We will be doing some benchmarks in a controlled environment to see if it really is that big of a deal.</p><h2>TL;DR</h2><p>In Kubernetes, <code>GOMAXPROCS</code> defaults to the number of CPU cores <strong>on the Node</strong>, not the Pod. If your Pod has a much lower CPU limit (e.g., 1 core on a 32-core node), your Go app will still try to run with <code>GOMAXPROCS=32</code>. This mismatch can hurt performance due to unnecessary context switching and CPU contention.<br><strong>Fix:</strong> Explicitly set <code>GOMAXPROCS</code> to match the Pod&#8217;s CPU limit.</p><p>There is an <a href="https://github.com/golang/go/issues/73193">issue</a> opened to make go runtime aware of CPU limits</p><h3>Update</h3><p>This has been fixed since Go 1.25 :) Let's Go Go Team!</p><blockquote><h4><a href="https://go.dev/blog/container-aware-gomaxprocs">Container-aware GOMAXPROCS - The Go Programming Language</a></h4></blockquote><h2>The Basics</h2><p>Before we jump into the benchmarks, it is important to understand how Go handles CPU concurrency and the effect of <code>GOMAXPROCS</code> in Kubernetes</p><h4>OS Threads and Go</h4><p>Go uses <code>goroutines</code> &#8211; lightweight, user threads. But, they don't run by themselves. They are scheduled onto the <strong>Operating System threads, </strong>which are managed by the Kernel and executed on actual CPU cores.</p><p>So, while we can spin up thousands of <code>goroutines</code>, they ultimately need to run on actual CPU cores (and threads), which are a lot less in number</p><h4>What is <code>GOMAXPROCS</code></h4><p>According to the official documentation</p><blockquote><p>The <code>GOMAXPROCS</code> variable limits the number of operating system threads that can execute user-level Go code simultaneously.</p></blockquote><blockquote><p>&#128161; By default <code>GOMAXPROCS</code> is set to the number of available CPU cores -- <strong>as seen by the Go runtime</strong></p></blockquote><h4>What is Context Switching</h4><h5>Let's consider the scenario</h5><ul><li><p>We have <strong>two applications</strong> running in Kubernetes on a single node</p></li><li><p>Each app has <strong>10 threads</strong></p></li><li><p>Only <strong>one physical core</strong> on the node</p></li></ul><h5>What we know</h5><ul><li><p>CPU can run only 1 thread at a time (Ignore SMT to keep things simpler)</p></li><li><p>But we have 20 total threads (10 from each app)</p></li><li><p>So the CPU <strong>must take turns</strong> running each thread</p></li></ul><h5>How the kernel makes it work</h5><ul><li><p>The Linux Kernel uses a scheduler to decide which thread to run next</p></li><li><p>It gives each thread a <strong>small time slice </strong>(few milliseconds)</p></li><li><p>After that time is up, the <strong>CPU switches to the next thread</strong></p><ul><li><p>This is called a <strong>context switch</strong></p></li></ul></li><li><p>The kernel will keep doing this over and over, making it look like threads are running at the same time</p></li><li><p>Now of course, if we have more cores, we can run more processes concurrently</p></li></ul><h3>Kubernetes CPU limits</h3><p>I strongly recommend my other post where I dive deep into Kubernetes internals, if you are interested!</p><blockquote><h4><a href="https://blog.esc.sh/kubernetes-containers-linux-processes/">How Kubernetes Runs Containers : A Practical Deep Dive</a></h4></blockquote><p>As you are probably aware of, Kubernetes lets us set resource <strong>requests</strong> and <strong>limits</strong> &#8211; for memory and CPU. In the context of CPU:</p><ul><li><p><strong>Request: </strong>the amount of CPU the container is guaranteed</p></li><li><p><strong>Limit: </strong>the maximum amount of CPU the container is allowed to use</p></li></ul><p>If we set a limit of <code>1</code>, Kubernetes will throttle the container to <strong>1 vCPUs worth of compute time</strong> even if the node has 32 available.</p><p>This is done using <strong>Linux cgroups</strong> (control groups), which allow the Kernel to constrain how much CPU time a process can use over time.</p><h2>Now, the problem</h2><p>Hope you have read the crash course on the OS basics ;) You probably have identified the issue already. Let's walk through it step by step</p><ul><li><p>You have a Kubernetes node with, say, <strong>32vCPU cores</strong></p></li><li><p>You have a pod running with <strong>CPU limit of 1</strong></p><ul><li><p>This pod is running a <strong>Go</strong> app with <strong>no extra configuration</strong></p><ul><li><p>The <strong>Go runtime</strong> in the pod sees that there are <strong>32 CPU cores</strong></p></li><li><p>So it sets the <code>GOMAXPROCS</code> to 32</p></li><li><p>And then, when needed, Go will spawn <strong>up to 32 threads</strong> even though the app will get to use only 1vCPU worth of time</p></li></ul></li></ul></li></ul><p>Now, let's say this Go app is a Web server receiving a high volume of requests. And say the average latency is <strong>20ms</strong></p><p>Here, the 32 threads spawned by go are competing for <strong>one core's worth of CPU time</strong></p><p><strong>Here is what happens:</strong></p><ul><li><p>Multiple requests arrive at once</p></li><li><p>Go spins up <code>goroutines</code> and schedules them across its 32 threads</p></li><li><p>If a pod is limited to 1 CPU, it can still run many threads in parallel across cores &#8212; but the <strong>total CPU time</strong> used across all threads can&#8217;t exceed <strong>1 core&#8217;s worth</strong> over a given time window</p></li></ul><p>Essentially, this leads to</p><ul><li><p>Increased request queuing</p></li><li><p>A lot more context switching (depending on how CPU intensive the application is &#8211; You will see this in action soon)</p></li><li><p>A lot more CPU throttling</p></li><li><p>And finally, the application latency will keep on increasing!</p></li></ul><p>Alright, enough theory, let's do some benchmarks!</p><h2>Let's do the Benchmark</h2><h3>The Hardware</h3><p>At first, I thought of running the benchmark in a Cloud Provider's Kubernetes cluster, but I decided against it because I have no control over the physical hardware and I did not want any <strong>noisy neighbors</strong> affecting the benchmark results. So I decided to keep things simple and run the tests on my own hardware. Maybe one day I will do another test in a Cloud Provider's cluster and compare the results.</p><p>For now, the benchmark runs on a machine from my homelab: a <strong>Lenovo ThinkCentre M720Q</strong>, powered by a <strong>6-core Intel i5-8400T</strong>.</p><p>As my grandma used to say:</p><blockquote><p>never waste an opportunity to flex a good <code>neofetch</code> screenshot</p></blockquote><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Zte4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fed48e554-223e-451c-b5c4-6fc6c2a39f4c_1689x862.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Zte4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fed48e554-223e-451c-b5c4-6fc6c2a39f4c_1689x862.png 424w, https://substackcdn.com/image/fetch/$s_!Zte4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fed48e554-223e-451c-b5c4-6fc6c2a39f4c_1689x862.png 848w, https://substackcdn.com/image/fetch/$s_!Zte4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fed48e554-223e-451c-b5c4-6fc6c2a39f4c_1689x862.png 1272w, https://substackcdn.com/image/fetch/$s_!Zte4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fed48e554-223e-451c-b5c4-6fc6c2a39f4c_1689x862.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Zte4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fed48e554-223e-451c-b5c4-6fc6c2a39f4c_1689x862.png" width="1689" height="862" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ed48e554-223e-451c-b5c4-6fc6c2a39f4c_1689x862.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:862,&quot;width&quot;:1689,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!Zte4!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fed48e554-223e-451c-b5c4-6fc6c2a39f4c_1689x862.png 424w, https://substackcdn.com/image/fetch/$s_!Zte4!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fed48e554-223e-451c-b5c4-6fc6c2a39f4c_1689x862.png 848w, https://substackcdn.com/image/fetch/$s_!Zte4!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fed48e554-223e-451c-b5c4-6fc6c2a39f4c_1689x862.png 1272w, https://substackcdn.com/image/fetch/$s_!Zte4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fed48e554-223e-451c-b5c4-6fc6c2a39f4c_1689x862.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">m720q neofetch</figcaption></figure></div><ul><li><p>Runs Proxmox on the host, and running a Debian Virtual Machine as the test machine on Proxmox</p><ul><li><p>The VM is assigned 6 CPU cores and 8GB of ram</p></li></ul></li><li><p>There are no other workload on this physical machine to skew the results</p></li><li><p>We will be running the benchmark tool from another physical machine on the same VLAN</p></li></ul><h3>The Software</h3><h4>Code</h4><p>I wrote (actually, ChatGPT did) a simple Go app for this test. You can find the source <a href="https://github.com/MansoorMajeed/k8s-bench/blob/669230b4e409d22872f5cad6670caba8ca2c301e/golang/GOMAXPROCS-test/app/main.go#L103-L108">HERE</a>. The code has a bunch of handlers, but we are only going to be using the <code>/cpu</code> one here. It takes an argument <code>n</code> which lets us make the request adjust the CPU usage pretty easily, which can be passed as a query string</p><blockquote><p>&#128161; Before you come at me with your pitchforks for building a "High Performance Fibonacci API", the test for <code>GOMAXPROCS</code> just need something that is CPU intensive. We are not comparing anything else, just the effect of wrongly configured <code>GOMAXPROCS</code> on cpu intensive applications</p></blockquote><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!fPQ8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10d1ba99-f41d-4fda-9644-ab0c2af899bb_760x266.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!fPQ8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10d1ba99-f41d-4fda-9644-ab0c2af899bb_760x266.png 424w, https://substackcdn.com/image/fetch/$s_!fPQ8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10d1ba99-f41d-4fda-9644-ab0c2af899bb_760x266.png 848w, https://substackcdn.com/image/fetch/$s_!fPQ8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10d1ba99-f41d-4fda-9644-ab0c2af899bb_760x266.png 1272w, https://substackcdn.com/image/fetch/$s_!fPQ8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10d1ba99-f41d-4fda-9644-ab0c2af899bb_760x266.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!fPQ8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10d1ba99-f41d-4fda-9644-ab0c2af899bb_760x266.png" width="760" height="266" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/10d1ba99-f41d-4fda-9644-ab0c2af899bb_760x266.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:266,&quot;width&quot;:760,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!fPQ8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10d1ba99-f41d-4fda-9644-ab0c2af899bb_760x266.png 424w, https://substackcdn.com/image/fetch/$s_!fPQ8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10d1ba99-f41d-4fda-9644-ab0c2af899bb_760x266.png 848w, https://substackcdn.com/image/fetch/$s_!fPQ8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10d1ba99-f41d-4fda-9644-ab0c2af899bb_760x266.png 1272w, https://substackcdn.com/image/fetch/$s_!fPQ8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10d1ba99-f41d-4fda-9644-ab0c2af899bb_760x266.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h4>Kubernetes</h4><p>I decided to run a single node k3s "cluster" for Kubernetes.</p><h4>Monitoring</h4><p>I am also using Prometheus (scrape time reduced to 5 seconds) with Grafana to show some sweet sweet context switching graphs. I have a blog post here to do that in Docker, if you are interested : <a href="https://selfhost.esc.sh/prometheus-grafana/">https://selfhost.esc.sh/prometheus-grafana/</a></p><h4>Benchmark tool</h4><p>We will use <a href="https://github.com/wg/wrk"><code>wrk</code></a> to run benchmarks. It will run from a different physical machine (not my laptop &#8211; been <em>burned</em> by thermal throttling before ) on the same VLAN</p><h2>The tests</h2><h3>The Baseline : Reasonable config - Not CPU bound</h3><p>We need to get a baseline in terms of performance when it is not CPU bound and with appropriately configured <code>GOMAXPROCS</code> and CPU limits</p><ul><li><p>CPU Limit : 1</p></li><li><p><code>GOMAXPROCS</code> = 1</p></li><li><p>Number of pods = 6</p></li></ul><p>This should make sense. There are <strong>6 cores</strong> assigned to this VM. So, we are running <strong>6 pods</strong> and telling each of the pod to run with <code>GOMAXPROCS=1</code> so the go runtime knows there is only one OS thread for it. Is this the best configuration? Who knows! we will see</p><p>Benchmark command</p><pre><code>wrk -t8 -c1000 -d180s 'http://192.168.61.100:7000/cpu?n=1'</code></pre><p>First, let us take a look at the <code>wrk</code> arguments</p><ul><li><p><code>-t8</code> - This is saying spawn 8 client threads</p></li><li><p><code>-c1000</code> - Make 1000 concurrent requests from each thread</p></li><li><p><code>-d180s</code> - Run the test for 3 minutes. (Why 3 minutes? Honestly, because 3-minute graphs looked the coolest in Grafana for these tests)</p></li></ul><p>Let's see what the result looks like</p><pre><code>~ &#10148; wrk -t8 -c1000 -d180s 'http://192.168.61.100:7000/cpu?n=1'
Running 3m test @ http://192.168.61.100:7000/cpu?n=1
  8 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    10.34ms    5.42ms 272.42ms   79.62%
    Req/Sec    12.19k     1.29k   22.29k    70.44%
  17462776 requests in 3.00m, 2.11GB read
Requests/sec:  96961.68
Transfer/sec:     12.02MB</code></pre><p>Okay, not bad!</p><h4>What are we seeing in wrk output</h4><ul><li><p>Latency:</p><ul><li><p>10ms avg</p></li><li><p>272ms max &#8211; a little spikey, but expected</p></li></ul></li><li><p>Requests/second:</p><ul><li><p>12k requests per second per thread</p></li><li><p>96k rps</p></li></ul></li></ul><h4>Let's take a look at Grafana</h4><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!rfeT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c79ded7-2c49-4225-90a5-4acbadccee50_1532x1139.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!rfeT!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c79ded7-2c49-4225-90a5-4acbadccee50_1532x1139.png 424w, https://substackcdn.com/image/fetch/$s_!rfeT!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c79ded7-2c49-4225-90a5-4acbadccee50_1532x1139.png 848w, https://substackcdn.com/image/fetch/$s_!rfeT!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c79ded7-2c49-4225-90a5-4acbadccee50_1532x1139.png 1272w, https://substackcdn.com/image/fetch/$s_!rfeT!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c79ded7-2c49-4225-90a5-4acbadccee50_1532x1139.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!rfeT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c79ded7-2c49-4225-90a5-4acbadccee50_1532x1139.png" width="1532" height="1139" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2c79ded7-2c49-4225-90a5-4acbadccee50_1532x1139.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1139,&quot;width&quot;:1532,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!rfeT!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c79ded7-2c49-4225-90a5-4acbadccee50_1532x1139.png 424w, https://substackcdn.com/image/fetch/$s_!rfeT!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c79ded7-2c49-4225-90a5-4acbadccee50_1532x1139.png 848w, https://substackcdn.com/image/fetch/$s_!rfeT!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c79ded7-2c49-4225-90a5-4acbadccee50_1532x1139.png 1272w, https://substackcdn.com/image/fetch/$s_!rfeT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c79ded7-2c49-4225-90a5-4acbadccee50_1532x1139.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">CPU Usage</figcaption></figure></div><ul><li><p>Nothing out of the ordinary, almost 100% CPU usage</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!kTUq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6ed531f-18ee-4019-8deb-dcb26a61528d_1532x1139.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!kTUq!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6ed531f-18ee-4019-8deb-dcb26a61528d_1532x1139.png 424w, https://substackcdn.com/image/fetch/$s_!kTUq!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6ed531f-18ee-4019-8deb-dcb26a61528d_1532x1139.png 848w, https://substackcdn.com/image/fetch/$s_!kTUq!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6ed531f-18ee-4019-8deb-dcb26a61528d_1532x1139.png 1272w, https://substackcdn.com/image/fetch/$s_!kTUq!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6ed531f-18ee-4019-8deb-dcb26a61528d_1532x1139.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!kTUq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6ed531f-18ee-4019-8deb-dcb26a61528d_1532x1139.png" width="1532" height="1139" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f6ed531f-18ee-4019-8deb-dcb26a61528d_1532x1139.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1139,&quot;width&quot;:1532,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!kTUq!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6ed531f-18ee-4019-8deb-dcb26a61528d_1532x1139.png 424w, https://substackcdn.com/image/fetch/$s_!kTUq!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6ed531f-18ee-4019-8deb-dcb26a61528d_1532x1139.png 848w, https://substackcdn.com/image/fetch/$s_!kTUq!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6ed531f-18ee-4019-8deb-dcb26a61528d_1532x1139.png 1272w, https://substackcdn.com/image/fetch/$s_!kTUq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6ed531f-18ee-4019-8deb-dcb26a61528d_1532x1139.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Process Schedule Stats - Running/Waiting</figcaption></figure></div><ul><li><p>baseline : Average around 130-15ms time waiting for the CPU</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!cIih!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F593ad572-177b-4fde-89c9-77c8b8ab6e8b_1532x1139.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!cIih!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F593ad572-177b-4fde-89c9-77c8b8ab6e8b_1532x1139.png 424w, https://substackcdn.com/image/fetch/$s_!cIih!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F593ad572-177b-4fde-89c9-77c8b8ab6e8b_1532x1139.png 848w, https://substackcdn.com/image/fetch/$s_!cIih!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F593ad572-177b-4fde-89c9-77c8b8ab6e8b_1532x1139.png 1272w, https://substackcdn.com/image/fetch/$s_!cIih!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F593ad572-177b-4fde-89c9-77c8b8ab6e8b_1532x1139.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!cIih!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F593ad572-177b-4fde-89c9-77c8b8ab6e8b_1532x1139.png" width="1532" height="1139" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/593ad572-177b-4fde-89c9-77c8b8ab6e8b_1532x1139.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1139,&quot;width&quot;:1532,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!cIih!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F593ad572-177b-4fde-89c9-77c8b8ab6e8b_1532x1139.png 424w, https://substackcdn.com/image/fetch/$s_!cIih!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F593ad572-177b-4fde-89c9-77c8b8ab6e8b_1532x1139.png 848w, https://substackcdn.com/image/fetch/$s_!cIih!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F593ad572-177b-4fde-89c9-77c8b8ab6e8b_1532x1139.png 1272w, https://substackcdn.com/image/fetch/$s_!cIih!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F593ad572-177b-4fde-89c9-77c8b8ab6e8b_1532x1139.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Context switches</figcaption></figure></div><ul><li><p>The Graph that I am most excited about. Around 7k average context switches</p></li></ul><h3>Benchmark 1 - GOMAXPROCS = 1</h3><p>For these tests, <strong>we will run 10 pods</strong>, to slightly mimick production where there will be often more pods than available CPU.</p><p><strong>Benchmark Command</strong></p><pre><code>wrk -t8 -c1000 -d180s 'http://192.168.61.100:7000/cpu?n=20'</code></pre><p>the only difference is the <strong>n=20 &#8211; This means a lot more CPU bound</strong></p><h4><code>wrk</code> output analysis</h4><pre><code>~ &#10148; wrk -t8 -c1000 -d180s 'http://192.168.61.100:7000/cpu?n=20'
Running 3m test @ http://192.168.61.100:7000/cpu?n=20
  8 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    20.00ms    8.61ms 255.50ms   70.68%
    Req/Sec     6.31k   543.85    13.54k    71.63%
  9043203 requests in 3.00m, 1.12GB read
Requests/sec:  50213.57
Transfer/sec:      6.37MB</code></pre><ul><li><p>Alright, as we expected, the performance has dropped. Still not too bad</p></li><li><p>Latency:</p><ul><li><p>20ms avg</p></li><li><p>255ms max</p></li></ul></li><li><p>Requests/second:</p><ul><li><p>6k requests per second per thread</p></li><li><p>50k rps</p></li></ul></li></ul><h3>Benchmark 2 - GOMAXPROCS = 32</h3><p>Same as before, we will run 10 pods. We will use the exact same command as well.</p><pre><code>~ &#10148; wrk -t8 -c1000 -d180s 'http://192.168.61.100:7000/cpu?n=20'
Running 3m test @ http://192.168.61.100:7000/cpu?n=20
  8 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    33.01ms   34.79ms 465.36ms   84.54%
    Req/Sec     5.07k   627.74     9.17k    70.07%
  7268074 requests in 3.00m, 0.90GB read
Requests/sec:  40356.13
Transfer/sec:      5.12MB</code></pre><p>I am not going to post the numbers again.</p><h3>Benchmark Results Comparison</h3><p>Metric GOMAXPROCS=1 GOMAXPROCS=32 % change Avg Latency 20ms 33ms +65% Max Latency 255ms 465ms +82% Overall RPS 50213 40356 -19.6%</p><h3>Grafana Metrics Comparison</h3><p>I think it is a lot more helpful to see it visualized</p><blockquote><p>&#128161; I will use <code>G</code> to denote <code>GOMAXPROCS</code> in the graphs</p></blockquote><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Sjwa!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd3d6f670-2d57-4993-b8ab-5e2566826882_1532x1139.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Sjwa!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd3d6f670-2d57-4993-b8ab-5e2566826882_1532x1139.png 424w, https://substackcdn.com/image/fetch/$s_!Sjwa!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd3d6f670-2d57-4993-b8ab-5e2566826882_1532x1139.png 848w, https://substackcdn.com/image/fetch/$s_!Sjwa!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd3d6f670-2d57-4993-b8ab-5e2566826882_1532x1139.png 1272w, https://substackcdn.com/image/fetch/$s_!Sjwa!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd3d6f670-2d57-4993-b8ab-5e2566826882_1532x1139.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Sjwa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd3d6f670-2d57-4993-b8ab-5e2566826882_1532x1139.png" width="1532" height="1139" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d3d6f670-2d57-4993-b8ab-5e2566826882_1532x1139.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1139,&quot;width&quot;:1532,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!Sjwa!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd3d6f670-2d57-4993-b8ab-5e2566826882_1532x1139.png 424w, https://substackcdn.com/image/fetch/$s_!Sjwa!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd3d6f670-2d57-4993-b8ab-5e2566826882_1532x1139.png 848w, https://substackcdn.com/image/fetch/$s_!Sjwa!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd3d6f670-2d57-4993-b8ab-5e2566826882_1532x1139.png 1272w, https://substackcdn.com/image/fetch/$s_!Sjwa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd3d6f670-2d57-4993-b8ab-5e2566826882_1532x1139.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">CPU Usage. Left: G=1, Right: G=32</figcaption></figure></div><ul><li><p>Nothing Interesting here. Both used almost full CPU</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!MhMX!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1d0213e-8a06-4561-8845-26f1d4e59a63_1532x1139.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!MhMX!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1d0213e-8a06-4561-8845-26f1d4e59a63_1532x1139.png 424w, https://substackcdn.com/image/fetch/$s_!MhMX!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1d0213e-8a06-4561-8845-26f1d4e59a63_1532x1139.png 848w, https://substackcdn.com/image/fetch/$s_!MhMX!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1d0213e-8a06-4561-8845-26f1d4e59a63_1532x1139.png 1272w, https://substackcdn.com/image/fetch/$s_!MhMX!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1d0213e-8a06-4561-8845-26f1d4e59a63_1532x1139.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!MhMX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1d0213e-8a06-4561-8845-26f1d4e59a63_1532x1139.png" width="1532" height="1139" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a1d0213e-8a06-4561-8845-26f1d4e59a63_1532x1139.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1139,&quot;width&quot;:1532,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!MhMX!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1d0213e-8a06-4561-8845-26f1d4e59a63_1532x1139.png 424w, https://substackcdn.com/image/fetch/$s_!MhMX!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1d0213e-8a06-4561-8845-26f1d4e59a63_1532x1139.png 848w, https://substackcdn.com/image/fetch/$s_!MhMX!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1d0213e-8a06-4561-8845-26f1d4e59a63_1532x1139.png 1272w, https://substackcdn.com/image/fetch/$s_!MhMX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1d0213e-8a06-4561-8845-26f1d4e59a63_1532x1139.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Process Schedule stats (Running/Waiting) : Left: G=1, Right G=32</figcaption></figure></div><ul><li><p>Things look a lot more interesting here.</p></li><li><p>Max time spent waiting for the CPU cores - around <strong>34 seconds</strong> when G=32 vs only <strong>~900ms</strong> when G=1</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!sF5t!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F958d2b8a-a3aa-4fb8-a105-6e537ddf9bac_1532x1139.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!sF5t!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F958d2b8a-a3aa-4fb8-a105-6e537ddf9bac_1532x1139.png 424w, https://substackcdn.com/image/fetch/$s_!sF5t!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F958d2b8a-a3aa-4fb8-a105-6e537ddf9bac_1532x1139.png 848w, https://substackcdn.com/image/fetch/$s_!sF5t!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F958d2b8a-a3aa-4fb8-a105-6e537ddf9bac_1532x1139.png 1272w, https://substackcdn.com/image/fetch/$s_!sF5t!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F958d2b8a-a3aa-4fb8-a105-6e537ddf9bac_1532x1139.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!sF5t!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F958d2b8a-a3aa-4fb8-a105-6e537ddf9bac_1532x1139.png" width="1532" height="1139" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/958d2b8a-a3aa-4fb8-a105-6e537ddf9bac_1532x1139.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1139,&quot;width&quot;:1532,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!sF5t!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F958d2b8a-a3aa-4fb8-a105-6e537ddf9bac_1532x1139.png 424w, https://substackcdn.com/image/fetch/$s_!sF5t!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F958d2b8a-a3aa-4fb8-a105-6e537ddf9bac_1532x1139.png 848w, https://substackcdn.com/image/fetch/$s_!sF5t!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F958d2b8a-a3aa-4fb8-a105-6e537ddf9bac_1532x1139.png 1272w, https://substackcdn.com/image/fetch/$s_!sF5t!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F958d2b8a-a3aa-4fb8-a105-6e537ddf9bac_1532x1139.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Context Switches: Left: G=1, Right: G=32</figcaption></figure></div><ul><li><p>As we expected, there is a huge increase in the number of context switches with GOMAXPROCS=32 compared to it being 1</p><ul><li><p><strong>Around 6.5k vs 30k context switches</strong></p></li></ul></li></ul><h2>The Fix</h2><p>The benchmarks clearly shows wasted performance due to misconfigured <code>GOMAXPROCS</code>.</p><p>Luckily, the <strong>fix is very simple</strong>, just <strong>set <code>GOMAXPROCS</code> same as the CPU limit </strong>applied to the deployment. In Kubernetes, this can be achieved through setting the environment variable <code>GOMAXPROCS</code></p><p>You can set it like this</p><pre><code>env:
  - name: GOMAXPROCS
    valueFrom:
      resourceFieldRef:
        resource: limits.cpu
        divisor: "1"</code></pre><p>For example, our test app full deployment yaml would look like this</p><pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-bench
  labels:
    app: k8s-bench
spec:
  replicas: 10
  selector:
    matchLabels:
      app: k8s-bench
  template:
    metadata:
      labels:
        app: k8s-bench
    spec:
      containers:
      - name: k8s-bench
        image: mansoor1/golang-bench:0.2
        env:
        - name: GOMAXPROCS
          valueFrom:
            resourceFieldRef:
              resource: limits.cpu
              divisor: "1"
        ports:
        - containerPort: 7000
        resources:
          requests:
            cpu: "100m" # Let us have more pods than CPU
          limits:
            cpu: "2"</code></pre><p>And if we do a describe, we will see that it has automatically set the value <code>GOMAXPROCS</code> to match the CPU limit here, which is <code>2</code></p><pre><code>&gt; kubectl describe pod k8s-bench-8db849fdf-tdz4x | grep -A2 Environment:
    Environment:
      GOMAXPROCS:  2 (limits.cpu)
    Mounts:</code></pre><h3>Bonus Fix</h3><p>A kind redditor pointed me to this <a href="https://github.com/uber-go/automaxprocs">cool library by Uber </a>, which I think is a good way to completely avoid this issue</p><h2>Conclusion</h2><p>It was a lot of fun doing this experiment and seeing the Grafana dashboard clearly showing how context switching silently eats away performance when <code>GOMAXPROCS</code> is misconfigured. The benchmarks are definitely a worst case scenario, but it costs nothing more than 5 lines to fix. So why not take advantage of it.</p><p>So if you&#8217;re running Go apps in Kubernetes, take a moment to check your <code>GOMAXPROCS</code>. You might be leaving performance on the table without even knowing it.</p><p>Curious to hear your thoughts on this &#8211; should I do more of these sorts of tests? Let me know in the comments!</p><p>And thanks for reading :)</p>]]></content:encoded></item><item><title><![CDATA[Exposing a selfhosted service using Cloudflare tunnel]]></title><description><![CDATA[Cloudflare tunnel is one of the easiest and safest option to expose a self hosted service to the internet. In this guide, we will walk through the full setup and the caveats of using Cloudflare tunnel]]></description><link>https://newsletter.esc.sh/p/cloudflare-tunnel</link><guid isPermaLink="false">https://newsletter.esc.sh/p/cloudflare-tunnel</guid><dc:creator><![CDATA[Mansoor]]></dc:creator><pubDate>Thu, 02 Jan 2025 12:37:52 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/549a531e-e87b-438d-af3c-9fa884f4071b_1908x1162.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!d0RA!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbb4eb69-37fc-4c90-9971-20db3fe28c27_1908x1162.png 424w, https://substackcdn.com/image/fetch/$s_!d0RA!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbb4eb69-37fc-4c90-9971-20db3fe28c27_1908x1162.png 848w, https://substackcdn.com/image/fetch/$s_!d0RA!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbb4eb69-37fc-4c90-9971-20db3fe28c27_1908x1162.png 1272w, https://substackcdn.com/image/fetch/$s_!d0RA!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbb4eb69-37fc-4c90-9971-20db3fe28c27_1908x1162.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!d0RA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbb4eb69-37fc-4c90-9971-20db3fe28c27_1908x1162.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/dbb4eb69-37fc-4c90-9971-20db3fe28c27_1908x1162.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!d0RA!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbb4eb69-37fc-4c90-9971-20db3fe28c27_1908x1162.png 424w, https://substackcdn.com/image/fetch/$s_!d0RA!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbb4eb69-37fc-4c90-9971-20db3fe28c27_1908x1162.png 848w, https://substackcdn.com/image/fetch/$s_!d0RA!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbb4eb69-37fc-4c90-9971-20db3fe28c27_1908x1162.png 1272w, https://substackcdn.com/image/fetch/$s_!d0RA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbb4eb69-37fc-4c90-9971-20db3fe28c27_1908x1162.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><p>If you have a service or website that you want to expose to the Internet, Cloudflare tunnel is one of the easiest, secure (with some major things to consider &#8211; more on that later) and free solution.</p><h2>Some context</h2><p>If you don't care about the context and just want to get into the action, feel free to skip this section, but I would still suggest you read the <strong>Security </strong>section</p><h3>How does it work</h3><p>You can read more about it in Cloudflare docs <a href="https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/">HERE</a></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!rMZf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f1ae13c-b4df-4696-8a87-575ffa73b911_1768x1102.webp" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!rMZf!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f1ae13c-b4df-4696-8a87-575ffa73b911_1768x1102.webp 424w, https://substackcdn.com/image/fetch/$s_!rMZf!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f1ae13c-b4df-4696-8a87-575ffa73b911_1768x1102.webp 848w, https://substackcdn.com/image/fetch/$s_!rMZf!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f1ae13c-b4df-4696-8a87-575ffa73b911_1768x1102.webp 1272w, https://substackcdn.com/image/fetch/$s_!rMZf!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f1ae13c-b4df-4696-8a87-575ffa73b911_1768x1102.webp 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!rMZf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f1ae13c-b4df-4696-8a87-575ffa73b911_1768x1102.webp" width="1768" height="1102" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9f1ae13c-b4df-4696-8a87-575ffa73b911_1768x1102.webp&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1102,&quot;width&quot;:1768,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!rMZf!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f1ae13c-b4df-4696-8a87-575ffa73b911_1768x1102.webp 424w, https://substackcdn.com/image/fetch/$s_!rMZf!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f1ae13c-b4df-4696-8a87-575ffa73b911_1768x1102.webp 848w, https://substackcdn.com/image/fetch/$s_!rMZf!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f1ae13c-b4df-4696-8a87-575ffa73b911_1768x1102.webp 1272w, https://substackcdn.com/image/fetch/$s_!rMZf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f1ae13c-b4df-4696-8a87-575ffa73b911_1768x1102.webp 1456w" sizes="100vw"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Cloudflare tunnel (source: cloudflare docs)</figcaption></figure></div><h4>Give me TL;DR</h4><ul><li><p>You install a simple <code>daemon</code> called <code>cloudflared</code> in your infrastructure. In my case, I have a <strong>Debian Virtual Machine</strong> running on an old Mini PC.</p></li><li><p>This daemon makes an <strong>outbound only connection</strong> to Cloudflare's network &#8211; meaning you do not need to setup any port forwarding or firewall rules in your router.</p></li><li><p>Now, Cloudflare can route traffic to your service through this tunnel</p></li></ul><h3>How secure is Cloudflare tunnel?</h3><p>So let us clarify what it protects you against and what it does not.</p><h4>Cloudflare can see all your traffic</h4><p>This should not come as a surprise. If you use Cloudflare, you are terminating your SSL/TLS connection in Cloudflare. This is no different than using Cloudflare CDN normally. So, if you are particularly concerned about Cloudflare looking into your traffic, you should not be using Cloudflare at all.</p><p>I personally only use Cloudflare CDN and Cloudflare Tunnel with publicly available services such as this website, my analytics etc. For all my private homelab services, I use Wireguard directly to my home network and is not exposed to the internet.</p><h4>Cloudflare will give you protection from bad traffic</h4><p>DDOS protection, CDN, basic web application firewall (WAF) etc. all come with the free plan too. This means Cloudflare will protect you against <em>some</em> attacks.</p><h4>Cloudflare cannot protect you against all vulnerabilities / exploits</h4><p>Let us assume that you have a WordPress blog you selfhost and is exposed using Cloudflare tunnel. If someone finds a vulnerability in that blog and exploits Cloudflare may not be able to protect you. Sure, if there it is some well known exploit, Cloudflare WAF may protect you, but remember that it is your responsibility to keep your service secure and updated.</p><h4>You should still isolate your internet facing host</h4><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!S3Vo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a498b99-1c3d-4c2b-b9a8-87fb4f97cc30_1118x482.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!S3Vo!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a498b99-1c3d-4c2b-b9a8-87fb4f97cc30_1118x482.png 424w, https://substackcdn.com/image/fetch/$s_!S3Vo!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a498b99-1c3d-4c2b-b9a8-87fb4f97cc30_1118x482.png 848w, https://substackcdn.com/image/fetch/$s_!S3Vo!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a498b99-1c3d-4c2b-b9a8-87fb4f97cc30_1118x482.png 1272w, https://substackcdn.com/image/fetch/$s_!S3Vo!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a498b99-1c3d-4c2b-b9a8-87fb4f97cc30_1118x482.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!S3Vo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a498b99-1c3d-4c2b-b9a8-87fb4f97cc30_1118x482.png" width="1118" height="482" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2a498b99-1c3d-4c2b-b9a8-87fb4f97cc30_1118x482.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:482,&quot;width&quot;:1118,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!S3Vo!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a498b99-1c3d-4c2b-b9a8-87fb4f97cc30_1118x482.png 424w, https://substackcdn.com/image/fetch/$s_!S3Vo!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a498b99-1c3d-4c2b-b9a8-87fb4f97cc30_1118x482.png 848w, https://substackcdn.com/image/fetch/$s_!S3Vo!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a498b99-1c3d-4c2b-b9a8-87fb4f97cc30_1118x482.png 1272w, https://substackcdn.com/image/fetch/$s_!S3Vo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a498b99-1c3d-4c2b-b9a8-87fb4f97cc30_1118x482.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Just as you would with any form of <em>exposing to the internet</em>, you should isolate your host where internet facing service and <code>cloudflared</code> is running. This is because, in the event the service (say, WordPress in this diagram) gets exploited and the hacker got into your network (<code>192.168.1.10</code>), if it has network access (by default it will), you should know that the hacker would be able to find and potentially exploit other services on your network.</p><p><strong>What can you do about it?</strong></p><p>Put the internet host in its own VLAN, if your router and switch supports it. And then create firewall rules to allow only internet access. This is called a DMZ (demilitarized zone)</p><h3>What can I host via Cloudflare tunnel</h3><p>When it comes to hosting via Cloudflare tunnel, only practical concern is routing services such as Plex. Anything else (as long as it is legal) should be fine.</p><blockquote><p><em>Use of the Services for serving video or a disproportionate percentage of pictures, audio files, or other non-HTML content is prohibited</em></p></blockquote><p>I personally do not use Cloudflare for exposing Plex, but you maybe able to get away with it, proceed at your own risk.</p><h2>Setting up Cloudflare Tunnel</h2><p>Alright, enough talking, let's get into it.</p><h3>Run your service</h3><p>This can be a physical or a virtual machine. In this example, I will be using a Debian virtual machine</p><p>I have already <a href="https://blog.esc.sh/debian-quick-docker-installation/">installed Docker</a> on this VM and now I can start my demo service by using</p><pre><code>mansoor@playground:~$ docker run -d -p 8080:8080 mansoor1/go-hello-world</code></pre><p>And I can see that it works locally, from the same machine</p><pre><code>mansoor@playground:~$ curl localhost:8080
Hello from: ea2a3e654bfb
Current Time: Thu, 02 Jan 2025 11:47:55 UTC</code></pre><p>Now, we want this service to be accessible from the internet over Cloudflare tunnel.</p><h3>Add your domain to Cloudflare</h3><p>You need a domain to use Cloudflare tunnel. If you do not have one, I strongly recommend that you buy one.</p><blockquote><p>&#128161; I recommend that you use Cloudflare itself as the domain registrar ( I have most of my domains registered with Cloudflare)</p></blockquote><p>If you already have a domain but have not added to Cloudflare, follow Cloudflare <a href="https://developers.cloudflare.com/fundamentals/setup/manage-domains/add-site/">docs </a>and add your domain to Cloudflare.</p><h3>Create the tunnel from Cloudflare dashboard</h3><p>Login to your Cloudflare account</p><p><strong>Account Home -&gt; Zero Trust</strong></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!DdnK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70845e05-b9f5-48bd-a349-c763926a4b7d_348x651.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!DdnK!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70845e05-b9f5-48bd-a349-c763926a4b7d_348x651.png 424w, https://substackcdn.com/image/fetch/$s_!DdnK!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70845e05-b9f5-48bd-a349-c763926a4b7d_348x651.png 848w, https://substackcdn.com/image/fetch/$s_!DdnK!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70845e05-b9f5-48bd-a349-c763926a4b7d_348x651.png 1272w, https://substackcdn.com/image/fetch/$s_!DdnK!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70845e05-b9f5-48bd-a349-c763926a4b7d_348x651.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!DdnK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70845e05-b9f5-48bd-a349-c763926a4b7d_348x651.png" width="348" height="651" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/70845e05-b9f5-48bd-a349-c763926a4b7d_348x651.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:651,&quot;width&quot;:348,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!DdnK!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70845e05-b9f5-48bd-a349-c763926a4b7d_348x651.png 424w, https://substackcdn.com/image/fetch/$s_!DdnK!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70845e05-b9f5-48bd-a349-c763926a4b7d_348x651.png 848w, https://substackcdn.com/image/fetch/$s_!DdnK!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70845e05-b9f5-48bd-a349-c763926a4b7d_348x651.png 1272w, https://substackcdn.com/image/fetch/$s_!DdnK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70845e05-b9f5-48bd-a349-c763926a4b7d_348x651.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><strong>Networks -&gt; Tunnels</strong></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!A0WI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd177c87-cdbe-4c83-87d9-0392e3727966_626x546.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!A0WI!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd177c87-cdbe-4c83-87d9-0392e3727966_626x546.png 424w, https://substackcdn.com/image/fetch/$s_!A0WI!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd177c87-cdbe-4c83-87d9-0392e3727966_626x546.png 848w, https://substackcdn.com/image/fetch/$s_!A0WI!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd177c87-cdbe-4c83-87d9-0392e3727966_626x546.png 1272w, https://substackcdn.com/image/fetch/$s_!A0WI!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd177c87-cdbe-4c83-87d9-0392e3727966_626x546.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!A0WI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd177c87-cdbe-4c83-87d9-0392e3727966_626x546.png" width="626" height="546" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bd177c87-cdbe-4c83-87d9-0392e3727966_626x546.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:546,&quot;width&quot;:626,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!A0WI!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd177c87-cdbe-4c83-87d9-0392e3727966_626x546.png 424w, https://substackcdn.com/image/fetch/$s_!A0WI!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd177c87-cdbe-4c83-87d9-0392e3727966_626x546.png 848w, https://substackcdn.com/image/fetch/$s_!A0WI!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd177c87-cdbe-4c83-87d9-0392e3727966_626x546.png 1272w, https://substackcdn.com/image/fetch/$s_!A0WI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd177c87-cdbe-4c83-87d9-0392e3727966_626x546.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>From there, click on <strong>Create tunnel</strong>, and choose <strong>Cloudflared</strong></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!bp21!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5457513-e96b-498b-aa53-7c8e5e989010_1186x623.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!bp21!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5457513-e96b-498b-aa53-7c8e5e989010_1186x623.png 424w, https://substackcdn.com/image/fetch/$s_!bp21!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5457513-e96b-498b-aa53-7c8e5e989010_1186x623.png 848w, https://substackcdn.com/image/fetch/$s_!bp21!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5457513-e96b-498b-aa53-7c8e5e989010_1186x623.png 1272w, https://substackcdn.com/image/fetch/$s_!bp21!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5457513-e96b-498b-aa53-7c8e5e989010_1186x623.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!bp21!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5457513-e96b-498b-aa53-7c8e5e989010_1186x623.png" width="1186" height="623" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c5457513-e96b-498b-aa53-7c8e5e989010_1186x623.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:623,&quot;width&quot;:1186,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!bp21!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5457513-e96b-498b-aa53-7c8e5e989010_1186x623.png 424w, https://substackcdn.com/image/fetch/$s_!bp21!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5457513-e96b-498b-aa53-7c8e5e989010_1186x623.png 848w, https://substackcdn.com/image/fetch/$s_!bp21!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5457513-e96b-498b-aa53-7c8e5e989010_1186x623.png 1272w, https://substackcdn.com/image/fetch/$s_!bp21!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5457513-e96b-498b-aa53-7c8e5e989010_1186x623.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Choose any name you would like. I will name it <code>playground-demo</code> and <strong>save tunnel</strong></p><p>At this point, Cloudflare will show you an option to choose your operating system. I chose <strong>Debian 64bit</strong></p><blockquote><p>&#128161; If you use a raspberrypi, you should choose <code>arm64-bit</code></p></blockquote><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!aCHU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F909a906e-ae7f-49ad-a08c-905b0977c2a6_835x812.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!aCHU!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F909a906e-ae7f-49ad-a08c-905b0977c2a6_835x812.png 424w, https://substackcdn.com/image/fetch/$s_!aCHU!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F909a906e-ae7f-49ad-a08c-905b0977c2a6_835x812.png 848w, https://substackcdn.com/image/fetch/$s_!aCHU!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F909a906e-ae7f-49ad-a08c-905b0977c2a6_835x812.png 1272w, https://substackcdn.com/image/fetch/$s_!aCHU!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F909a906e-ae7f-49ad-a08c-905b0977c2a6_835x812.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!aCHU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F909a906e-ae7f-49ad-a08c-905b0977c2a6_835x812.png" width="835" height="812" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/909a906e-ae7f-49ad-a08c-905b0977c2a6_835x812.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:812,&quot;width&quot;:835,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!aCHU!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F909a906e-ae7f-49ad-a08c-905b0977c2a6_835x812.png 424w, https://substackcdn.com/image/fetch/$s_!aCHU!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F909a906e-ae7f-49ad-a08c-905b0977c2a6_835x812.png 848w, https://substackcdn.com/image/fetch/$s_!aCHU!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F909a906e-ae7f-49ad-a08c-905b0977c2a6_835x812.png 1272w, https://substackcdn.com/image/fetch/$s_!aCHU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F909a906e-ae7f-49ad-a08c-905b0977c2a6_835x812.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><blockquote><p>&#128161; Be very careful with the command. This includes a token that authenticates the daemon to Cloudflare. Keep it secure.</p></blockquote><h3>Install and start cloudflared</h3><p>Copy the whole command and paste it into your host where you want to run your service.</p><pre><code>mansoor@playground:~$ curl -L --output cloudflared.deb https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb &amp;&amp;

sudo dpkg -i cloudflared.deb &amp;&amp;

sudo cloudflared service install eyJhIjoXXXXXXXXXXXXX-redacted-XXXXXXXX
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 17.6M  100 17.6M    0     0  13.3M      0  0:00:01  0:00:01 --:--:-- 16.3M
[sudo] password for mansoor:
Selecting previously unselected package cloudflared.
(Reading database ... 37359 files and directories currently installed.)
Preparing to unpack cloudflared.deb ...
Unpacking cloudflared (2024.12.2) ...
Setting up cloudflared (2024.12.2) ...
Processing triggers for man-db (2.11.2-2) ...
2025-01-02T12:04:26Z INF Using Systemd
2025-01-02T12:04:27Z INF Linux service for cloudflared installed successfully
mansoor@playground:~$</code></pre><p>We can see that the <code>cloudflared</code> service has successfully started. You can check the status of the <code>daemon</code> using <code>sudo systemctl status cloudflared</code></p><p>Now, back in Cloudflare dashboard, click <strong>Next</strong></p><h3>Configure the Cloudflare tunnel</h3><p>At this point, you should see a page like this</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!RSLo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a34af4-4d63-49cc-b764-0b5b9044ebd4_1293x683.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!RSLo!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a34af4-4d63-49cc-b764-0b5b9044ebd4_1293x683.png 424w, https://substackcdn.com/image/fetch/$s_!RSLo!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a34af4-4d63-49cc-b764-0b5b9044ebd4_1293x683.png 848w, https://substackcdn.com/image/fetch/$s_!RSLo!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a34af4-4d63-49cc-b764-0b5b9044ebd4_1293x683.png 1272w, https://substackcdn.com/image/fetch/$s_!RSLo!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a34af4-4d63-49cc-b764-0b5b9044ebd4_1293x683.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!RSLo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a34af4-4d63-49cc-b764-0b5b9044ebd4_1293x683.png" width="1293" height="683" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/59a34af4-4d63-49cc-b764-0b5b9044ebd4_1293x683.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:683,&quot;width&quot;:1293,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!RSLo!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a34af4-4d63-49cc-b764-0b5b9044ebd4_1293x683.png 424w, https://substackcdn.com/image/fetch/$s_!RSLo!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a34af4-4d63-49cc-b764-0b5b9044ebd4_1293x683.png 848w, https://substackcdn.com/image/fetch/$s_!RSLo!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a34af4-4d63-49cc-b764-0b5b9044ebd4_1293x683.png 1272w, https://substackcdn.com/image/fetch/$s_!RSLo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a34af4-4d63-49cc-b764-0b5b9044ebd4_1293x683.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>This is where we tell Cloudflare how to route a domain or subdomain to the service running on our local machine.</p><blockquote><p>&#128161; Remember that we need the domain to be added to Cloudflare. If you have successfully added your domain, you should be able to see it in the dropdown under <code>Domain</code></p></blockquote><p>I want my demo service to be accessible over the internet at the URL <code>tunnel-demo.esc.sh</code>.</p><blockquote><p>&#128161; <strong>Note about subdomains</strong><br>If you want a second level subdomain such as <code>one.two.example.com</code>, you need to purchase Cloudflare's Advanced certificates. With the free plan, you can only use the root domain and single level subdomains. So, <code>one.example.com</code> and <code>example.com</code> will work fine, NOT <code>one.two.example.com</code> with the free Cloudflare</p></blockquote><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!-Z0X!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75a828d1-0f22-44eb-95dd-75605ee19c69_1072x320.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!-Z0X!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75a828d1-0f22-44eb-95dd-75605ee19c69_1072x320.png 424w, https://substackcdn.com/image/fetch/$s_!-Z0X!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75a828d1-0f22-44eb-95dd-75605ee19c69_1072x320.png 848w, https://substackcdn.com/image/fetch/$s_!-Z0X!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75a828d1-0f22-44eb-95dd-75605ee19c69_1072x320.png 1272w, https://substackcdn.com/image/fetch/$s_!-Z0X!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75a828d1-0f22-44eb-95dd-75605ee19c69_1072x320.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!-Z0X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75a828d1-0f22-44eb-95dd-75605ee19c69_1072x320.png" width="1072" height="320" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/75a828d1-0f22-44eb-95dd-75605ee19c69_1072x320.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:320,&quot;width&quot;:1072,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!-Z0X!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75a828d1-0f22-44eb-95dd-75605ee19c69_1072x320.png 424w, https://substackcdn.com/image/fetch/$s_!-Z0X!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75a828d1-0f22-44eb-95dd-75605ee19c69_1072x320.png 848w, https://substackcdn.com/image/fetch/$s_!-Z0X!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75a828d1-0f22-44eb-95dd-75605ee19c69_1072x320.png 1272w, https://substackcdn.com/image/fetch/$s_!-Z0X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75a828d1-0f22-44eb-95dd-75605ee19c69_1072x320.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><ol><li><p><strong>Subdomain: </strong>If you want to host your service at the root of your domain, you can leave this empty. Make sure to read the <strong>Note about subdomains </strong>above.</p></li><li><p><strong>Domain: </strong>This is the domain you own and already configured in Cloudflare</p></li><li><p><strong>Path: </strong>If you want your service to be accessible only on a path, you can add that here. For example, if I wanted this <code>Hello World</code> service to be exposed only on <code>esc.sh/hello-world</code>, I could put <code>hello-world</code> as the <strong>Path </strong>and cloudflare will route only that path to the tunnel</p></li><li><p><strong>Type: </strong>If you click on the dropdown, you should see a bunch of options. <strong>For most of the self hosted web services, it will be <code>HTTP</code>.</strong></p><ol><li><p>This is the protocol with which the <code>cloudflared</code> daemon will connect to your locally running service. There is no reason to use <code>HTTPS</code> here since Cloudflare will anyway terminate SSL/TLS at their edge.</p></li></ol></li><li><p><strong>URL: </strong>This will be the URL where your local service is available. <strong>Most of the time, it will be <code>localhost</code> with a port number</strong>. In this case, I have my docker container running on <code>localhost:8080</code></p><ol><li><p>This can be an actual domain other than <code>localhost</code>, if and when you use a reverse proxy as an entry point for <code>cloudflared</code>. If you don't know what that means, you should be fine with simply using <code>localhost:&lt;port number&gt;</code></p></li></ol></li></ol><p>Finally, <strong>save tunnel</strong></p><h3>Testing and verifying</h3><p>At this point, Cloudflare will automatically create a DNS record for <code>tunnel-demo.esc.sh</code> to point to this tunnel. Here is mine</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!tj2B!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61289cf8-4ad1-4f1c-b284-fcc7aba3c576_1169x42.png 424w, https://substackcdn.com/image/fetch/$s_!tj2B!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61289cf8-4ad1-4f1c-b284-fcc7aba3c576_1169x42.png 848w, https://substackcdn.com/image/fetch/$s_!tj2B!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61289cf8-4ad1-4f1c-b284-fcc7aba3c576_1169x42.png 1272w, https://substackcdn.com/image/fetch/$s_!tj2B!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61289cf8-4ad1-4f1c-b284-fcc7aba3c576_1169x42.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!tj2B!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61289cf8-4ad1-4f1c-b284-fcc7aba3c576_1169x42.png" width="1169" height="42" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/61289cf8-4ad1-4f1c-b284-fcc7aba3c576_1169x42.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:42,&quot;width&quot;:1169,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!tj2B!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61289cf8-4ad1-4f1c-b284-fcc7aba3c576_1169x42.png 424w, https://substackcdn.com/image/fetch/$s_!tj2B!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61289cf8-4ad1-4f1c-b284-fcc7aba3c576_1169x42.png 848w, https://substackcdn.com/image/fetch/$s_!tj2B!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61289cf8-4ad1-4f1c-b284-fcc7aba3c576_1169x42.png 1272w, https://substackcdn.com/image/fetch/$s_!tj2B!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61289cf8-4ad1-4f1c-b284-fcc7aba3c576_1169x42.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>And now we should be able to access <a href="https://tunnel-demo.esc.sh/"><code>https://tunnel-demo.esc.sh/</code></a></p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!pFlR!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6dd9c11e-a2de-4205-83b8-591155236df7_380x111.png 424w, https://substackcdn.com/image/fetch/$s_!pFlR!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6dd9c11e-a2de-4205-83b8-591155236df7_380x111.png 848w, https://substackcdn.com/image/fetch/$s_!pFlR!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6dd9c11e-a2de-4205-83b8-591155236df7_380x111.png 1272w, https://substackcdn.com/image/fetch/$s_!pFlR!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6dd9c11e-a2de-4205-83b8-591155236df7_380x111.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!pFlR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6dd9c11e-a2de-4205-83b8-591155236df7_380x111.png" width="380" height="111" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6dd9c11e-a2de-4205-83b8-591155236df7_380x111.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:111,&quot;width&quot;:380,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!pFlR!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6dd9c11e-a2de-4205-83b8-591155236df7_380x111.png 424w, https://substackcdn.com/image/fetch/$s_!pFlR!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6dd9c11e-a2de-4205-83b8-591155236df7_380x111.png 848w, https://substackcdn.com/image/fetch/$s_!pFlR!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6dd9c11e-a2de-4205-83b8-591155236df7_380x111.png 1272w, https://substackcdn.com/image/fetch/$s_!pFlR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6dd9c11e-a2de-4205-83b8-591155236df7_380x111.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div>]]></content:encoded></item><item><title><![CDATA[Plex 4K Streaming and Cross-Continent Playback: Making It Work]]></title><description><![CDATA[Background]]></description><link>https://newsletter.esc.sh/p/plex-cross-continent-4k-streaming</link><guid isPermaLink="false">https://newsletter.esc.sh/p/plex-cross-continent-4k-streaming</guid><dc:creator><![CDATA[Mansoor]]></dc:creator><pubDate>Fri, 30 Aug 2024 00:00:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/755d0e23-6bf3-4efe-9124-af8ef703c432_1792x1024.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!jZIL!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdb4c3d6-a6a6-4f09-a8e0-f08d8ebd7b82_1792x1024.webp 424w, https://substackcdn.com/image/fetch/$s_!jZIL!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdb4c3d6-a6a6-4f09-a8e0-f08d8ebd7b82_1792x1024.webp 848w, https://substackcdn.com/image/fetch/$s_!jZIL!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdb4c3d6-a6a6-4f09-a8e0-f08d8ebd7b82_1792x1024.webp 1272w, https://substackcdn.com/image/fetch/$s_!jZIL!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdb4c3d6-a6a6-4f09-a8e0-f08d8ebd7b82_1792x1024.webp 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!jZIL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdb4c3d6-a6a6-4f09-a8e0-f08d8ebd7b82_1792x1024.webp" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fdb4c3d6-a6a6-4f09-a8e0-f08d8ebd7b82_1792x1024.webp&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!jZIL!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdb4c3d6-a6a6-4f09-a8e0-f08d8ebd7b82_1792x1024.webp 424w, https://substackcdn.com/image/fetch/$s_!jZIL!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdb4c3d6-a6a6-4f09-a8e0-f08d8ebd7b82_1792x1024.webp 848w, https://substackcdn.com/image/fetch/$s_!jZIL!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdb4c3d6-a6a6-4f09-a8e0-f08d8ebd7b82_1792x1024.webp 1272w, https://substackcdn.com/image/fetch/$s_!jZIL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffdb4c3d6-a6a6-4f09-a8e0-f08d8ebd7b82_1792x1024.webp 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div><figcaption class="image-caption">DallE being Dramatic</figcaption></figure></div><h2>Background</h2><p>I have a Plex server, I want to share that with my friends and family</p><h2><strong>1st attempt : Plex + Simple Reverse Proxy</strong></h2><p>This is the simplest solution. I have the Plex server running on an old Mini PC in my home network. I have a Digital Ocean VPS acting as a reverse proxy so that I don't have to expose my home IP directly to the internet.</p><p>If you are interested in reading more about it, I have a post explaining how I expose my self-hosted services to the internet. You can read it here</p><blockquote><h4><a href="https://blog.esc.sh/expose-selfhosted-services-to-internet/">How I expose my Selfhosted services to the Internet</a></h4></blockquote><p>This method worked great for people who live close to me.</p><h2>The Problem</h2><p>I currently live in North America, while most of my friends and family are on the other side of<br>the planet, in Asia. Streaming 4k media over such a long distance is challenging</p><p>Of course, you can't beat physics, packets can't travel faster than the speed of light. However, latency isn't the main issue for streaming; network stability is. One of the biggest factors contributing to an unstable network is the poor routes that packets take when traveling such a long distance. Not all routes are created equal&#8212;some are more congested and prone to network problems, including packet loss.</p><h2>Potential Solutions</h2><p>Well, just use a CDN, right? Yes, sort of. Let's talk about it</p><h3>What about Cloudflare free tier?</h3><p>Cloudflare's free tier seems like an obvious choice, but it can't be used for video streaming. Doing so violates their Terms of Service, and you risk getting your account banned.</p><h3>What about a paid CDN?</h3><p>I am okay with paying a reasonable amount a month for this selfhosting hobby. Take a look at Fastly pricing for just 1TB a month</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!DC_u!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41dd3156-7ac9-4dc3-9067-dcdcd3a85c6e_705x512.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!DC_u!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41dd3156-7ac9-4dc3-9067-dcdcd3a85c6e_705x512.png 424w, https://substackcdn.com/image/fetch/$s_!DC_u!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41dd3156-7ac9-4dc3-9067-dcdcd3a85c6e_705x512.png 848w, https://substackcdn.com/image/fetch/$s_!DC_u!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41dd3156-7ac9-4dc3-9067-dcdcd3a85c6e_705x512.png 1272w, https://substackcdn.com/image/fetch/$s_!DC_u!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41dd3156-7ac9-4dc3-9067-dcdcd3a85c6e_705x512.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!DC_u!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41dd3156-7ac9-4dc3-9067-dcdcd3a85c6e_705x512.png" width="705" height="512" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/41dd3156-7ac9-4dc3-9067-dcdcd3a85c6e_705x512.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:512,&quot;width&quot;:705,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!DC_u!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41dd3156-7ac9-4dc3-9067-dcdcd3a85c6e_705x512.png 424w, https://substackcdn.com/image/fetch/$s_!DC_u!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41dd3156-7ac9-4dc3-9067-dcdcd3a85c6e_705x512.png 848w, https://substackcdn.com/image/fetch/$s_!DC_u!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41dd3156-7ac9-4dc3-9067-dcdcd3a85c6e_705x512.png 1272w, https://substackcdn.com/image/fetch/$s_!DC_u!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41dd3156-7ac9-4dc3-9067-dcdcd3a85c6e_705x512.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><blockquote><p>&#128161; I am not paying $150 a month for this.</p></blockquote><h2>My Solution : Build a poor man's CDN</h2><p>The reason streaming over such long distances can be problematic is that the routes often taken by the packets can be pretty bad. The traffic goes through several ISPs, and some of these routes can be quite congested. A potential solution is to rely on a cloud provider that can help mitigate these routing issues by setting up two virtual machines&#8212;one in North America and one in Asia.</p><h3>What makes a traffic between two Cloud VMs better</h3><p>When we route between Cloud VMs, the packets between the VM in Asia and the VM in the US takes a much better route which reduces the chances of congestion, offers better latency and offers a stable streaming experience. This is because cloud providers usually have direct peering agreements and connections with major backbone providers. As a result, even if the packets are not going through their own "private network", they are still routed through more reliable connections compared to if we were to route directly between two home ISPs</p><h3>What size Cloud VM do I need?</h3><p>Honestly, if you will be doing only a few streams simultaneously, you can go with the smallest possible size. The limit that matters is how much bandwidth it comes with</p><h3>Alright, which Cloud Provider?</h3><p>I have personally used Google Cloud, Digital Ocean and Linode.</p><h4>Google Cloud</h4><p><strong>Pros</strong></p><p>Google Cloud offers global VPC (virtual private cloud), which means we can create a VM in Asia and one in US but both in the same private network within Google Cloud</p><p>Example:</p><ul><li><p>10.0.1.2 : asia-vm</p></li><li><p>10.0.1.3 : us-vm</p></li></ul><p>Now we can connect the asia-vm directly to 10.0.1.3, which ensures that all the traffic remains within Google's internal network which offers the best case scenario for our streams compared to a public route.</p><p><strong>Cons</strong></p><p>Google Cloud is great if you don't plan on streaming a lot. They offer free network egress for up to 200GB per month. However, if you want to go above that, you will start spending some good money. Around <code>$0.12</code> per GB. That comes around to $120/TB. Again, that is too expensive for just the network egress for a homelab</p><h4>Digital Ocean and Linode</h4><p>I put both of them under the same umbrella because they feel more or less the same.</p><ul><li><p>Digital Ocean offers 1TB network transfer per month for $6/month</p></li><li><p>Linode offers 1TB transfer per month for $5/month</p></li></ul><p>I initially used Digital Ocean and now trying Linode. I have no data to say one is better than the other.</p><h2><strong>How will it work?</strong></h2><p>It is very straight forward. Two VMs in the two geographic regions running Nginx. Using Route53 for DNS to have latency based routing</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!sXba!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3472139e-7a8a-4b92-a102-170152f20f20_910x785.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!sXba!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3472139e-7a8a-4b92-a102-170152f20f20_910x785.png 424w, https://substackcdn.com/image/fetch/$s_!sXba!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3472139e-7a8a-4b92-a102-170152f20f20_910x785.png 848w, https://substackcdn.com/image/fetch/$s_!sXba!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3472139e-7a8a-4b92-a102-170152f20f20_910x785.png 1272w, https://substackcdn.com/image/fetch/$s_!sXba!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3472139e-7a8a-4b92-a102-170152f20f20_910x785.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!sXba!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3472139e-7a8a-4b92-a102-170152f20f20_910x785.png" width="910" height="785" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3472139e-7a8a-4b92-a102-170152f20f20_910x785.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:785,&quot;width&quot;:910,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!sXba!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3472139e-7a8a-4b92-a102-170152f20f20_910x785.png 424w, https://substackcdn.com/image/fetch/$s_!sXba!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3472139e-7a8a-4b92-a102-170152f20f20_910x785.png 848w, https://substackcdn.com/image/fetch/$s_!sXba!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3472139e-7a8a-4b92-a102-170152f20f20_910x785.png 1272w, https://substackcdn.com/image/fetch/$s_!sXba!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3472139e-7a8a-4b92-a102-170152f20f20_910x785.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Route53 for latency based routing</strong></h3><blockquote><p>&#128161; <strong>Note</strong>: For this to work, your domain needs to be managed by Route53 for DNS. However, if you already use another provider, you don't really have to move the entire DNS setup for that domain to Route53. Instead, you can simply delegate a subdomain to Route53. Read this Cloudflare documentation for an example <a href="https://developers.cloudflare.com/dns/manage-dns-records/how-to/subdomains-outside-cloudflare">HERE</a></p></blockquote><blockquote><p>&#128161; For example, <em><strong>demo.esc.sh </strong></em>can be delegated to Route53 and use Route53 to manage all subdomains below that while continuing to use the existing DNS provider for all the existing DNS records. You can choose whatever is more convenient to you.</p></blockquote><p>My plex domain is <code>plex.example.com</code>. This domain's DNS is handled by AWS Route53. Through the magic of latency based routing in AWS Route53, the domain <code>plex.example.com</code> will resolve to the IP that is closer to the client that is making the DNS request. You can read more about it in AWS documentation <a href="https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-latency.html">HERE</a></p><p><strong>For example:</strong></p><ul><li><p><code>plex.example.com</code> is the domain</p></li><li><p><code>104.21.6.130</code> is the IP of the VM in Asia</p></li><li><p><code>172.67.154.225</code> is the IP of the VM in the US</p></li></ul><p>Then we create two A records in Route53, and choose the routing policy of "latency based" and use these IPs as the values.</p><p>Now, when a client from Asia is making a DNS query for <code>plex.example.com</code>, it will resolve to <code>104.21.6.130</code></p><blockquote><p>&#128161; <strong>But why do we need to use this complicated DNS setup?</strong><br>In Plex, you list the server domains in an order. So you will end up with <code>plex-asia.example.com</code>, <code>plex-us.example.com</code>. And when a client tries to play a stream, Plex will use the <strong>first URL that is reachable</strong>. So, if we put the asia domain first, <strong>everyone will be routed to the asia VM</strong>. So, you see the problem</p></blockquote><h4>But is Route53 expensive?</h4><p>No, it is $0.50/month per zone and then $0.60 per million queries. In my billing, it is barely above $0.5 per month</p><h2>Let's do this</h2><p>We got a few things to do here. Please try to read carefully.</p><h3>Plex setup</h3><p>I assume that you already have a Plex server configured and exposed to the internet. I will assume that this Plex server is available at <code>plex-origin.example.com</code>, which points to your home IP address.</p><p>If you have a different setup, like I have explained in <a href="https://blog.esc.sh/expose-selfhosted-services-to-internet/">How I expose my Selfhosted services to the Internet</a> the idea remains the same. We will be proxying from an Nginx to this "Plex Origin" which connects to the plex instance in your home network.</p><h3><strong>Route53 Setup</strong></h3><p>We should create a dedicated IAM user in AWS to use with Let's Encrypt to automate certificates. We will use this in a step below.</p><h4><strong>Create the IAM policy to allow editing the zone</strong></h4><p>At first, we need to create a Route53 IAM policy that allows our user to edit the Zone. This is needed by the certbot, which we will use to retrieve TLS certificates.<br><br>Login to AWS. Go to IAM -&gt; Policies -&gt; Create policy -&gt; JSON</p><p>Replace the whole json with the following</p><pre><code>{
    "Version": "2012-10-17",
    "Id": "certbot-dns-route53 sample policy",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "route53:ListHostedZones",
                "route53:GetChange"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect" : "Allow",
            "Action" : [
                "route53:ChangeResourceRecordSets"
            ],
            "Resource" : [
                "arn:aws:route53:::hostedzone/&lt;put your zone ID here&gt;"
            ]
        }
    ]
}</code></pre><p>Make sure to replace your zone ID. You can find this under AWS -&gt; Route53 -&gt; Hosted Zones -&gt; Your zone you will use for plex -&gt; Hosted Zone ID<br><br>Click Next -&gt; Give a name (Example: certbot-edit-zone) -&gt; Create</p><h4>Create the IAM user for certbot</h4><p>Go to IAM -&gt; Users -&gt; Create user</p><ul><li><p>Give a suitable username. Example : "certbot-user"</p></li><li><p>No need to give access to AWS management console</p></li><li><p>Choose <code>Attach policies directly</code></p></li><li><p>Select the policy we just created -&gt; Next -&gt; Create user</p></li></ul><h4>Generate the Access Key and Secret</h4><p>Click on the user we just created -&gt; Security credentials -&gt; Access keys -&gt; Create access key</p><p>Choose the CLI option -&gt; Next -&gt; Create access key -&gt; Download the .csv file. Keep it very safe. You can't see this again in AWS console</p><h4>Create the latency based DNS records in Route53</h4><p>Now, I assume you have added your domain or subdomain to Route53. If not, you can follow <a href="https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/migrate-dns-domain-in-use.html">THIS</a> documentation to do that.</p><p>Now, go to Route53 -&gt; Hosted Zones -&gt; Your domain -&gt; Create record.</p><p>Create a record like shown in the screenshot</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!HHnS!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98f18c99-63df-4aab-98d2-988dca2ab958_1114x963.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!HHnS!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98f18c99-63df-4aab-98d2-988dca2ab958_1114x963.png 424w, https://substackcdn.com/image/fetch/$s_!HHnS!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98f18c99-63df-4aab-98d2-988dca2ab958_1114x963.png 848w, https://substackcdn.com/image/fetch/$s_!HHnS!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98f18c99-63df-4aab-98d2-988dca2ab958_1114x963.png 1272w, https://substackcdn.com/image/fetch/$s_!HHnS!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98f18c99-63df-4aab-98d2-988dca2ab958_1114x963.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!HHnS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98f18c99-63df-4aab-98d2-988dca2ab958_1114x963.png" width="1114" height="963" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/98f18c99-63df-4aab-98d2-988dca2ab958_1114x963.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:963,&quot;width&quot;:1114,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!HHnS!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98f18c99-63df-4aab-98d2-988dca2ab958_1114x963.png 424w, https://substackcdn.com/image/fetch/$s_!HHnS!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98f18c99-63df-4aab-98d2-988dca2ab958_1114x963.png 848w, https://substackcdn.com/image/fetch/$s_!HHnS!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98f18c99-63df-4aab-98d2-988dca2ab958_1114x963.png 1272w, https://substackcdn.com/image/fetch/$s_!HHnS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98f18c99-63df-4aab-98d2-988dca2ab958_1114x963.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><ul><li><p>As you can see, the Record Name : plex.demo.esc.sh. Replace this with your domain name</p></li><li><p>Value : Should be the IP of the Asia VM (or whatever region you want to use - region 1)</p></li><li><p>Region : Choose a region that is as close to the VM as possible. It does not matter much as long as the other record is much further away</p></li><li><p>Record ID : Choose something that would make sense to you</p></li></ul><p>Create the record.</p><p><br>Now, <strong>repeat the same</strong> for the US VM IP address.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!mN1p!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20b5344d-f703-4ecb-9700-21e276e60899_1137x966.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mN1p!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20b5344d-f703-4ecb-9700-21e276e60899_1137x966.png 424w, https://substackcdn.com/image/fetch/$s_!mN1p!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20b5344d-f703-4ecb-9700-21e276e60899_1137x966.png 848w, https://substackcdn.com/image/fetch/$s_!mN1p!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20b5344d-f703-4ecb-9700-21e276e60899_1137x966.png 1272w, https://substackcdn.com/image/fetch/$s_!mN1p!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20b5344d-f703-4ecb-9700-21e276e60899_1137x966.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!mN1p!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20b5344d-f703-4ecb-9700-21e276e60899_1137x966.png" width="1137" height="966" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/20b5344d-f703-4ecb-9700-21e276e60899_1137x966.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:966,&quot;width&quot;:1137,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!mN1p!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20b5344d-f703-4ecb-9700-21e276e60899_1137x966.png 424w, https://substackcdn.com/image/fetch/$s_!mN1p!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20b5344d-f703-4ecb-9700-21e276e60899_1137x966.png 848w, https://substackcdn.com/image/fetch/$s_!mN1p!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20b5344d-f703-4ecb-9700-21e276e60899_1137x966.png 1272w, https://substackcdn.com/image/fetch/$s_!mN1p!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20b5344d-f703-4ecb-9700-21e276e60899_1137x966.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>At this point, we have two A records for <code>plex.example.com</code> pointing to two different IP addresses in two geographic location</p><h3>Cloud VM configuration</h3><p>Now let us configure our reverse proxy VMs (aka poor man's CDN POPs).</p><blockquote><p>Login to both the VMs via SSH.</p></blockquote><h4>Domains we will use</h4><p>This is an overview of the domain names involved</p><blockquote><p>Note: Make sure to create the remaining DNS records. These are <code>Simple routing</code> records</p></blockquote><ul><li><p><code>plex.example.com</code> : This is our main domain for plex. We have two A records pointing to two VMs</p></li><li><p><code>plex-us.example.com</code> : Static DNS record pointing to the VM in the US</p></li><li><p><code>plex-asia.example.com</code> : Static DNS record pointing to the VM in Asia</p></li><li><p><code>plex-origin.example.com</code> : The IP of wherever your Plex server is. This is usually your dynamic DNS if hosting on your home network</p></li></ul><h4>Configuring TLS certificates</h4><p>We will use Let'sEncrypt to encrypt all traffic. Since you use Route53, it is very easy to get Letsencrypt certificates using DNS challenge.</p><p>First, install the required packages on both the VMs.</p><pre><code>sudo apt update
sudo apt install certbot python3-certbot-dns-route53 nginx
</code></pre><p>Now, under the root user, create a file <code>/root/.aws/credentials</code>. Populate it with the credentials from the csv file we downloaded in the previous step.</p><p>It should look something like this.</p><pre><code>root@server-us:~# cat /root/.aws/credentials
[default]
aws_access_key_id = AKIAxxxxxxxxxxxx
aws_secret_access_key = 43ryNAxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
root@server-us:~#
</code></pre><blockquote><p>Note: Remember, anyone who has access to this credential have full access to modify any records on your hosted zone. So keep this safe.</p></blockquote><p>Make sure to have the same in the asia VM too.</p><p><strong>Next, let us retrieve the certificates</strong></p><p>With the credentials in place, we are ready to get our TLS certificates.</p><p><strong>On both the VMs</strong><br>We need to get the <code>plex.example.com</code> certificate on both the VMs</p><pre><code>sudo certbot certonly --dns-route53 -d plex.example.com
</code></pre><p>Follow the prompts. If you have correctly configured the Route53 credentials, it should work</p><p><strong>On the US VM</strong></p><p>The VM in asia needs to connect to the VM in US via TLS. So, we also need the <code>plex-us.example.com</code> domain.</p><pre><code>sudo certbot certonly --dns-route53 -d plex-us.example.com
</code></pre><blockquote><p>Note: We don't need the plex-asia.example.com certs because no one will be connecting to that domain</p></blockquote><h4><strong>Configuring Nginx - US VM</strong></h4><p>I adopted some of the configurations from <a href="https://github.com/toomuchio/plex-nginx-reverseproxy">HERE</a> so if you face any issues with Nginx config specifically, check that repo for any potential solutions</p><p>Create the configuration at <code>/etc/nginx/sites-enabled/plex.example.com</code>.</p><p>Make sure to replace all instance of <code>plex.example.com</code> with your own domain.</p><pre><code>server {
    server_name plex.example.com;
    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/plex.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/plex.example.com/privkey.pem;


    # Plex has a lot of text script which is easily compressed.
    # If these settings cause playback issues with devices, remove them. (Haven't encountered any yet)
    gzip on;
    gzip_vary on;
    gzip_min_length 1000;
    gzip_proxied any;
    gzip_types text/plain text/css text/xml application/xml text/javascript application/x-javascript image/svg+xml;
    gzip_disable "MSIE [1-6]\.";

    # nginx default client_max_body_size is 1MB, which breaks Camera Upload feature from phones.
    # Increasing the limit fixes the issue.
    # Note if you are sending VERY LARGE files (e.g. 4k videos) you will need to increase this much further.
    client_max_body_size 100M;


    proxy_set_header X-Real-IP $remote_addr;
    # When using ngx_http_realip_module change $proxy_add_x_forwarded_for to '$http_x_forwarded_for,$realip_remote_addr'
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Sec-WebSocket-Extensions $http_sec_websocket_extensions;
    proxy_set_header Sec-WebSocket-Key $http_sec_websocket_key;
    proxy_set_header Sec-WebSocket-Version $http_sec_websocket_version;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Accept-Encoding ""; # Disables compression between Plex and Nginx

    # Disable buffering - send to the client as soon as the data is received from Plex.
    proxy_redirect off;
    proxy_buffering off;

    location / {
        proxy_pass https://plex-origin.example.com;
    }
}
server {
    if ($host = plex.example.com) {
        return 301 https://$host$request_uri;
    } 

    listen 80 ;
    server_name plex.example.com;
    return 404;
}</code></pre><blockquote><p>&#128161; <strong>READ BEFORE PROCEEDING</strong></p></blockquote><ul><li><p>Ensure <code>plex.example.com</code> is replaced with your correct domain name that use within Plex.</p></li><li><p>Ensure <code>plex-origin.example.com</code> is replaced with the correct domain for your home IP, or wherever plex is hosted. Note that I am using <code>https</code> here, so you need to ensure that your home IP has TLS as well. If you have a wireguard/tunnel from your home network to this VM, then you can replace the <code>https://plex-origin.example.com;</code> with your address like <code>http://10.1.0.3:32400</code> for example. You get the idea</p></li></ul><p><br>Now <strong>on the same US VM</strong> we need to configure the second server block which the Asia VM will connect to<br><br>Create <code>/etc/nginx/sites-enabled/plex-us.example.com</code></p><pre><code>server {
    server_name plex-us.example.com;
    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/plex-us.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/plex-us.example.com/privkey.pem;


    # Plex has a lot of text script which is easily compressed.
    # If these settings cause playback issues with devices, remove them. (Haven't encountered any yet)
    gzip on;
    gzip_vary on;
    gzip_min_length 1000;
    gzip_proxied any;
    gzip_types text/plain text/css text/xml application/xml text/javascript application/x-javascript image/svg+xml;
    gzip_disable "MSIE [1-6]\.";

    # nginx default client_max_body_size is 1MB, which breaks Camera Upload feature from phones.
    # Increasing the limit fixes the issue.
    # Note if you are sending VERY LARGE files (e.g. 4k videos) you will need to increase this much further.
    client_max_body_size 100M;


    proxy_set_header X-Real-IP $remote_addr;
    # When using ngx_http_realip_module change $proxy_add_x_forwarded_for to '$http_x_forwarded_for,$realip_remote_addr'
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Sec-WebSocket-Extensions $http_sec_websocket_extensions;
    proxy_set_header Sec-WebSocket-Key $http_sec_websocket_key;
    proxy_set_header Sec-WebSocket-Version $http_sec_websocket_version;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Accept-Encoding ""; # Disables compression between Plex and Nginx

    # Disable buffering - send to the client as soon as the data is received from Plex.
    proxy_redirect off;
    proxy_buffering off;

    location / {
        proxy_pass https://plex-origin.example.com;
    }

}</code></pre><h4><strong>Configuring Nginx - Asia VM</strong></h4><p>Create <code>/etc/nginx/sites-enabled/plex.example.com</code> with the following content</p><blockquote><p>&#128161; Note: This is the same config as the US config, <strong>except one thing</strong>. The <code>proxy_pass</code> should be to the <code>plex-us.example.com</code><br></p></blockquote><pre><code>server {
    server_name plex.example.com;
    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/plex.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/plex.example.com/privkey.pem;


    # Plex has a lot of text script which is easily compressed.
    # If these settings cause playback issues with devices, remove them. (Haven't encountered any yet)
    gzip on;
    gzip_vary on;
    gzip_min_length 1000;
    gzip_proxied any;
    gzip_types text/plain text/css text/xml application/xml text/javascript application/x-javascript image/svg+xml;
    gzip_disable "MSIE [1-6]\.";

    # nginx default client_max_body_size is 1MB, which breaks Camera Upload feature from phones.
    # Increasing the limit fixes the issue.
    # Note if you are sending VERY LARGE files (e.g. 4k videos) you will need to increase this much further.
    client_max_body_size 100M;


    proxy_set_header X-Real-IP $remote_addr;
    # When using ngx_http_realip_module change $proxy_add_x_forwarded_for to '$http_x_forwarded_for,$realip_remote_addr'
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Sec-WebSocket-Extensions $http_sec_websocket_extensions;
    proxy_set_header Sec-WebSocket-Key $http_sec_websocket_key;
    proxy_set_header Sec-WebSocket-Version $http_sec_websocket_version;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Accept-Encoding ""; # Disables compression between Plex and Nginx

    # Disable buffering - send to the client as soon as the data is received from Plex.
    proxy_redirect off;
    proxy_buffering off;

    location / {
        proxy_pass https://plex-us.example.com;
    }
}
server {
    if ($host = plex.example.com) {
        return 301 https://$host$request_uri;
    } 

    listen 80 ;
    server_name plex.example.com;
    return 404;
}</code></pre><h4>Test Nginx on both VM</h4><p>Run this command to verify that Nginx syntax is correct</p><pre><code>sudo nginx -t
</code></pre><p>If you see any errors, fix them. Make sure the domain names and certificate path are correct -- this is the most common mistake</p><h4>Start Nginx</h4><p>If the config is fine, run</p><pre><code>sudo systemctl restart nginx
</code></pre><h3>Configure Plex for the domain</h3><p>Now open your plex server settings -&gt; Choose your correct server -&gt; Settings -&gt; Network -&gt; Custom server access URLs<br>Put the domain we just configured in this post. That is, in this example, <code>plex.example.com</code></p><p>That should be it. If you have everything configured correctly, you should be able to visit <code>plex.example.com</code> and see your Plex login page.</p>]]></content:encoded></item><item><title><![CDATA[How I expose my Selfhosted services to the Internet]]></title><description><![CDATA[The Flow Chart]]></description><link>https://newsletter.esc.sh/p/expose-selfhosted-services-to-internet</link><guid isPermaLink="false">https://newsletter.esc.sh/p/expose-selfhosted-services-to-internet</guid><dc:creator><![CDATA[Mansoor]]></dc:creator><pubDate>Tue, 02 Jan 2024 00:00:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/4ee3fddc-4ded-46f8-9529-15f119434bc6_1792x1024.webp" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!KJkz!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F752cddb7-60cc-4e92-b717-9d331888219b_1792x1024.webp 424w, https://substackcdn.com/image/fetch/$s_!KJkz!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F752cddb7-60cc-4e92-b717-9d331888219b_1792x1024.webp 848w, https://substackcdn.com/image/fetch/$s_!KJkz!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F752cddb7-60cc-4e92-b717-9d331888219b_1792x1024.webp 1272w, https://substackcdn.com/image/fetch/$s_!KJkz!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F752cddb7-60cc-4e92-b717-9d331888219b_1792x1024.webp 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!KJkz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F752cddb7-60cc-4e92-b717-9d331888219b_1792x1024.webp" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/752cddb7-60cc-4e92-b717-9d331888219b_1792x1024.webp&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!KJkz!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F752cddb7-60cc-4e92-b717-9d331888219b_1792x1024.webp 424w, https://substackcdn.com/image/fetch/$s_!KJkz!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F752cddb7-60cc-4e92-b717-9d331888219b_1792x1024.webp 848w, https://substackcdn.com/image/fetch/$s_!KJkz!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F752cddb7-60cc-4e92-b717-9d331888219b_1792x1024.webp 1272w, https://substackcdn.com/image/fetch/$s_!KJkz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F752cddb7-60cc-4e92-b717-9d331888219b_1792x1024.webp 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><h2>The Flow Chart</h2><p>I made an <strong>interactive flowchart</strong>/roadmap that may help you find the best way to expose your service to the internet. Check it here <a href="https://roadmap.esc.sh/self-hosting?node=expose-to-internet">roadmap.esc.sh/self-hosting</a></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!7k7X!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73bdfaa7-ff90-4628-b815-3c4ca0c472b7_2000x1180.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!7k7X!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73bdfaa7-ff90-4628-b815-3c4ca0c472b7_2000x1180.png 424w, https://substackcdn.com/image/fetch/$s_!7k7X!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73bdfaa7-ff90-4628-b815-3c4ca0c472b7_2000x1180.png 848w, https://substackcdn.com/image/fetch/$s_!7k7X!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73bdfaa7-ff90-4628-b815-3c4ca0c472b7_2000x1180.png 1272w, https://substackcdn.com/image/fetch/$s_!7k7X!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73bdfaa7-ff90-4628-b815-3c4ca0c472b7_2000x1180.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!7k7X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73bdfaa7-ff90-4628-b815-3c4ca0c472b7_2000x1180.png" width="2000" height="1180" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/73bdfaa7-ff90-4628-b815-3c4ca0c472b7_2000x1180.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1180,&quot;width&quot;:2000,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!7k7X!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73bdfaa7-ff90-4628-b815-3c4ca0c472b7_2000x1180.png 424w, https://substackcdn.com/image/fetch/$s_!7k7X!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73bdfaa7-ff90-4628-b815-3c4ca0c472b7_2000x1180.png 848w, https://substackcdn.com/image/fetch/$s_!7k7X!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73bdfaa7-ff90-4628-b815-3c4ca0c472b7_2000x1180.png 1272w, https://substackcdn.com/image/fetch/$s_!7k7X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73bdfaa7-ff90-4628-b815-3c4ca0c472b7_2000x1180.png 1456w" sizes="100vw"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h2>Update</h2><p>Since writing this post, I have been using Cloudflare tunnel for exposing some of the services. If you are hosting mostly public facing websites, I recommend you try out Cloudflare tunnel.</p><p><a href="https://selfhost.esc.sh/cloudflare-tunnel/">Here is my write up on using Cloudflare tunnel</a></p><p>If you are hosting something like Plex, I still think the below method is the best option, so read on!</p><h2>My Setup</h2><p>I have an old ThinkPad laptop running Debian 12 that is hosting some services that I use<br>on a daily basis. I also use the same laptop to host some internet facing services. That is, these services should be accessible to <strong>everyone on the internet</strong>. And this is the setup I came up with which seem to be a good tradeoff between cost, complexity and security.</p><p>I have a YouTube video with live demo, if you are interested</p><div class="captioned-image-container"><figure><div id="youtube2-dfnNEn9-mLs" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;dfnNEn9-mLs&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/dfnNEn9-mLs?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div></figure></div><h2>Common ways to expose a service to the Internet</h2><p>First let us take a look at some of the common ways to expose a service to the Internet</p><p>You can also checkout this FlowChart explaining different options to expose a service to the internet at <a href="https://roadmap.esc.sh/self-hosting?node=expose-to-internet">roadmap.esc.sh/self-hosting</a></p><h3>Port Forwarding</h3><p>Port forwarding is when you configure your router to redirect requests from the internet to a specific machine in your local network.</p><p><strong>Pros</strong>:</p><ul><li><p>Fairly easy to set up</p></li><li><p>Direct access, no added cost</p></li></ul><p><strong>Cons</strong>:</p><ul><li><p>Huge security risk : It exposes your local network directly to the internet.</p></li><li><p>Privacy concerns : Your home IP address is exposed. You can use services such as cloudflare to get slightly better protection</p></li><li><p>Need for Dynamic DNS : Since your ISP probably uses dynamic IP address, you need to setup something to map a DNS address to your current IP address</p></li></ul><p><strong>My advice</strong>:<br>I personally recommend that you skip this option unless you know exactly what you are doing</p><h3>Using a traditional/Mesh VPN</h3><p>While using a VPN is not directly exposing a service to the internet, I'm mentioning it because there are folks who just want to access their services while away from home. The VPN setup means creating a secure tunnel from the internet to your local network, effectively making your remote device part of the local network.</p><p><strong>Pros</strong>:</p><ul><li><p>Very secure as long as you don't lose your VPN configuration</p></li><li><p>Fairly easy to set up</p></li><li><p>Privacy friendly</p></li></ul><p><strong>Cons</strong>:</p><ul><li><p>Public internet users cannot access your service if you want them to.</p></li><li><p>Complicated (Unless you use Tailscale, then it cannot be any simpler)</p></li></ul><p><strong>My advice</strong>: If all you need is to access your self hosted services on your own devices, just use <a href="https://tailscale.com/">TailScale</a> and do nothing else. But of course, keep reading for educational purposes</p><h3>Tunneling services</h3><p>Services like Cloudflare Tunnel and Ngrok offer easy ways to expose your local services to the internet.</p><p><strong>Pros</strong>:</p><ul><li><p>Very easy to setup</p></li><li><p>Secure</p></li></ul><p><strong>Cons</strong>:</p><ul><li><p>Thirdparty services, less flexibility</p></li></ul><p><strong>My advice</strong>: If you want a fairly easy setup, but are willing to trade privacy and flexibility, I suggest <a href="https://www.cloudflare.com/products/tunnel/">Cloudflare tunnel</a> But if you are willing to put a little bit more effort, I suggest you try the method<br>I use, which is explained below.</p><h2><strong>How I Expose my services to the Internet</strong></h2><p>TL;DR : I have a Cloud virtual machine running Nginx that acts as a reverse proxy. I have a Wireguard server on the Cloud VM. My ThinkPad laptop connects to this CloudVM via Wireguard. The Nginx reverse proxies directly to the VPN interface. I use Docker to run all of the services</p><p><br>Here is a simplified diagram showing how it fits together</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!oPzI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefa880c8-e248-4ebd-af1e-d4ec5f2dc16a_1010x468.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!oPzI!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefa880c8-e248-4ebd-af1e-d4ec5f2dc16a_1010x468.png 424w, https://substackcdn.com/image/fetch/$s_!oPzI!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefa880c8-e248-4ebd-af1e-d4ec5f2dc16a_1010x468.png 848w, https://substackcdn.com/image/fetch/$s_!oPzI!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefa880c8-e248-4ebd-af1e-d4ec5f2dc16a_1010x468.png 1272w, https://substackcdn.com/image/fetch/$s_!oPzI!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefa880c8-e248-4ebd-af1e-d4ec5f2dc16a_1010x468.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!oPzI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefa880c8-e248-4ebd-af1e-d4ec5f2dc16a_1010x468.png" width="1010" height="468" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/efa880c8-e248-4ebd-af1e-d4ec5f2dc16a_1010x468.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:468,&quot;width&quot;:1010,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!oPzI!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefa880c8-e248-4ebd-af1e-d4ec5f2dc16a_1010x468.png 424w, https://substackcdn.com/image/fetch/$s_!oPzI!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefa880c8-e248-4ebd-af1e-d4ec5f2dc16a_1010x468.png 848w, https://substackcdn.com/image/fetch/$s_!oPzI!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefa880c8-e248-4ebd-af1e-d4ec5f2dc16a_1010x468.png 1272w, https://substackcdn.com/image/fetch/$s_!oPzI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefa880c8-e248-4ebd-af1e-d4ec5f2dc16a_1010x468.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><strong>Pros</strong>:</p><ul><li><p>Secure</p></li><li><p>Very flexible</p></li><li><p>Private</p></li><li><p>Does not depend on a third party</p></li><li><p>It's fun</p></li></ul><p><strong>Cons</strong>:</p><ul><li><p>You gotta pay $5 a month at least</p></li><li><p>A bit more involve setup</p></li></ul><h2>How to set it up</h2><p>For this example, I will use my laptop to host a simple web service and then I willuse my Digital Ocean VM to expose it to the internet, with SSL and everything.You can replicate the same example with slight modifications for any sort of service.</p><h3>Step 1 : Let's host a simple service locally</h3><p>I have a simple demo app <a href="https://github.com/MansoorMajeed/go-hello-world">HERE</a>, I am going to use that.</p><p>I assume you have Docker setup and running, if not, it does not matter how you host your service,<br>it just have to be listening on a local port.</p><p>In my case, I am going to install Docker on a Virtual Machine and start the service from there.</p><pre><code>docker run -p 8080:8080 mansoor1/go-hello-world
</code></pre><p>Once that is running, from the same machine, I can access this service</p><pre><code>mansoor@debian:~$ curl localhost:8080
Hello from: 228c90c9b4c0
Current Time: Sun, 31 Dec 2023 03:03:50 UTC
</code></pre><p>Great! So we have our service running locally</p><h3>Step 2 : Setting up our CloudVM</h3><p>For this, you will need a Cloud VM. I personally recommend DigitalOcean or Linode.<br>We only need the smallest possible instance since the server will be using only<br>minimal stuff. You can even sign up for a AWS free account or Google Cloud trial.</p><p>For this, I will use a DigitalOcean VM I already have.</p><p>In case you are not familiar with this, I have an old video explaining how to set it up from scratch</p><blockquote><p>Note: You do not have to configure Nginx yet, just until Nginx is installed</p></blockquote><div class="captioned-image-container"><figure><div id="youtube2-kDcn9npjoPs" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;kDcn9npjoPs&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/kDcn9npjoPs?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div></figure></div><p>At this point, you should have a Cloud VM running with nginx installed.</p><h3>Step 3 : <strong>Connecting your local machine to the Cloud VM via Wireguard</strong></h3><p>Now, we need to create a tunnel between the local virtual machine and the Cloud VM.</p><p>I have another post going into details about how to do that. Go ahead and follow that to set up WireGuard. HERE : <a href="https://blog.esc.sh/wireguard-vpn-setup/">https://blog.esc.sh/wireguard-vpn-setup</a></p><blockquote><p>NOTE : In the Wireguard setup, we <strong>DO NOT WANT our client to send INTERNET</strong> traffic via the Cloud VM<br>Make sure you don't configure that.</p></blockquote><p>Once you are done with setting up Wireguard following the post I linked above, your local VM and the Cloud VM should be connected<br>via a tunnel.</p><p>The <strong>Cloud VM</strong> should be able to ping the <strong>local VM</strong>.</p><pre><code>mansoor@server:~$ ping -c 2 10.0.0.2  # this is the client's IP
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=16.3 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=8.21 ms

--- 10.0.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 8.212/12.261/16.310/4.049 ms
</code></pre><blockquote><p>Note : If that does not work, there is likely an issue with the Wireguard configuration<br>Fix this before proceeding</p></blockquote><h3>Step 4 : Testing the service access from the Cloud VM</h3><p>Now that we have a tunnel in place, our Nginx server (The Cloud VM) should be able to access our simple service directly.</p><p>Here, I am using <code>curl</code> to test connection to <code>10.0.0.2</code> which is the Wireguard tunnel IP for the client VM<br>where our service is running on port 8080 using Docker.</p><pre><code>mansoor@cloud-vm:~$ curl 10.0.0.2:8080
Hello from: 1a2974ee0667
Current Time: Mon, 01 Jan 2024 21:17:09 UTC
mansoor@cloud-vm:~$
</code></pre><p>Perfect! It works.</p><blockquote><p>If that does not work, but the <code>ping</code> worked in the previous step, make sure the service is running properly on the client machine, that docker is running with the proper flags to expose the port 8080</p></blockquote><h3>Step 5 : Pointing your domain name to the Cloud VM</h3><p>I want to expose my service to the internet at the domain name <code>demo.ext.esc.sh</code>.<br>For this, you would need a domain name, I assume you do have one.</p><p>We need to point the domain name of our choice to the <strong>IP address of our Cloud VM</strong>.<br>I use Cloudflare for DNS, so I can do that in the CloudFlare dashboard.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!jyXR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32f4baf0-b358-4a0e-b927-bcd848720d7f_1223x405.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!jyXR!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32f4baf0-b358-4a0e-b927-bcd848720d7f_1223x405.png 424w, https://substackcdn.com/image/fetch/$s_!jyXR!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32f4baf0-b358-4a0e-b927-bcd848720d7f_1223x405.png 848w, https://substackcdn.com/image/fetch/$s_!jyXR!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32f4baf0-b358-4a0e-b927-bcd848720d7f_1223x405.png 1272w, https://substackcdn.com/image/fetch/$s_!jyXR!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32f4baf0-b358-4a0e-b927-bcd848720d7f_1223x405.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!jyXR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32f4baf0-b358-4a0e-b927-bcd848720d7f_1223x405.png" width="1223" height="405" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/32f4baf0-b358-4a0e-b927-bcd848720d7f_1223x405.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:405,&quot;width&quot;:1223,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!jyXR!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32f4baf0-b358-4a0e-b927-bcd848720d7f_1223x405.png 424w, https://substackcdn.com/image/fetch/$s_!jyXR!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32f4baf0-b358-4a0e-b927-bcd848720d7f_1223x405.png 848w, https://substackcdn.com/image/fetch/$s_!jyXR!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32f4baf0-b358-4a0e-b927-bcd848720d7f_1223x405.png 1272w, https://substackcdn.com/image/fetch/$s_!jyXR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F32f4baf0-b358-4a0e-b927-bcd848720d7f_1223x405.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Verify that the DNS has been updated. I like to use <code>dig</code> command to do that. You can use<br><code>dig</code> or <code>nslookup</code> or even <code>ping</code></p><pre><code>mansoor@demo:~$ ping -c 1 demo.ext.esc.sh
PING demo.ext.esc.sh (138.197.67.72) 56(84) bytes of data.
64 bytes from demo (138.197.67.72): icmp_seq=1 ttl=64 time=0.034 ms
</code></pre><p>So we can see that the DNS is propagated</p><h3>Step 6 : Configuring Nginx</h3><h4><strong>Getting TLS certificates</strong></h4><p>We can get free TLS certificates using Lets' Encrypt.</p><p>I have another blog post explaining how to do that, check that out : <a href="https://esc.sh/blog/lets-encrypt-and-nginx-definitive-guide/">Let's Encrypt + Nginx - Definitive Guide</a></p><p>Make sure to use the correct domain names.</p><blockquote><p>Note: I chose not to use <code>www.</code> part and use only <code>demo.ext.esc.sh</code>. This is up to you<br>If you plan on having other people visit your website/service, it is better to have <code>www.</code><br>enabled and redirected to your root (or main) domain (or subdomain)</p></blockquote><p>At the end, we should have these on the Cloud VM</p><ul><li><p>The file <code>/etc/nginx/snippets/letsencrypt.conf</code> in place</p></li><li><p>Directory <code>/var/www/letsencrypt</code> created</p></li><li><p>The certificate and private key under <code>/etc/letsencrypt/live/demo.ext.esc.sh/</code></p></li><li><p>The proper Nginx configuration at <code>/etc/nginx/sites-enabled/demo.ext.esc.sh</code></p></li></ul><p>Mine looks like this</p><pre><code>mansoor@demo:~$ cat /etc/nginx/sites-enabled/demo.ext.esc.sh

server {

    include /etc/nginx/snippets/letsencrypt.conf;

    server_name demo.ext.esc.sh;

    root /var/www/demo.ext.esc.sh;
    index index.html;

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/demo.ext.esc.sh/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/demo.ext.esc.sh/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}
server {
    if ($host = demo.ext.esc.sh) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    listen 80;

    server_name demo.ext.esc.sh;
    return 404; # managed by Certbot
}

</code></pre><h4>Proxying to our service</h4><p>All we have to do now is to update the Nginx config to proxy any request to our backend.</p><blockquote><p>Note: what to add to the proxy configuration depends on what sort of application we<br>are hosting. A quick google search should get you the appropriate configs</p></blockquote><p>For a very simple proxy, a simple <code>proxy_pass</code> is enough</p><p>So, we add this to our Nginx configuration</p><pre><code>    location / {
            proxy_pass http://10.0.0.2:8080;
    }
</code></pre><p>The full config looks like this</p><pre><code>mansoor@demo:~$ cat /etc/nginx/sites-enabled/demo.ext.esc.sh
server {

    include /etc/nginx/snippets/letsencrypt.conf;

    server_name demo.ext.esc.sh;

    root /var/www/demo.ext.esc.sh;
    index index.html;

    listen 443 ssl; # managed by Certbot
    #---snip TLS stuff----

    location / {
            proxy_pass http://10.0.0.2:8080;
    }

}
#---snip----
</code></pre><p>Once that is done, reload Nginx</p><pre><code>sudo systemctl reload nginx
</code></pre><h3><strong>Step 7 : Testing and verifying</strong></h3><p>We are ready to test if our setup is working. Open a browser (or `curl`) to the site we configured</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!RcTw!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57384d73-b4d5-481b-82bf-15cd6e183222_660x228.png 424w, https://substackcdn.com/image/fetch/$s_!RcTw!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57384d73-b4d5-481b-82bf-15cd6e183222_660x228.png 848w, https://substackcdn.com/image/fetch/$s_!RcTw!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57384d73-b4d5-481b-82bf-15cd6e183222_660x228.png 1272w, https://substackcdn.com/image/fetch/$s_!RcTw!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57384d73-b4d5-481b-82bf-15cd6e183222_660x228.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!RcTw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57384d73-b4d5-481b-82bf-15cd6e183222_660x228.png" width="660" height="228" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/57384d73-b4d5-481b-82bf-15cd6e183222_660x228.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:228,&quot;width&quot;:660,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!RcTw!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57384d73-b4d5-481b-82bf-15cd6e183222_660x228.png 424w, https://substackcdn.com/image/fetch/$s_!RcTw!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57384d73-b4d5-481b-82bf-15cd6e183222_660x228.png 848w, https://substackcdn.com/image/fetch/$s_!RcTw!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57384d73-b4d5-481b-82bf-15cd6e183222_660x228.png 1272w, https://substackcdn.com/image/fetch/$s_!RcTw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57384d73-b4d5-481b-82bf-15cd6e183222_660x228.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><h3>Step 8 : Optional - Making it even more secure</h3><p>If you are connecting from the local machine directly to the remote server, that means<br>the remote server has full network access to your local machine for all the services that<br>are listening on the host. This is totally fine unless someone gains access to your remote server (Cloud VM).</p><p>We can go a step futher and make it even more secure.</p><p>This is what I would do:</p><ul><li><p>Create a single Docker network for all my services that needs to be exposed to the internet</p></li><li><p>The same network will also run a wireguard client container that can connect to the Wireguard server</p></li><li><p>Using a docker-compose.yaml would be the best option, take a look <a href="https://github.com/linuxserver/docker-wireguard?tab=readme-ov-file#docker-compose-recommended-click-here-for-more-info">here for an example</a></p></li></ul><p>This way, even the worst case scenario where your remote Cloud VM gets hacked, they will<br>only be able to get network access to this docker network that is housing a bunch of private<br>facing services</p><p>Another thing you can consider is to use a dedicated VM for all of your private facing<br>services, and connect via Wireguard from that dedicated VM. This is similar to the docker<br>approach, just that it is a VM instead of Containers.</p><h2>Conclusion</h2><p>We are all set here. If you plan to use this for any sort of production services,<br>It would make sense to add some sort of simple monitoring to make sure that your<br>service stays up.</p><p>Leave a comment if you have any questions</p>]]></content:encoded></item></channel></rss>