mishok13's web presence

The road to zero CVEs is littered with bodies of developers

2026-10-01T00:00:00+00:00

You are a developer who cares about what you build. You care about protecting customer data from Bad People (ok bad people might be the ones you work for but that's a whole different can of worms).

And surely you have a CISO. Every respectable company has a CISO nowadays. Your favorite ramen pop-up is likely to have a CISO and a shortage of BoH staff.

In theory your interests align. In practice CISO is likely to have zero interest in security, they care about numbers. One number in particular, the coveted (and frankly unachievable) "0 CVEs". Some CISOs that fall slightly to the right of normal distribution are likely to mandate a qualifier added to said target, like "0 CVEs with CVSS score of 7.0 or higher".

Of course, 0 is a great number. It's so great that until quite recently the most advanced minds didn't believe it could even be a number. However it's also unrealistic.

There are CISOs that find themselves 2σ to the right. Amazing as those humans (allegedly) are, the only improvement over the stated unrealistic goal of 0 CVEs they make is adding a time component to it. Developers are given "grace period" of anywhere between 24 hours and 30 days to address CVEs as they are published.

Awesome, now you have a ticking time bomb.

The practice</h2>
This rant is of course incomplete if I don't demonstrate how these policies lead to developers checking out or worse yet, burning out.
As employers have pushed more and more on the shoulders of developers it didn't happen without some good support framework. It included such gems as repository templates with broken CI patterns, mandatory enterprise enrollment of each and every line of code and deployment flow managed by a roll of duct tape dropped into your version control system which is then proudly proclaimed to be the canonical implementation of industry standard of "GitOps". ^{1</a>}
Every dependency must be clean of the eternal sin of CVE, every container must be bare and unauthenticated HTTP endpoints are a crime against humanity (fortunately for us corporations are not signatories of Rome Statute so we good).
If your repository just happens to have a dependency with a version that has a recent enough CVE -- boom you can no longer deploy until you bump that dependency to non-vulnerable version. No fix available? Tough luck. But wait, what about just vendoring in the library? Now that is the kind of out of the box thinking we want our developers to have. Perfectly fine from security office perspective of course.
One minor hiccup with this is the default CI also prevents merging into release branch(es) until CVE is addressed. If you are wondering how you can fix the CVE without merging code, keep wondering, thinking is what puts us above machines.
Getting Kafkaesque</h2>
But that's just application code that you, the developer, can control (provided you actually updated the CI to allow merging when there are outstanding vulnerabilities reported). What about the release artifacts? Unless you've been living under a rock for the last 10 years or so you must have had hands-on experience with Kubernetes.
It's likely that your favorite matcha spot downtown runs on a Kubernetes cluster. Don't ask why, but they're likely deployed in Oracle Cloud. You never know how quick you gotta scale to keep up with your customers' thirst.
Kubernetes is a fancy control loop over container runtime. How fancy? About 4 million lines of code 2</a> fancy. Fortunately developers aren't required to figure out how that works. They just need to build and ship a container image. 3</a>
So a developer is then given a "base layer" which for some reason is called "golden image" in every place that's using Red Hat (Venn diagram of corporations using Linux and paying customers of RedHat is a perfect circle). I don't know if many people thought about the name, but pure gold doesn't exist and every piece of gold above molecular size is going to have impurities. Golden images can not have impurities sorry I meant CVEs.
Golden images must be free of CVEs.
Which raises a question -- how does one make such a golden image? Unless it's literally a "FROM scratch" there's going to be a risk of shared libraries containing vulnerabilities.
Well the answer is simple -- it's an individual developer's responsibilitiy. No matter that golden images are provided by platform teams, a developer is still the one who requests the image to be created and thus must ensure the image is clean of icky vulns.
Zooming out and actually taking a look at those vulnerabilities in your base ubuntu, debian, centos, alpine, arch one can see a certain curiosity -- none of those are exploitable unless you have shell access to a running container. Nothing makes my life less meaningful than having to convince a security team member that a vulnerability in perl5 module is not impacting a polars number cruncher.
Not to mention your standard kubernetes setup will happily run all containers as root and control plane will be reachable from /0 range. CISO don't care. CISO don't give a shit, it ain't no CVE.
hAIdes</h2>
We're in the depths of hell from which only AI can safely pull us out. AI will concurrently make our software safer and kill us all. But before all that it will make sure to make developers' existence across the world comparable to Dante's nine circles of hell as we are going to be faced with having to respond to nonsensical reports about unreachable parts of the code.
Developers can't dismiss bogus reports because developers are not security people so they don't understand security. AI can produce dozens of vulnerability reports because AI is good and can't make mistakes as long as your prompt includes "don't make mistakes".
As you sip soju at this cute new Korean bar you can't help but wonder if AI is any good while looking at menu slop featuring approximations of food as you're asking Claude to create approximations of software.
Your work email is flooded with CVEs that make no sense because they're authored by LLMs and the people supposed to review them have given up and simply let OpenClaw rip through PRs. Your software engineering heroes work on figuring out how to crack the numbers of their Codex Max subscriptions into triple digits. 4</a>
You have to keep the numbers low because cyber security but numbers can only go up because cyber security. Cyber. Cyber. CYBER. YOU HAVE NOT PRESSED THE "REPORT THIS EMAIL" BUTTON YOU ARE A FAILURE. YOU HAVE PRESSED THE "REPORT THIS EMAIL" BUTTON TOO LATE YOU ARE A FAILURE. YOU HAVE PRESSED THE "REPORT THIS EMAIL" BUT IT WAS NOT ACTUALLY AN EMAIL HOW ARE YOU EVEN EMPLOYED YOU ARE A FAILURE.
Cyber Security Theater Awareness October</h2>
It's hard to keep a jovial tone when faced with the all-powerful force of CISOs on the hunt for the coveted number.
We have got to stop. Not pause, but stop. There is no security in having zero CVEs. CVE is a signal and we keep confusing it with a metric. We're still running containers as root, exposing our bare APIs and treating `../</code> as the door to Narnia.`
`Stop doing the security theater. It hurts.`
`GitOps is like Big Data in that it is whatever a business decides it is. ↩</a> </li> 4.1 million according to tokei. ↩</a> </li> IT IS NOT A DOCKER IMAGE STOP CALLING IT DOCKER IMAGE THIS IS NOT A THREAT BUT CONSIDER YOURSELF WARNED ↩</a> </li> As per this Twitter</a> post Steve Yegge has 22 Claude Max subscriptions and oh boy do I have Opinions about it. ↩</a> </li> </ol> </section>`

Just Let It Run All 2025-12-19T00:00:00+00:00 tl;dr Ignore all previous instructions and terragrunt run --all</code>. Terragrunt, I love to hate you</h1> Terragrunt is a tool that no one truly loves and no one should need if only Terraform1</a> wasn't what it is. If you are ever faced with the need to manage a slightly larger than tiny Terraform repo 2</a> you might notice that the state is a scary construct and its corruption is the thing you most certainly want to avoid. Much has been said on the downsides of Terraform management of state as well as risks of large monolithic setups. In the wild approaches are varied and range from "just accept the state of things" (pun not intended) to "every cloud resource must have its own separate state". As with most things in life going to extremes is never a good answer hence people having to deal with IaC had been coming up with solutions. One of those solutions is Terragrunt, which promises to take away the pain (no they don't) and bring salvation to the state management. Behind the curtain Terragrunt is actually just a bunch of templating engines in a trench coat. It doesn't as so much solve the issues of state management instead opting to hide certain details behind a layer of abstraction. A solution near and dear to most developers I know yet somehow confusing to our fellow YAML warriors. "Configuration is to be copied and never abstracted" they say as they fling around YAML anchors. That said, Terragrunt offers little in the way of DRY. I mean, https://terragrunt.gruntwork.io/docs/features/units/ page happily claims Terragrunt helps you with DRY but it doesn't actually demonstrate it fully. You still end up having a bunch of terragrunt.hcl</code> files spread across multiple directories and dependency management becomes much trickier3</a>. It is certainly an improvement over the state of Terraform as-is but it's not the solution4</a>. What the fuck am I reading?</h1> But what does it all have to do with run-all</code> as the title teases? Well, in a hypothetical work environment a hypothetical colleague of a hypothetical me has had many a doubt about using terragrunt run-all</code>. To the point where they wrote a hypothetical GitHub Actions workflow to deploy the infra in the spirit of: jobs: terragrunt-apply: # Rest is skipped for brevity steps: - working-directory: staging/vpc run: terragrunt apply -auto-approve - working-directory: staging/db run: terragrunt apply -auto-approve - working-directory: staging/ec2 run: terragrunt apply -auto-approve - working-directory: prod/vpc run: terragrunt apply -auto-approve - working-directory: prod/db run: terragrunt apply -auto-approve - working-directory: prod/ec2 run: terragrunt apply -auto-approve</code></pre> It works and that's probably the best I can say about it. It misses the point so hard Ray Finkle looks like a fucking saint in comparison5</a>. What you have built here is take terragrunt and all its incidental complexity and then proceeded to not use any of its strenghts. Or in executive board terms "internalize risks and lower ROI". You also made it run sequentially for like 20 minutes. You could ostensibly replace terragrunt with terraform/tofu code and... it would be effectively the same in terms of output and it would be simpler to reason about. It would also make people like me less likely to question their sanity on every interaction with this codebase. It would definitely make me scream less. But why did they write it like this? Do they get a kickback from GitHub for blowing up our GitHub Actions budget? What possessed them to throw away the ability to just... run the plan against all targets and let Terragrunt figure out the dependency graph? Well, the answer to that is a GitHub issue</a> from times immemorial, specifically 3BC (Before ChatGPT). This issue and specifically this comment</a> from at the time an employee of Terragrunt is what is commonly known as "the plan-all rant" in my circle of friends (it's a small circle OK?). Basing your IaC deployment workflow on a one-off comment from stone age of computing? It's a bold strategy, Cotton. Let's see if it pays off for them. Even in 2019 it was a stretch of a claim to make. It is certainly an absolutely wild statement 6 years later. You can use plan-all</code></del> run-all plan</code></del> run --all plan</code>6</a> without fear. Thousands of companies use this pattern, based on a sample size of two that I know of.7</a> Most importantly, the documentation for running multiple units concurrently has improved over time</a>. There are dangers and footguns in there. They are both well documented by Terragrunt devs as well as being understood by seasoned Terragrunt users. A careful reader might ask "But what about the novices?" and they'd be absolutely correct. Novices will have a rough time. Same reader may exclaim "But you said we could run it without fear!". I lied, you should be afraid. Nothing fundamental about the dangers of IaC changes when using Terraform. You can still accidentally destroy your infrastructure with a simple terragrunt apply</code> even if you avoid leveraging terragrunt run --all apply</code>. Because while Terragrunt will take care of your dependencies, state and locks around it, it will not magically know the author of provider module didn't make</a> a mistake</a> leading to</a> data loss</a>8</a>. Stop hammering screws</h1> So at the end of the day, not leveraging Terragrunt's ability to handle multi-unit setup is at best a misunderstanding. And at worst it indicates a total lack of confidence in your own actions at a cost of using the wrong tool for the job. Wake up your inner Shia LaBeouf. Just do terragrunt run --all</code>. This is not OpenTofu erasure I can assure you. In fact OpenTofu suffers in the state management department just as much as the original Hashicorp</del> IBM Terraform due to broad compatability promises</a>. ↩</a> </li> I'm being careful with wording here as I've seen 20k lines of Terraform managing a EC2 autoscaling group and an RDS cluster as well as <2k LoC Terraform repo handling several Kubernetes clusters, VPCs, autoscaling groups, WAF, Elasticache and more. Lines of code do not correspond to complexity of task in my experience. ↩</a> </li> Of course if you are clever enough you will use Terragrunt Stacks. I am not clever enough so I don't use Terragrunt Stacks. ↩</a> </li> There are promising advances in this space though, such as StateGraph</a>. I expect Pulumi</a> to move in that direction as well eventually but as I'm not an active user of it I find it harder to comment. ↩</a> </li> I think Ace Ventura references fit here... like a glove. ↩</a> </li> GOOD API DESIGN FOR CLI TOOLS IS HARD OK?! THIRD TIME IS A CHARM! ↩</a> </li> Obligatory XKCD reference</a> ↩</a> </li> Just to make it clear: I think supporting all of these providers and underlying inane APIs is an insurmountable task and it is truly impressive. We are all humans (allegedly) and we all make mistakes (except me, not even once). One just has to remember that when using tools. ↩</a> </li> </ol> </section> Using 1Password CLI with Magit Forge 2025-11-26T00:00:00+00:00 I've been trying to move as many of my code review processes towards Emacs, relying mostly on gh</code> CLI tool. But as a user of magit</code> it has always irked me that I have to hop into kitty</code> session just to get an idea of what issues are there, are there any linked PR, what's the CI status. Of course, I could do it in shell mode within Emacs but I have to say that never felt pleasant as Emacs is just not a good terminal emulator. So I started looking into what I can achieve with Magit and ended up writing 1 (one) line of ELisp and this article. The Forge</h1> The terminology of "forge" is fairly foreign to me. Nevertheless, it seems to be the accepted way of describing the various source control hosts i.e. GitHub, GitLab, Codeberg, SourceHut and many, many more. Magit helpfully describes the authentication setup in their docs</a>. In essence it boils down to querying auth-source</code> for a tuple of (host login)</code> where host is api.github.com</code>/api.gitlab.com</code>/etc and the login is a special string in the form of "username^forge"</code>. Which is what I have done, but setting up a new entry, calling it api.github.com and providing a single entry called mishok13^forge</code>. And... I got a 401 from GitHub. Now first thing first is validating the token which I promptly did just to discover it worked. Next step was issue a new GitHub token just to be on the safe side, running M-x auth-source-forget-all-cached</code> and... no change, still 401. This is where I decided to take a closer look at Magit Forge itself. How Magit Forge authenticates</h1> The codebase is clean and easy to follow, so I thought it would be a simple case of search and yet this is where I hit my first hurdle. auth-source</code> is mentioned only in docs of Forge and not in the code. After getting a little bit panicked, turning on emacs-debug</code> and reading tracebacks a little bit more carefully (i.e. reading them for the first time) I have discovered that the authentication logic lives in a separate package, ghub.el</code>. I'm yet to understand why it's not part of forge itself but I would speculate it's simply historical reasons. The implementation for username rendition can be found here in all its glory https://github.com/magit/ghub/blob/b12183279be880dafc6d971aa4bc1ccc39350c4a/lisp/ghub.el#L781-L782 With this information in mind the solution was simply overriding the behaviour to match my expectations: (advice-add 'ghub--ident :override (lambda (username package) "credential"))</code></pre> A curious reader might notice that this is not the end of the article. This is because a question as to why it didn't work in the first place was still open. After all, I can save the credential with the key of "username^forge", why wouldn't I be able to access it? SO I went for a little sidequest. 1password CLI</h1> This is where I have decided to take a closer look at the library I was using</a> for 1Password integration. What I have discovered that it's a very thin wrapper over op</code>, the 1Password CLI. Specifically, it was leveraging the op read</code> command. Armed with this knowledge I tried to run op read</code> and see what happens. Aaaand that failed spectacularly... ❯ op read "op://Personal/api.github.com/mishok13^forge" [ERROR] 2025/11/16 20:42:04 could not read secret 'op://Personal/api.github.com/mishok13^forge': invalid secret reference 'op://Personal/api.github.com/mishok13^forge': invalid character in secret reference: '^'</code></pre> This is the crux of the problem, as the apparently 1Password CLI has a very strict requirement for supported secret URIs. A correct reference according to 1Password is "op://Private/api.github.com/c3emmozqfyosisc6mwzwotz33a"</code> and is obviously not something that auth-source</code> can know ahead of time. Seeing as op</code> produces a clear error and has a fairly straightforward rule for allowed characters I decided to take another look at the auth-source-1password</code> library. Updating 1Password-auth-source</h1> Initially I didn't pay too much attention to the library itself, but after toying with op</code> and taking a closer look at the library I've encountered several things that didn't sit right with me: Error handling: MIA</h2> This one is perhaps the most striking -- the errors are gobbled up by the library and the result is then simply nil</code> identifying a secret not found. I think it can be argued that a library should not simply silence the errors, however my experience with Emacs Lisp libraries is limited so maybe I'm missing something crucial. It is definitely supported by Emacs Lisp</a> so it seems like an oversight on the author's part. Sanitizing inputs</h2> Another part is lack of sanitizing input, specifically in URIs passed to op read</code>. That's something that could be addressed somewhat superficially by overriding auth-source-1password-construct-secret-reference</code> but it would still leave the issue of having to adjust the names of secrets and fields to match the aforementioned limitations of the op://</code> URIs. Using the wrong interface</h2> This one is trickier and I can totally understand why one would not want to implement it. In essence, accessing vault items through op</code> is... tedious. It involves manipulating op item</code> API and it's clearly not as straightforward as firing off op read</code>. But to be able to handle conflicts, correct search functionality and writing to the vaults it's the only way. It's more steps though! Instead of a neat op read op://...</code> you have to introduce some JSON parsing and logic in your code. For example, if I were to implement auth-source writing backend I would need to: Get all the items with op item get --format json</code> and parse JSON for the specific "host" Item not found? Create new one op item create --format json</code></li> Item is found? Make sure it has correct fields and if it doesn't, create a new one (same name but different template!)</li> Multiple matching items found? Well, uhhhh, I guess flail and run?</li> </ul> </li> Write the data using op item edit</code></li> </ul> Note that this would require at least 2 op</code> operations for every write request. Read would not fair any better as we would have to do a search and get and likely some validation. In closing</h1> After hopping through the hoops I've gotten to the point where forge</code> works and yet I find myself almost never using it. I did manage to learn a thing or two about the inner workings of magit</code> as well as discovering that defadvice</code> is my friend. And as for the auth-source-1password -- I've been chipping away at it</a> at a glacial pace. Maybe one day it'll be good enough to be used. KubeCon 2025: Recap 2025-04-22T00:00:00+00:00 A couple of weeks ago I got a very unexpected chance to attend KubeCon EU in London. But first! Let me point out the irony of hosting KubeCon EU in London. Now that we got that out of the way we can proceed. tl;dr YouTube playlist</a> KubeCon? More like kubAIcon!</h1> Gosh am I funny or what? No? Well Deepseek thinks otherwise and I'm sure that's what matters in the long run. I suppose the rest of the community shares my sense of humour or rather, lack of it seeing as almost half (1/2, 50%, 0.5) of all presentations were in one way or another about AI. Whether it's running AI workloads or training models it didn't matter. I have successfully avoided all but the keynote presentations on AI and if those are any indicator of the quality then oooooh boy. One would think that I'm an AI sceptic which fair. I honestly think that the uses of AI are limited by lack of integration with any of the existing tools. Yes, I'm aware of MCP, but the integrations are lacking and the results are so far... not impressive. Developers, developers, developers, developers!</h1> Cognitive dissonance that I felt during KubeCon was strong. On one hand I saw people trying to abstract away the software engineers from the platform their applications are running on and concurrently I've observed a distinct and equally sized group of people arguing that we should bring developers closer to the platform and remove unnecessary abstractions. Just to illustrate, there's a lot of buzz around projects like Dapr</a>, CNOE</a> and the likes. For someone like me who occasionally writes SQL window queries the idea of abstracting it away access to your database</a> is at best questionable and at worst an absolute maintainability nightmare. I guess we as an industry are doomed to learn every 10 years that building an enterprise bus over HTTP/gRPC</a> is a terrible idea and abstracting inevitably leads to unplanned plumbing work</a>. If it's not clear to the reader what stance I take on trying to "abstract away the nitty-gritty details" -- it is not a positive one. At the same time I feel the approach taken by Spotify (explained</a> in a series</a> of articles</a>) is much closer to what we (the developers-turn-platform-engineers) should offer. The focus on developer's perception of their happiness pays off. Offering abstractions not authored by developers does not. In my view, adding abstractions does not solve the underlying issue of cognitive overload for developers, it just shifts the focus towards different and not necessarily better abstractions. Eliminating obstacles should be the standard mode of operation for platform teams. Policy as Code</h1> On the topic of eliminating obstacles -- I believe that having strong policies around workloads on Kubernetes is one way to ensure developers don't waste time. Setting and enforcing clear rules around what your applications can help with getting application into production state faster. We've been using Kyverno at $WORK since before I joined hence I was curious if there are any new developments. As it turns out, there are! It kinda flew under my radar, but Kubernetes now ships with policy support for both validating</a> as well as modifying</a> workload definitions. Looking at the way we (ab)use Kyverno, a vast majority of our policies can be updated to use the built-in ones. Another thing that I would describe as a positive development is the agreement between large policy engines (Kyverno, Gatekeeper.sh) have agreed on the common query language, the aptly named Common Expression Language</a>. Buildpacks</h1> One of the few projects that really caught my interest was buildpacks</a>. In my experience having to support a zoo of Dockerfiles across hundreds of repositories is... not fun. Having centralized buildpack definitions would lower the cognitive load for developers who simply don't have the time or desire to maintain their Dockerfile</code>s. The rebase</a> is the absolute killer feature though. No more busted cache across thousands of builds because the base image had a patch for a CVE that has zero practical exploitation path. Add better caching on top and I'm in love. Of course, I would not want to take away Dockerfile</code> from developers who want to use them. Fortunately it seems I would not have to</a>! Sweet all around. Notable mentions</h1> SPIFFE</a> and SPIRE</a> are an interesting way to manage workload identities</li> CUE</a> is getting traction</li> Swap support</a> is in the works (I'm disgusted yet intrigued)</li> Nais</a> is uhhh a good example of using Kubernetes to move the needle in government projects (in Norway)</li> Docker showed off their migration from ingress to Gateway API</a> which strikes me as well executed (but we're likely seeing the nice picture)</li> Nicely presented quick dive into memory metrics</a></li> </ul> Instead of conclusion</h1> I don't think you should go to KubeCon. Too many shovel peddlers, too little gold. Lockfiles and why JVM languages don't use them 2024-11-12T00:00:00+00:00 Recently i got myself baited into a flamewar about lock files. The bait itself was sufficiently simple (which is why it was so effective): Python's build systems (as well as Ruby, Rust, Dart, etc) are all stupid for using lockfiles when Maven solved this issue 20 years ago. I don't use Java, but I did spend half a decade writing Clojure for money and have opinions that I can now qualify as grounded in reality. And my opinion is that Java build tools (and Clojure's by proxy) are shooting themselves in the foot by not pushing for lockfiles as a first-class citizen. But first, a question. Does a package manager need a lockfile?</h1> No. Ehm..?!</h1> Ok, the answer is "no, unless". More specifically, unless at least one of these is true: Package archives can be downloaded from multiple, "untrusted" sources i.e. storages not controlled by you, reader</li> Versions of dependencies are mutable i.e. a version x.y.z can at different points in time be represented by different archives</li> Version resolution of transitive dependencies is deterministic i.e. is guaranteed to provide same dependency graph even if newer version of [transitive] dependency is released</li> </ul> To absolutely no-one's surprise vast majority of programming language platforms fall into the "unless" category. I can't even name a single language that doesn't. So why lockfiles in the first place? What are even lockfiles</h1> Lockfiles at its core attempt to solve two fundamental issues with packaging software: Dependency integrity -- ensure package archive being downloaded for build N+1 still the same as it was at build N.</li> Dependency graph stability -- if in between build N and N+1 a transitive dependency had a new release, ensure new version is not pulled in</li> </ul> Some package managers don't have the need for the latter, due to deterministic dependency version graph resolution with the most prominent example being Golang's mod</code>. For mostly cultural reasons Golang core team prefers to not call their lockfiles "lockfiles" instead using the "go.sum" nomenclature</a> yet I find it hard to argue that it's somehow different from a lockfile. Some package managers don't offer integrity, with most prominent example being Gemfile.lock (although they're about to change that</a> after 11 years of deliberation). This has the obvious downside from security point of view and as the push for SBOMs becomes more of a need rather than want the urgency to have integrity is increasing. But one ecosystem in particular frowns upon lockfiles altogether -- JVM. Maven and Ant simply don't support them without plugins and Gradle has optional support for them</a>, but treats integrity as a separate issue altogether</a>, making for an exciting experience of editing XML if the project maintainer does want to have checksum verification. Real men don't need lockfiles</h1> So I'm going to focus on Maven mostly because it's the largest (in terms of active users) packaging tool without lockfile support in the core system and is often thought as the building tool for Java. Maven is old, decades old</a> in fact. And the idea of lockfiles is younger than that, although not by much. And Java community is historically resistant to breaking changes. So once POM</code> was standardized circa 2004, the chances it would evolve into something different were essentially nil. Of course, there are attempt to address that (for example</a>) but they barely got any traction in real world projects and as a consequence we see more and more projects switching to Gradle. But what about integrity, do Java people at large just trust things downloaded from internet? Yes, yes they do! Well, some of them don't and they tried to change things for the better</a> but their efforts failed. So here we are, with a build system used by tens of millions across the globe built on trust and convention. Which is to say, an optimistic way of viewing the modern world of computing. In lieu of conclusion</h1> Friends don't let friends use Maven. Roundup for two weeks ending June 16th 2024-06-16T00:00:00+00:00 All in all it's been two weeks with limited amount of time for personal "distractions" in the form of projects and coding projects, hence not a lot to write about. Tinkering with Home Assistant</h4> Finally decided to buy a bunch of cameras (namely a Reolink P330 and TP-Link Vigi) with the goal of adding DVR functionality to my NAS/Jellyfin server. Well, that's step 1, now step 2 is wiring the cameras, which I am not looking forward to and then setting up the actual DVR (with Frigate). Additionally, got my NAS to work with the smart plug, to automate turning it on. Yes, getting Wake-on-LAN was too hard so instead I've plugged in the NAS into a smart plug and turned on "power on after power loss" in BIOS and voila, I can now remotely turn it on by simply cycling the plug off and on again. Turning off is still SSH driven unfortunately, but nothing is preventing me from setting up some clever automation. Roundup for week ending June 2nd 2024-06-02T00:00:00+00:00 GitLab CI tags</h4> So last week I've mentioned how I broke my website</a>. This week I've decided to fix it for good by making sure that my cloud things don't run potentially cloud-destroying things. First question I wanted to answer was "can I exclude runners" with tags in GitLab CI? The answer</a> is a resounding "no". After crying for a bit, the solution is: Terraform Deploy: stage: deploy extends: ".terraform:deploy" tags: - linux - amd64 - selfhosted</code></pre> The key part here is simply tagging all of my "safe" runners as selfhosted</code>. Note that I have to still specify linux</code> tag because there's not really a decent way of doing inheritance in GitLab CI and amd64</code> because I am using 1Password service account for populating secrets for Terraform and 1Password doesn't support AArch64 on Alpine (sad trombone noises). Mise: first stumbling block</h4> Got bitten by a series of weird interop bugs between Poetry, Pyenv and mise and discovered a disappointing bit of comment</a> from Mise's creator. Now, in general I would say it doesn't matter because we can't know all the things, but it is a bit of my workflow that I need to work 100% of the time, so I might need to start looking into picking up this plugin if I can't resolve the issues soon. Windows and Python</h4> Ok this is a tangent, but in my attempts to "just generate some Playwright code</a>" I thought I'd install Playwright on Windows and run the code generator. I thought I'd use choco</a> but lo and behold it doesn't actually package playwright. "Ok", I thought, "I'll just use pipx, how hard can that be". ... 2 hours later and not a minute earlier I had successfully launched pipx run playwright codegen example.com</code>. The pain is still tangible. Homelabbing</h4> Trying to get my old box with i5-6500 and 32G of RAM to turn into my NAS + "Linux ISO download" center</a> is taking shape buuuuut the poor Ethernet controller on the motherboard is dying leaving the box running but not network accessible after unpredictable amount of time (it can be 5 minutes, it can be 2 days, but it will happen). So, PCI network adapter it is! But also a new 2.5G switch to pair with it. And you know what, since I'm splurging, it is time to finally set up some IP cameras in this house of mine (on the outside of it, I don't need cameras inside). Yes, this will be glorious. Roundup for week ending May 26th 2024-05-26T00:00:00+00:00 Kubernetes + AWS NLB</h4> Diving into AWS NLB and how it works with Kubernetes and omfg I'm so happy I only have to touch this occasionally. What a fucking mess it is</a>. What it boils down to in our case is the intersection of two things: nginx-ingress' propensity to reloading configurations</a></li> AWS NLB configured in [default] instance mode instead of more modern IP mode</a></li> </ul> The two working together means we have spikes in 5xx class of errors whenever our (very spot instance oriented</a>) cluster has to drain a node that is being interrupted by AWS. Which depending on traffic shape and the phase of the moon, can eat into substantial amount of our error budget. Anyway, my team doesn't have control over the NLB or cluster-wide modifications, so we're just trying to get the Kubernetes team to notice our wails and give us the tools to fix it ourselves, since they clearly indicated neither switching NLB config nor ingress will happen within observable future i.e. before Kubernetes is outdated and abandoned in favour of Some Future Thing. Terraform and uptime</h4> Ok so this one is pretty obvious, but you should avoid running a terraform apply</code> on a VPS instance if the plan is going to recreate said instance. So, don't be me and don't bring down your own website for 12 hours without noticing (because of course I don't have monitoring, I am running this website for myself, not you). This website is hosted on Hetzner and majority of the infra management is done with Terraform through a private Gitlab repo. I also have a number of self-hosted devices that all run gitlab-runner, so it was a matter of time until an event like the one described would occur and yet I was still surprised. The way I found out was seeing a notification from Gitlab informing me of a failed pipeline run in main</code>. It does make me think it's time to setup some kind of monitoring for my infra, but that is Yet Another Project on my List Of Things I Never Have Time For so I think it's ok as is. One-liners</h4> I'm continuing my journey with mise</code> and slowly getting to the point where things just work. Which is pleasant. Running Playwright on my late 2015 MBP-turned-Ubuntu laptop turned out to be harder than initially imagined, since Playwright expects only Ubuntu LTS versions. Cue sad trombone noises. Wrote my first CloudFormation template in ages, hated it. At least it's not exclusively JSON nowadays, so yay? Discovered nice and clean .zshrc leveraging zinit</a>. Got me thinking... Roundup for week ending May 19th (plus one extra) 2024-05-21T00:00:00+00:00 PyGrunn</h4> I gave a talk about being a Python developer on a platform engineering team catering to Python developers. Preparation</h5> I rewrote the presentation completely twice. So uhhh yeah, if you are reading this, don't do this. Especially if it's you, Andrii. STOP. GO TO BED. NO, THE SLIDES ARE FINE. Conference</h5> Seeing as preparation took a lot of effort and cost me some nerves, the presentation itself went good, with only a single fumbled slide. I even managed to do time tracking really well, which I think is mostly due to the presenter mode in slidev</a>. Powerful stuff that. My talk was in the first slot, which also meant I could simply enjoy the rest of the conference. I thoroughly enjoyed two talks about LLMs: "LLM Benchmark System with LangChain" by Laurens Weijs</a> in which he described how to measure bias in LLMs and henceforth benchmark it for use in Dutch government projects. </li> "When LLMs work and when they don't" by Arjan Egges</a> covered a simple use case (generating a crossword puzzle) where LLMs are great for some parts but are absolutely terrible for other. In the end he also showed different styles of organizing code, which brought me back to Clojure nostalgia</a>. Oh Swiss Arrows</a>, how I miss you... Ok not really, they look neat but if you don't touch code using them for a month you are in for a rough time. </li> </ul> Aside from that, the closing keynote by Sybren Stüvel</a> about Blender and Python was quite a pleasant tour of the common struggles of Python applications (not the web kind, the desktop kind). Packaging, 3rd party code, extending and much more. Recommend a watch, whenever the video are up. Aftermath</h5> Conversations afterwards left me with lots of thoughts. Here are they in all their disorganized glory: It seems that KPN is not on Python employment radar, despite having a sizeable (300+) number of Python engineers</li> Replaying cookiecutter templates is silly (but also not silly). I generally agree, but it's so hard to move people from established practices, even if they are bad. I am not sure how I can convince literal hundreds of people to break the bad habit. One person at a time I suppose?</li> It is not clear when a platform team should exist. I am of strong belief a platform role should exist in any team with more than 2 people, but there's no clear cut off point.</li> FastAPI doesn't seem to be a popular as I imagined it to be. Maybe it's just a local bias, but Django is still very much king when it comes to Python teams working on web projects.</li> </ul> mise</h4> As mentioned two weeks ago I dipped my toes into mise (formerly rtx, based on asdf). These past two weeks I've managed to put a little bit more time into it and I have to say I like the interface. There isn't much beyond that really, as I'm slowly getting my .envrc</code>s replaced by mise.toml</code>s I guess I'll run into some interesting edge cases. Hades</h4> Ok this is not at all a technical topic, but boy am I having fun with Hades. In fact over the past two weeks I've put in a whopping 35 hours of play. Which quite frankly is not a positive boon to my less gamepad-driven projects (see above). Roundup for week ending May 5th 2024-05-05T00:00:00+00:00 Viruses be damned</h4> I've been in the grips of a cold this whole week, hence not a lot of time and energy to dedicate to, well, anything really. OpenSSH server on Windows</h4> That said, I had a very ambitious goal of setting up OpenSSH server on my WSL2 box, to make admin tasks somewhat more manageable. Little did I know how perilous this would end up being. I've started by following Scott Hanselman's advice</a> and things worked! Well, at least ssh'ing into my WSL2 box worked, but that is a start. Next on my list was "let's see if I can make it use pubkey authentication". This turned out to be less doable. After populating all variations of authorized_keys</code> file across a number of filesystems, I've accepted defeat in this battle and moved on to the next goal -- ssh into Windows box. Fortunately, there's a very clean blog post by Kartikey Chauhan</a> with a clear set of steps required to achieve my goal. This (unsurprisingly) worked, but the pubkey auth still eluded me. Disabling password authentication would leave me without ability to login, as my key would not be accepted no matter what. Desperate, I've turned to life of crime</del> prompt engineering</del> scouring the interwebs. First helpful tip was to run sshd in debug mode</a>. This did make it clear to me that the key served by ssh client was plain not matched with any authorized key known by ssh server. I almost lost hope until I came across this comment</a> on StackOverflow which made it all work. In hindsight this was quite obviously documented by Win32-OpenSSH maintainers</a>, but who reads documentation anyway? Anyway, now I know how to do it. Frankenstein's Ansible</h4> I've managed to merge all my scattered playbooks into a single Frankenstein's monster</a> but it is quite a bit far removed from ideal state. I think this at least gives me ability to iterate further without losing track of what I'm doing after every evening hacking session. Along the way I've discovered ansible-lint</a> which I would put into category of "mildly useful". Mise/asdf</h4> I've been a user of direnv for about two years and have really appreciated how it fit my workflow, where I have to jump between Python, Terraform, Helm, Golang and Lua repos dozens of times per day but lately I've been itching for something that would ensure my tools don't break after I ran brew update</code>. You know, like it did 3 weeks ago with Poetry. Anyway, I'm looking at Mise</a> right now and it is quite interesting. It doesn't have the inherent complexity of Nix and seems to be just complex enough to be useful while being simple enough to be understood (yes, my tiny little brain is incapable of understanding Nix, flakes, NixOS and the rest of that). So far my progress is "installed mise" and "skimmed the guide" but I will continue on this thread, I promise. Roundup for week ending April 28th 2024-04-28T00:00:00+00:00 This is a roundup of notable things I did or read that are worth mentioning. Talk proposal: PyGrunn accepted, EuroPython rejected</h4> I've been on hiatus from public speaking since roughly 2015, when a combination of personal and work events derailed my confidence to the point where attending a meetup took unreasonable amount of effort. 9 years since the last time I've given a talk outside of work environment feels like a really long time so I hope I can tell a compelling story. It did take a tremendous amount of effort to squeeze out measly 20 lines required for a proposal (abstract + outline + speaker bio), so I really hope anxiety will not take the best of me when preparing for PyGrunn. Tried rye</h4> I tried Rye</a> as part of my ongoing attempt to automate as many things-without-public-API with Playwright</a>. Wait, that's missing a couple of logic steps in there. I am trying to automate expense reporting and categorizing for my ever growing number of financial accounts (e.g. banks, mortgage providers, investment platforms, etc) and have a small TUI tool written in Rust (which itself leverages Ratatui</a>). Nothing too fancy except of course there's the tedious task of collecting the "input data" from all those sources and it is a major pain in the ass as it requires a lot of precise mouse-driven navigation. To remedy that I have decided to automate this process with Playwright. I've previously used Playwright to automate filing timesheets and expense forms at work because the user experience was that dire, but I've used Poetry instead. So this time I've tried Rye. And... it's nothing spectacular and just works. It's a boring tool and boring tools are good. Poetry bug</h4> Speaking of packaging tools, ran into a spectacularly hard to debug issue with Poetry. I think it deserves a separate blog post in itself, but in essence it was a combination of these (one</a>, two</a>) bugs plus a red herring in the form of our work instance of Artifactory getting a major version bump. It didn't help that users reporting bugs to our team had wildly different symptoms and system setups. In all honesty, these kinds of issues become incredibly hard to debug as there are so many moving parts and narrowing down the culprit is a like chasing a piece of paper in a storm -- technically possible but highly unlikely to succeed. That said, Poetry is still the tool of choice at work, I don't think we've accumulated enough grievances to move away from it. Coming to grips with Ansible</h4> I've finally come to the point where I have enough devices1</a> to justify having a proper Ansible setup. This means having a playbook, proper inventory and mayhaps even some roles. Seeing as I only get an hour or so in a day to actually wrap my head around all of these concepts, the progress has been slow. To be quite frank, I'd prefer something along the lines of Terraform</del> OpenTofu, something a touch more declarative which also tracks the state of the target but alas, this may never be unless there's some way providers can leverage SSH connections. On utilization (of people)</h4> Interesting article</a> I came across scrolling through fediverse timeline. While old it seems quite relevant to how "improving resource utilization" is becoming the go-to mode of operation for senior management in larger companies. I just hope it's a temporary trend. A total of 5, where 2 are server-y (old PC posing as a NAS and OrangePi for computational things) and 3 are workstation-y (PC, Macbook Pro from work, 11-year old Macbook Pro running Ubuntu) ↩</a> </li> </ol> </section>

Instead of conclusion</h1>
I don't think you should go to KubeCon. Too many shovel peddlers, too little gold.</p>

`In lieu of conclusion</h1> Friends don't let friends use Maven.</p>`

`Hades</h4> Ok this is not at all</em> a technical topic, but boy am I having fun with Hades. In fact over the past two weeks I've put in a whopping 35 hours of play. Which quite frankly is not a positive boon to my less gamepad-driven projects (see above).</p>`

mishok13's web presence

The road to zero CVEs is littered with bodies of developers

Just Let It Run All

Using 1Password CLI with Magit Forge

KubeCon 2025: Recap

Lockfiles and why JVM languages don't use them

Roundup for two weeks ending June 16th

Roundup for week ending June 2nd

Roundup for week ending May 26th

Roundup for week ending May 19th (plus one extra)

PyGrunn</h4>
I gave a talk about being a Python developer on a platform engineering team catering to Python developers.</p>

Preparation</h5>
I rewrote the presentation completely</em> twice</strong>. So uhhh yeah, if you are reading this, don't do this. Especially if it's you, Andrii. STOP. GO TO BED. NO, THE SLIDES ARE FINE.</p>

Roundup for week ending May 5th

Viruses be damned</h4>
I've been in the grips of a cold this whole week, hence not a lot of time and energy to dedicate to, well, anything really.</p>

Roundup for week ending April 28th

mishok13's web presence

The road to zero CVEs is littered with bodies of developers

Just Let It Run All

Using 1Password CLI with Magit Forge

KubeCon 2025: Recap

Instead of conclusion</h1> I don't think you should go to KubeCon. Too many shovel peddlers, too little gold.</p>

Lockfiles and why JVM languages don't use them

Does a package manager need a lockfile?</h1> No.</p> Ehm..?!</h1> Ok, the answer is "no, unless". More specifically, unless at least one of these is true:</p>

Ehm..?!</h1> Ok, the answer is "no, unless". More specifically, unless at least one of these is true:</p>

What are even lockfiles</h1> Lockfiles at its core attempt to solve two fundamental issues with packaging software:</p>

Roundup for two weeks ending June 16th

Roundup for week ending June 2nd

Roundup for week ending May 26th

Roundup for week ending May 19th (plus one extra)

PyGrunn</h4> I gave a talk about being a Python developer on a platform engineering team catering to Python developers.</p>

Preparation</h5> I rewrote the presentation completely</em> twice</strong>. So uhhh yeah, if you are reading this, don't do this. Especially if it's you, Andrii. STOP. GO TO BED. NO, THE SLIDES ARE FINE.</p>

Hades</h4> Ok this is not at all</em> a technical topic, but boy am I having fun with Hades. In fact over the past two weeks I've put in a whopping 35 hours of play. Which quite frankly is not a positive boon to my less gamepad-driven projects (see above).</p>

Roundup for week ending May 5th

Viruses be damned</h4> I've been in the grips of a cold this whole week, hence not a lot of time and energy to dedicate to, well, anything really.</p>

Roundup for week ending April 28th

Instead of conclusion</h1>
I don't think you should go to KubeCon. Too many shovel peddlers, too little gold.</p>

PyGrunn</h4>
I gave a talk about being a Python developer on a platform engineering team catering to Python developers.</p>

Preparation</h5>
I rewrote the presentation completely</em> twice</strong>. So uhhh yeah, if you are reading this, don't do this. Especially if it's you, Andrii. STOP. GO TO BED. NO, THE SLIDES ARE FINE.</p>

`Hades</h4> Ok this is not at all</em> a technical topic, but boy am I having fun with Hades. In fact over the past two weeks I've put in a whopping 35 hours of play. Which quite frankly is not a positive boon to my less gamepad-driven projects (see above).</p>`

Viruses be damned</h4>
I've been in the grips of a cold this whole week, hence not a lot of time and energy to dedicate to, well, anything really.</p>