Adventures in DevOps

Share Episode

We welcome David Burns, Head of Open Source at BrowserStack and previously Mozilla, this week to help explain how can browser and mobile app testing be so bad still in 2026. "Even if we had to do it all over again, we probably still wouldn't have gotten it right", might just be the biggest learning. We review some of the core challenges that cause even the most experienced web engineers pain and how some companies are attempting to avoid them.

We also review shares lessons from Mozilla’s 'War on Orange,' demonstrating why chasing a 100% green CI suite at scale is a fool's errand, and why engineering teams must instead manage delta tolerances and adhere to a strict testing pyramid. Additionally, we investigate the physical hardware nightmares of running 30,000 real phones in data centers; from battery swelling and custom Faraday cages to complying with Apple EULA requirements using racks of Mac Minis. And finally, we discuss WebMCP, Project Fugu, and why LLM browser agents still struggle against edge-case browser behaviors like the Back-Forward Cache (BF Cache).


💡 Notable Links:
✨ Episode: Sam Goto — Security challenges in the Browser World
✨ Episode: Actually think about how are your users use your tools
Mozilla CI - Tree Herder
The Worst UX site with a perfect 100
Project Fugu Browser initiative
WebMCP Project Specification
🎯 Picks:
Warren - Paper — The Corporate Bullshit Receptivity Scale
David - Anbernic Retro Handheld Game Console

Share Episode

From your feedback, we know that on-call issues, especially those that arise at the database layer are always the most challenging to solve. This episode builds on our Grafana Observability episode as we go deeper into the challenge of data ownership in data lakes. Apache Druid creator, previous Yahoo Distinguished Engineer, Fellow at Splunk, and currently the Chief Architect as Imply, Eric Tschetter joins us to confirm that there are unsolved problems in data platforms, and it doesn't get better when we throw agents into the mix.

In the true DevOps mindset aligment, Eric reminds us that whoever is operating the system, knows what is going on, they know their service boundaries, and so it's their choice to make. And the sad reality is that at large enough companies, you will always have independent teams making independent decisions, it's never possible to fully align them, and realistically we get down to why it is even a mistake to try.

💡 Notable Links:
✨ Episode: Grafana — Observability
✨ Episode: AST can run anything
Book: Lean Manufacturing — The Toyota Way
Monkeys typing the Complete Works of Shakespeare
Paper: Ironies of Automation (& AI)
Jeff Bezos API Rant — 2011
Book: The Formula, Science Success
Show: Burn Notice
🎯 Picks:
Warren - The Agency
Eric - 継続は力なり — keizoku wa chikara nari

Share Episode

Sauce Labs CTO Anoop Tripathi addresses the root causes of broken test suites and why teams frequently become numb to intermittent test failures. Anoop shares concrete lessons from his career, detailing how bad practices like embedding fixed sleep routines instead of signal-based waiting ruin test reliability. We discuss the necessity of designing products for testability from day one—such as using pair programming where one developer writes the implementation while the other writes the test suite, or building dedicated test interfaces directly into the application architecture. We also analyze the hidden business costs of poor quality, where engineering debt forces sales teams to spend their time managing upset clients instead of acquiring new revenue.

We explore organizational anti-patterns—such as teams claiming to be 'Agile' with one-year sprint cycles, or shipping debug builds to production because release builds continuously crash, highlighting a pair-programming model where one developer writes the code while another writes the corresponding tests

Finally, we tackle the dangers of chasing vanity metrics, from engineers writing automated assert pass statements to claim fake coverage, to organizations bragging about more code. Anoop distinguishes between output gain and outcome gain, warning that LLMs are creating a massive testing and review crisis by generating 20-page documentation files for minor features.

💡 Notable Links:
Quality Iron Triange of Software
Book: Sapiens
✨ Episode: DORA Report
✨ Episode: Grafana — Managing Open Source
🎯 Picks:
Warren - Proof of Quantum Consciousness - Paper
Annop - Book: Genesis: Artificial Intelligence, Hope, and the Human Spirit

Share Episode

In this episode, we sit down with Nikhil Mungel, Head of AI R&D at Cribl, to discuss how LLMs have dramatically transformed the tech hiring landscape. The conversation opens with an honest look at the flood of AI-generated, low-quality resumes and remote interview cheating tactics, which have pushed many organizations toward in-person whiteboard interviews or closed-network referrals.

We explore how software development has evolved from character-by-character coding into "judgment work," where an engineer's value relies on holistic decision-making, taste, and business alignment rather than raw syntax memorization. Nikhil emphasizes the importance of an "ownership mindset," urging developers to move beyond acting as mere specification-translators and instead evaluate technical trade-offs through a business-focused lens. We also pull back the curtain on modern technical interviews, highlighting why expressing strong convictions, asking intentional career questions, and proving human reasoning without LLM assistance remain essential for standing out.

💡 Notable Links:
Show: Pluribus
🎯 Picks:
Warren - Sony made the walkman worse
Nikhil - Book: Leadership Strategy and Tactics: Field Manual

Share Episode

Pushkar Gopalakrishna, Senior Staff Software Engineer at Snap, previously Cruise and AWS, joins to explore why engineering organizations pivot away from Terraform and HCL toward Kubernetes-native tools even when they might not be better. We unpack how developer friction, copy-pasted control structures, and misaligned organizational incentives create massive tech debt—often forcing SREs to manually update infrastructure repositories for compliance, resulting in broken pipelines and severe operational friction.

We debate over the mechanics of Crossplane, detailing how its continuous reconciliation loop and Custom Resource Definitions (CRDs) allow teams to express cloud infrastructure as YAML alongside their application manifests at potentially the cost of async validation. Pushkar pulls back the curtain on how Cruise managed infrastructure at scale using a custom internal platform called 'Juno' to bootstrap GCP projects, repositories, and permissions, while leveraging Crossplane for application-level resources. We also dive into the dangers of using CI tools for continuous deployment, detailing a terrifying incident where a pipeline bug accidentally marked three production Kubernetes namespaces for deletion, and how moving to ArgoCD and Argo Rollouts helped prevent future outages for autonomous vehicles.

Finally, we touch on the realities of non-production environment isolation, testing against live APIs, and why platform teams must balance providing a seamless developer experience without stripping away developer accountability.

💡 Notable Links:
Crossplane
Podcast Guest Request for Principal Engineer — What work are you doing?
Amazon Multi-level fullyment center for drones
✨ Episode: Terraform vs OpenTofu
🎯 Picks:
Warren - Books: The Murderbot Diaries
Pushkar - DJI mini drone