Python Bytes

• #436 Slow tests go last

  Topics covered in this episode: * Free-threaded Python no longer “experimental” as of Python 3.14* typed-ffmpeg pyleak * Optimizing Test Execution: Running live_server Tests Last with pytest* Extras Joke Watch on YouTube About the show Sponsored by PropelAuth: pythonbytes.fm/propelauth66 Connect with the hosts Michael: @[email protected] / @mkennedy.codes (bsky) Brian: @[email protected] / @brianokken.bsky.social Show: @[email protected] / @pythonbytes.fm (bsky) Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Monday at 10am PT. Older video versions available there too. Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to our friends of the show list, we'll never share it. Brian #1: Free-threaded Python no longer “experimental” as of Python 3.14 “PEP 779 ("Criteria for supported status for free-threaded Python") has been accepted, which means free-threaded Python is now a supported build!” - Hugo van Kemenade PEP 779 – Criteria for supported status for free-threaded Python As noted in the discussion of PEP 779, “The Steering Council (SC) approves PEP 779, with the effect of removing the “experimental” tag from the free-threaded build of Python 3.14.” We are in Phase II then. “We are confident that the project is on the right path, and we appreciate the continued dedication from everyone working to make free-threading ready for broader adoption across the Python community.” “Keep in mind that any decision to transition to Phase III, with free-threading as the default or sole build of Python is still undecided, and dependent on many factors both within CPython itself and the community. We leave that decision for the future.” How long will all this take? According to Thomas Wouters, a few years, at least: “In other words: it'll be a few years at least. It can't happen before 3.16 (because we won't have Stable ABI support until 15) and may well take longer.” Michael #2: typed-ffmpeg typed-ffmpeg offers a modern, Pythonic interface to FFmpeg, providing extensive support for complex filters with detailed typing and documentation. Inspired by ffmpeg-python, this package enhances functionality by addressing common limitations, such as lack of IDE integration and comprehensive typing, while also introducing new features like JSON serialization of filter graphs and automatic FFmpeg validation. Features : Zero Dependencies: Built purely with the Python standard library, ensuring maximum compatibility and security. User-Friendly: Simplifies the construction of filter graphs with an intuitive Pythonic interface. Comprehensive FFmpeg Filter Support: Out-of-the-box support for most FFmpeg filters, with IDE auto-completion. Integrated Documentation: In-line docstrings provide immediate reference for filter usage, reducing the need to consult external documentation. Robust Typing: Offers static and dynamic type checking, enhancing code reliability and development experience. Filter Graph Serialization: Enables saving and reloading of filter graphs in JSON format for ease of use and repeatability. Graph Visualization: Leverages graphviz for visual representation, aiding in understanding and debugging. Validation and Auto-correction: Assists in identifying and fixing errors within filter graphs. Input and Output Options Support: Provide a more comprehensive interface for input and output options, including support for additional codecs and formats. Partial Evaluation: Enhance the flexibility of filter graphs by enabling partial evaluation, allowing for modular construction and reuse. Media File Analysis: Built-in support for analyzing media files using FFmpeg's ffprobe utility, providing detailed metadata extraction with both dictionary and dataclass interfaces. Michael #3: pyleak Detect leaked asyncio tasks, threads, and event loop blocking with stack trace in Python. Inspired by goleak. Use as context managers or function dectorators When using no_task_leaks, you get detailed stack trace information showing exactly where leaked tasks are executing and where they were created. Even has great examples and a pytest plugin. Brian #4: Optimizing Test Execution: Running live_server Tests Last with pytest Tim Kamanin “When working with Django applications, it's common to have a mix of fast unit tests and slower end-to-end (E2E) tests that use pytest's live_server fixture and browser automation tools like Playwright or Selenium. ” Tim is running E2E tests last for Faster feedback from quick tests To not tie up resources early in the test suite. He did this with custom “e2e” marker Implementing a pytest_collection_modifyitems hook function to look for tests using the live_server fixture, and for them automatically add the e2e marker to those tests move those tests to the end The reason for the marker is to be able to Just run e2e tests with -m e2e Avoid running them sometimes with -m "not e2e" Cool small writeup. The technique works for any system that has some tests that are slower or resource bound based on a particular fixture or set of fixtures. Extras Brian: Is Free-Threading Our Only Option? - Interesting discussion started by Eric Snow and recommended by John Hagen Free-threaded Python on GitHub Actions - How to add FT tests to your projects, by Hugo van Kemenade Michael: New course! LLM Building Blocks in Python Talk Python Deep Dives Complete: 600K Words of Talk Python Insights .folders on Linux Write up on XDG for Python devs. They keep pulling me back - ChatGPT Pro with o3-pro Python Bytes is the #1 Python news podcast and #17 of all tech news podcasts. Python 3.13.4, 3.12.11, 3.11.13, 3.10.18 and 3.9.23 are now available Python 3.13.5 is now available! Joke: Naming is hard

  --------  

  36:43
• #435 Stop with .folders in my ~/

  Topics covered in this episode: platformdirs poethepoet - “Poe the Poet is a batteries included task runner that works well with poetry or with uv.” Python Pandas Ditches NumPy for Speedier PyArrow pointblank: Data validation made beautiful and powerful Extras Joke Watch on YouTube About the show Sponsored by us! Support our work through: Our courses at Talk Python Training The Complete pytest Course Patreon Supporters Connect with the hosts Michael: @[email protected] / @mkennedy.codes (bsky) Brian: @[email protected] / @brianokken.bsky.social Show: @[email protected] / @pythonbytes.fm (bsky) Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Monday at 10am PT. Older video versions available there too. Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to our friends of the show list, we'll never share it. Michael #1: platformdirs A small Python module for determining appropriate platform-specific dirs, e.g. a "user data dir". Why the community moved on from appdirs to platformdirs At AppDirs: Note: This project has been officially deprecated. You may want to check out pypi.org/project/platformdirs/ which is a more active fork of appdirs. Thanks to everyone who has used appdirs. Shout out to ActiveState for the time they gave their employees to work on this over the years. Better than AppDirs: Works today, works tomorrow – new Python releases sometimes change low-level APIs (win32com, pathlib, Apple sandbox rules). platformdirs tracks those changes so your code keeps running. First-class typing – no more types-appdirs stubs; editors autocomplete paths as Path objects. Richer directory set – if you need a user’s Downloads folder or a per-session runtime dir, there’s a helper for it. Cleaner internals – rewritten to use pathlib, caching, and extensive test coverage; all platforms are exercised in CI. Community stewardship – the project lives in the PyPA orbit and gets security/compatibility patches quickly. Brian #2: poethepoet - “Poe the Poet is a batteries included task runner that works well with poetry or with uv.” from Bob Belderbos Tasks are easy to define and are defined in pyproject.toml Michael #3: Python Pandas Ditches NumPy for Speedier PyArrow Pandas 3.0 will significantly boost performance by replacing NumPy with PyArrow as its default engine, enabling faster loading and reading of columnar data. Recently talked with Reuven Lerner about this on Talk Python too. In the next version, v3.0, PyArrow will be a required dependency, with pyarrow.string being the default type inferred for string data. PyArrow is 10 times faster. PyArrow offers columnar storage, which eliminates all that computational back and forth that comes with NumPy. PyArrow paves the way for running Pandas, by default, on Copy on Write mode, which improves memory and performance usage. Brian #4: pointblank: Data validation made beautiful and powerful “With its … chainable API, you can … validate your data against comprehensive quality checks …” Extras Brian: Ruff rules Ruff users, what rules are using and what are you ignoring? Python 3.14.0b2 - did we already cover this? Transferring your Mastodon account to another server, in case anyone was thinking about doing that I’m trying out Fathom Analytics for privacy friendly analytics Michael: Polars for Power Users: Transform Your Data Analysis Game Course Joke: Does your dog bite?

  --------  

  25:34
• #434 Most of OpenAI’s tech stack runs on Python

  Topics covered in this episode: Making PyPI’s test suite 81% faster People aren’t talking enough about how most of OpenAI’s tech stack runs on Python PyCon Talks on YouTube Optimizing Python Import Performance Extras Joke Watch on YouTube About the show Sponsored by Digital Ocean: pythonbytes.fm/digitalocean-gen-ai Use code DO4BYTES and get $200 in free credit Connect with the hosts Michael: @[email protected] / @mkennedy.codes (bsky) Brian: @[email protected] / @brianokken.bsky.social Show: @[email protected] / @pythonbytes.fm (bsky) Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Monday at 10am PT. Older video versions available there too. Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to our friends of the show list, we'll never share it. Brian #1: Making PyPI’s test suite 81% faster Alexis Challande The PyPI backend is a project called Warehouse It’s tested with pytest, and it’s a large project, thousands of tests. Steps for speedup Parallelizing test execution with pytest-xdist 67% time reduction --numprocesses=auto allows for using all cores DB isolation - cool example of how to config postgress to give each test worker it’s on db They used pytest-sugar to help with visualization, as xdist defaults to quite terse output Use Python 3.12’s sys.monitoring to speed up coverage instrumentation 53% time reduction Nice example of using COVERAGE_CORE=sysmon Optimize test discovery Always use testpaths Sped up collection time. 66% reduction (collection was 10% of time) Not a huge savings, but it’s 1 line of config Eliminate unnecessary imports Use python -X importtime Examine dependencies not used in testing. Their example: ddtrace A tool they use in production, but it also has a couple pytest plugins included Those plugins caused ddtrace to get imported Using -p:no ddtrace turns off the plugin bits Notes from Brian: I often get questions about if pytest is useful for large projects. Short answer: Yes! Longer answer: But you’ll probably want to speed it up I need to extend this article with a general purpose “speeding up pytest” post or series. -p:no can also be used to turn off any plugin, even builtin ones. Examples include nice to have developer focused pytest plugins that may not be necessary in CI CI reporting plugins that aren’t needed by devs running tests locally Michael #2: People aren’t talking enough about how most of OpenAI’s tech stack runs on Python Original article: Building, launching, and scaling ChatGPT Images Tech stack: The technology choices behind the product are surprisingly simple; dare I say, pragmatic! Python: most of the product’s code is written in this language. FastAPI: the Python framework used for building APIs quickly, using standard Python type hints. As the name suggests, FastAPI’s strength is that it takes less effort to create functional, production-ready APIs to be consumed by other services. C: for parts of the code that need to be highly optimized, the team uses the lower-level C programming language Temporal: used for asynchronous workflows and operations inside OpenAI. Temporal is a neat workflow solution that makes multi-step workflows reliable even when individual steps crash, without much effort by developers. It’s particularly useful for longer-running workflows like image generation at scale Michael #3: PyCon Talks on YouTube Some talks that jumped out to me: Keynote by Cory Doctorow 503 days working full-time on FOSS: lessons learned Going From Notebooks to Scalable Systems And my Talk Python conversation around it. (edited episode pending) Unlearning SQL The Most Bizarre Software Bugs in History The PyArrow revolution in Pandas And my Talk Python episode about it. What they don't tell you about building a JIT compiler for CPython And my Talk Python conversation around it (edited episode pending) Design Pressure: The Invisible Hand That Shapes Your Code Marimo: A Notebook that "Compiles" Python for Reproducibility and Reusability And my Talk Python episode about it. GPU Programming in Pure Python And my Talk Python conversation around it (edited episode pending) Scaling the Mountain: A Framework for Tackling Large-Scale Tech Debt Brian #4: Optimizing Python Import Performance Mostly pay attention to #'s 1-3 This is related to speeding up a test suite, speeding up necessary imports. Finding what’s slow Use python -X importtime <the reset of the command Ex: python -X importtime ptyest Techniques Lazy imports move slow-to-import imports into functions/methods Avoiding circular imports hopefully you’re doing that already Optimize __init__.py files Avoid unnecessary imports, heavy computations, complex logic Notes from Brian Some questions remain open for me Does module aliasing really help much? This applies to testing in a big way Test collection imports your test suite, so anything imported at the top level of a file gets imported at test collection time, even if you only are running a subset of tests using filtering like -x or -m or other filter methods. Run -X importtime on test collection. Move slow imports into fixtures, so they get imported when needed, but NOT at collection. See also: option -X in the standard docs Consider using import_profile Extras Brian: PEPs & Co. PEP is a ‘backronym”, an acronym where the words it stands for are filled in after the acronym is chosen. Barry Warsaw made this one up. There are a lot of “enhancement proposal” and “improvement proposal” acronyms now from other communities pythontest.com has a new theme More colorful. Neat search feature Now it’s excruciatingly obvious that I haven’t blogged regularly in a while I gotta get on that Code highlighting might need tweaked for dark mode Michael: git-bug Pyrefly follow up Joke: There is hope.

  --------  

  29:01
• #433 Dev in the Arena

  Topics covered in this episode: git-flight-rules Uravelling t-strings neohtop Introducing Pyrefly: A new type checker and IDE experience for Python Extras Joke Watch on YouTube About the show Sponsored by us! Support our work through: Our courses at Talk Python Training The Complete pytest Course Patreon Supporters Connect with the hosts Michael: @[email protected] / @mkennedy.codes (bsky) Brian: @[email protected] / @brianokken.bsky.social Show: @[email protected] / @pythonbytes.fm (bsky) Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Monday at 10am PT. Older video versions available there too. Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to our friends of the show list, we'll never share it. Michael #1: git-flight-rules What are "flight rules"? A guide for astronauts (now, programmers using Git) about what to do when things go wrong. Flight Rules are the hard-earned body of knowledge recorded in manuals that list, step-by-step, what to do if X occurs, and why. Essentially, they are extremely detailed, scenario-specific standard operating procedures. [...] NASA has been capturing our missteps, disasters and solutions since the early 1960s, when Mercury-era ground teams first started gathering "lessons learned" into a compendium that now lists thousands of problematic situations, from engine failure to busted hatch handles to computer glitches, and their solutions. Steps for common operations and actions I want to start a local repository What did I just commit? I want to discard specific unstaged changes Restore a deleted file Brian #2: Uravelling t-strings Brett Cannon Article walks through Evaluating the Python expression Applying specified conversions Applying format specs Using an Interpolation class to hold details of replacement fields Using Template class to hold parsed data Plus, you don’t have to have Python 3.14.0b1 to try this out. The end result is very close to an example used in PEP 750, which you do need 3.14.0b1 to try out. See also: I’ve written a pytest version, Unravelling t-strings with pytest, if you want to run all the examples with one file. Michael #3: neohtop Blazing-fast system monitoring for your desktop Features Real-time process monitoring CPU and Memory usage tracking Beautiful, modern UI with dark/light themes Advanced process search and filtering Pin important processes Process management (kill processes) Sort by any column Auto-refresh system stats Brian #4: Introducing Pyrefly: A new type checker and IDE experience for Python From Facebook / Meta Another Python type checker written in Rust Built with IDE integration in mind from the beginning Principles Performance IDE first Inference (inferring types in untyped code) Open source I mistakenly tried this on the project I support with the most horrible abuses of the dynamic nature of Python, pytest-check. It didn’t go well. But perhaps the project is ready for some refactoring. I’d like to try it soon on a more well behaved project. Extras Brian: Python: The Documentary Official Trailer Tim Hopper added Setting up testing with ptyest and uv to his “Python Developer Tooling Handbook” For a more thorough intro on pytest, check out courses.pythontest.com pocket is closing, I’m switching to Raindrop I got one question about code formatting. It’s not highlighted, but otherwise not bad. Michael: New course! Polars for Power Users: Transform Your Data Analysis Game Apache Airflow 3.0 Released Paste 5 Joke: Theodore Roosevelt’s Man in the Arena, but for programming

  --------  

  28:40
• #432 How To Fix Your Computer

  Topics covered in this episode: pre-commit: install with uv PEP 773: A Python Installation Manager for Windows (Accepted) Changes for Textual The Best Programmers I Know Extras Joke Watch on YouTube About the show Sponsored by NordLayer: pythonbytes.fm/nordlayer Connect with the hosts Michael: @[email protected] / @mkennedy.codes (bsky) Brian: @[email protected] / @brianokken.bsky.social Show: @[email protected] / @pythonbytes.fm (bsky) Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Monday at 10am PT. Older video versions available there too. Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to our friends of the show list, we'll never share it. Brian #1: pre-commit: install with uv Adam Johnson uv tool works great at keeping tools you use on lots of projects up to date quickly, why not use it for pre-commit. The extension of pre-commit-uv will use uv to create virtual environments and install packages fore pre-commit. This speeds up initial pre-commit cache creation. However, Adam is recommending this flavor of using pre-commit because it’s just plain easier to install pre-commit and dependencies than the official pre-commit install guide. Win-win. Side note: No Adam, I’m not going to pronounce uv “uhv”, I’ll stick with “you vee”, even Astral tells me I’m wrong Michael #2: PEP 773: A Python Installation Manager for Windows (Accepted) via pycoders newsletter One manager to rule them all – PyManager. PEP 773 replaces all existing Windows installers (.exe “traditional” bundle, per-version Windows Store apps, and the separate py.exe launcher) with a single MSIX app called Python Install Manager (nick-named PyManager). PyManager should be mainstream by CPython 3.15, and the traditional installer disappears no earlier than 3.16 (≈ mid-2027). Simple, predictable commands. python → launches “the best” runtime already present or auto-installs the latest CPython if none is found. py → same launcher as today plus management sub-commands: py install, py uninstall, py list, py exec, py help. Optional python3 and python3.x aliases can be enabled by adding one extra PATH entry. Michael #3: Changes for Textual Bittersweet news: the business experiment ends, but the code lives on. Textual began as a hobby project layered on top of Rich, but it has grown into a mature, “makes-the-terminal-do-the-impossible” TUI framework with an active community and standout documentation. Despite Textual’s technical success, the team couldn’t pinpoint a single pain-point big enough to sustain a business model, so the company will wind down in the coming weeks. The projects themselves aren’t going anywhere: they’re stable, battle-tested, and will continue under the stewardship of the original author and the broader community. Brian #4: The Best Programmers I Know Matthias Endler “I have met a lot of developers in my life. Lately, I asked myself: “What does it take to be one of the best? What do they all have in common?”” The list Read the reference Know your tools really well Read the error message Break down problems Don’t be afraid to get your hands dirty Always help others Write Never stop learning Status doesn’t matter Build a reputation Have patience Never blame the computer Don’t be afraid to say “I don’t know” Don’t guess Keep it simple Each topic has a short discussion. So don’t just ready the bullet points, check out the article. Extras Brian: I had a great time in Munich last week. I a talk at a company event, met with tons of people, and had a great time. The best part was connecting with people from different divisions working on similar problems. I love the idea of internal conferences to get people to self organize by topic and meet people they wouldn’t otherwise, to share ideas. Also got started working on a second book on the plane trip back. Michael: Talk Python Clips (e.g. mullet) Embrace your cloud firewall (example). Python 3.14.0 beta 1 is here Congrats to the new PSF Fellows. Cancelled faster CPython bsky.app/profile/snarky.ca post Joke: How To Fix Your Computer

  --------  

  25:48

More Technology podcasts

Trending Technology podcasts

About Python Bytes