Behind the Craft

📌 Copy the AI system I use to run a one-person $1M+ business: https://www.behindthecraft.com
🤝 Subscribe to my practical AI newsletter read by 150K+ professionals: https://creatoreconomy.so

Shreya and Hamel teach the industry standard course for AI evaluations taken by 4,500+ students from OpenAI, Google, and more. In our interview, I asked them to give live feedback on the AI evals that I built for my skills and demo how anyone can run evals with ChatGPT/Claude and their free evals skill. Don’t miss this up to date primer on evals from the real experts. 🙂

P.S., I asked Shreya and Hamel to give my audience 25% off the next cohort of their AI evals course. It has 4.7 stars with 900 reviews which is almost unheard of on Maven. Sign up via the link here: https://maven.com/parlance-labs/evals?utm_campaign=peteryang&utm_medium=affiliate&utm_source=maven&promoCode=PETERYANG

We talked about:
(00:00) Why AI evals are completely different now with the latest models
(02:30) Demo: Live audit of the evals that I built for my AI skills
(05:14) Top down vs. bottom up evals and why AI sucks at the latter
(08:16) Spinning up AI agents to grade each eval criteria
(14:32) Demo: Using Shreyas AI skill to run evals in Claude
(23:46) How to review output labels to identify failures
(34:37) How to turn failures into reusable evals
(43:19) Where human judgement is still needed

Resources mentioned:
Free eval skills from Shreya: https://github.com/ai-evals-course/evals-skills
Do automated evals work?: https://parlance-labs.com/blog/posts/auto-evals/

Thanks to our sponsors:
Wispr Flow — 4x faster than typing with your voice — https://ref.wisprflow.ai/peteryang

📌 Get the takeaways: https://creatoreconomy.so/p/how-to-build-better-ai-evals-with-claude-code-shreya-hamel

Where to find Shreya and Hamel:
Shreya: https://www.sh-reya.com/
Hamel: https://hamel.dev/

Subscribe to this channel - more interviews coming soon!

📌 Copy the AI system I use to run a one-person $1M+ business: https://www.behindthecraft.com
🤝 Subscribe to my practical AI newsletter read by 150K+ professionals: https://creatoreconomy.so

Riley is a top AI creator and founder of Agent Native. In this episode, he showed me how he uses Codex to run his entire content business from generating engaging thumbnails to writing incredible video hooks. Riley also shared his full AI stack including some lesser known AI tools to edit videos, draw diagrams, and more.

We talked about:
(00:00) How Codex runs Riley’s 1.5M+ follower content business
(03:16) Demo: Using Codex to create animations and B-roll
(07:28) Demo: Codex can reverse-engineer a top YouTube hook
(13:24) The BRENS framework to turn an idea into a strong hook
(17:24) Demo: Riley’s Codex stack for diagrams, thumbnails, and posts
(22:01) How Riley creates so much content every week
(25:41) Demo: Using Codex and Paper for viral YouTube thumbnails
(35:01) How to set up AI agents to find content ideas proactively

Thanks to our sponsors:
Google AI Studio: The fastest way to go from prompt to production https://aistudio.google.com/?utm_source=peteryang&utm_medium=partner&utm_campaign=26Q3-peteryang_ais
Riverside: All-in-one AI studio for podcasts and video https://creators.riverside.com/PeterYang
Wispr Flow: 4x faster than typing with your voice https://ref.wisprflow.ai/peteryang

📌 Get the takeaways: https://creatoreconomy.so/p/how-i-run-my-content-business-with-codex-riley-brown

Where to find Riley:
Agent Native, Riley’s new company: https://agentnative.inc/
X: https://x.com/rileybrown
Website: https://rileybrown.xyz/

Subscribe to this channel - more interviews coming soon!

📌 Copy the AI system I use to run a one-person $1M+ business: https://www.behindthecraft.com
🤝 Subscribe to my practical AI newsletter read by 150K+ professionals: https://creatoreconomy.so

Nan and Jacob built Linear Agent, which has transformed how the $1.25B company gets work done. In this episode, they took me from the initial memo to launch, showing how to give an agent tools to find the context it needs, use evals to improve reliability, and more. This episode is a must-watch if you want a concrete, behind-the-scenes example of how to build a production agent end to end.

We talked about:
(00:00) What’s holding AI agents back as models improve?
(00:51) Three things every good agent needs
(03:00) Demo: The internal memo that started Linear Agent
(09:50) The biggest mistake to avoid when building agents
(11:29) Demo: From Slack to code in 6 minutes with the agent
(17:01) When prototyping, “use the biggest model until it works”
(24:02) Give agents tools to find context, not more context
(34:07) The first thing to do before building an agent

Thanks to our sponsors:
Oceans: Hire AI-native EAs http://oceanstalent.com/peter
Riverside: All-in-one AI studio for podcasts and video https://creators.riverside.com/PeterYang
Linear: The AI agent platform for modern teams https://linear.app/behind-the-craft

📌 Get the takeaways: https://creatoreconomy.so/p/5-rules-for-building-ai-agents-in-production-linear-nan-jacob

Where to find Nan and Jacob:
Nan on LinkedIn: https://www.linkedin.com/in/thenanyu
Jacob on LinkedIn: https://www.linkedin.com/in/jacob-shumway-314a8366
Linear Agent: https://linear.app/docs/linear-agent

Subscribe to this channel - more interviews coming soon!

📌 Copy my personal AI system to save time and build with AI: https://www.behindthecraft.com/
🤝 Get my free AI newsletter read by 150K+ professionals: https://creatoreconomy.so/

Karan co-founded Nous Research, the company behind Hermes, the #1 personal agent and AI app overall on OpenRouter. I found him incredibly down to earth as we discussed why open-source AI must win, how Hermes gets better the more you use it, and even a live demo of how Hermes can mod your favorite childhood games.

Karan and I talked about:
(00:00) How Hermes differs from Claude Code and Codex
(02:20) Why every AI model now sounds exactly the same
(11:16) How to stop AI from saying “you’re absolutely right”
(15:16) How Hermes creates skills without bloating its context
(16:40) The open vs. closed-source debate on AI safety
(23:40) Using one Kanban board for humans and AI agents
(26:45) Demo: Rebuilding a childhood Sonic world with Hermes
(36:07) How two Discord friends built a self-improving agent
(42:40) How Hermes became its own biggest contributor

Thanks to our sponsors:
Riverside: All-in-one AI studio for podcasts and video https://creators.riverside.com/PeterYang
Linear: The AI agent platform for modern teams https://linear.app/behind-the-craft
Wispr Flow: 4x faster than typing with your voice https://ref.wisprflow.ai/peteryang

📌 Get the takeaways: https://creatoreconomy.so/p/hermes-co-founder-on-building-an-ai-agent-that-improves-itself-karan-malhotra

Where to find Karan:
X: https://x.com/karan4d
Website: https://www.karan4d.com/
Hermes Agent: https://github.com/NousResearch/hermes-agent

Subscribe to this channel - more interviews coming soon!

Jason is a DevEx engineer on the Codex team. In our episode, he showed me how he runs his entire workday using Codex, from his chief of staff thread to the way he builds skills that learn how he works. Jason wrote the official OpenAI guidebook on how to get the most out of Codex, so don’t miss this episode.

Jason and I talked about:

(00:00) The only job left after AI can do the work
(02:40) Turn pinned threads into long-running workspaces
(05:50) Build a chief of staff across Slack, email, and Linear
(10:12) Teach Codex to write like you for every audience
(13:33) Mine past sessions for new skills and workflows
(20:01) How Codex revised a launch video while Jason was out
(24:23) Build a drum practice site around how you learn
(25:59) Goal, plan, and worklog for long-running Codex tasks
(33:52) Why taste starts with knowing what feels wrong

Thanks to our sponsor:

Google AI Studio: The fastest way to go from prompt to production https://aistudio.google.com/?utm_source=peteryang&utm_medium=partner&utm_campaign=26Q3-peteryang_ais

Riverside: All-in-one AI studio for podcasts and video
https://creators.riverside.com/PeterYang

📌 Get my personal AI OS with useful AI skills, $270 in AI tool credits, live monthly workshops, and courses on Hermes and Codex agents: https://www.behindthecraft.com/

Where to find Jason:
X: https://x.com/jxnlco

Subscribe to this channel - more interviews coming soon!