Writing

Writing

Short pieces on honest evaluation and building AI for the population you actually have. Each one comes out of a project on this site, and each leads with what didn't work.

July 14, 20264 min read
When Your LLM Judge Fails Its Own Eval
I built an LLM judge to score an agent's output, checked it against human ratings before trusting it, and it failed badly enough that I cut it and kept the objective metric instead.
- evals
- llm-as-judge
- healthcare-ai
- agents
July 14, 20264 min read
Fine-Tuning Won In-Distribution and Lost Out-of-Distribution
A controlled LoRA fine-tuning experiment beat the prompted pipeline on synthetic data and lost to it, and to the untuned base model, on real notes.
- fine-tuning
- evals
- llm
- healthcare-ai
- lora
July 14, 20264 min read
What a 63.8% No-Email Cohort Taught Me About Building for Medicaid
A majority of the Medicaid cohort had no email on file, so the product had to be built SMS-first from day one instead of email-first with a fallback.
- healthcare-ai
- medicaid
- product-design
- sms