W29: Cost breakers and on-air guardrails
The week two independent resource overruns landed at once — LLM spend in the Vanilla Travel chat and the ElevenLabs quota on Radio Viens — and forced us to install the ceilings that were never there.
- Added the first-ever LLM spend breaker to the vt-blog chat — until now answer generation had no cost ceiling at all
- Split the Radio Viens on-air TTS key from experiments, so music iterations can never knock the broadcast offline again
- biznesam.ai kept its daily AI briefings running, but the engineering weight of the week sat squarely on cost discipline across the ecosystem
- Every generative resource needs a pre-check and a hard ceiling — not a prompt, a limit that stops one iteration from draining the shared quota
- One key serving two purposes is a time bomb — separate the on-air and experiment paths physically, not with good intentions
- A breaker that blocks on failure can be more dangerous than the problem it solves — decide deliberately when to fail-closed and when to fail-open
- ElevenLabs music iterations burned the entire 140k-credit quota, and the overnight build fell over with a TTS error — nobody checked the quota first
- A fail-closed breaker killed the admin chat itself (P0 regression) — the right choice was fail-open when the spend ledger can't be read
Ecosystem overview
This was a cost-discipline week. Two independent resource overruns — LLM spend in the Vanilla Travel chat and the ElevenLabs quota on the Radio Viens station — lined up and forced the same question: where do our generative systems have cost ceilings? At the start of the week the answer was alarming — in places, none. The week went into installing them.
Vanilla Travel
The main work was the vt-blog chat spend breaker. It turned out answer generation had no cost ceiling whatsoever (P0) — the system could spend without limit. We shipped an LLM spend breaker with admin-llm-budget observability so the spend ledger can actually be read. A painful lesson came along the way: the first breaker version was fail-closed, and when it failed to read the ledger it killed Anita’s admin chat — we reworked it to fail-open.
In parallel we tightened model costs: moved admin-review and content-qc to the fast Flash model, test sessions categorically no longer get Opus (CEO directive — the last three Opus calls gone), and cut the daily LLM test volume from 19 to 8 calls. On the chat side, behavior work continued — a deterministic season-lead runtime gate and deal-timing answers that no longer open with the tourism season, plus honesty about destinations outside the portfolio.
The Vanilla Travel repo itself ran on the marketing agents’ daily rhythm — cmo-agent and paid-control daily briefings and fast-gate reports, with no large code commits.
biznesam.ai and Radio Viens
Radio Viens took the week’s hardest lesson. ElevenLabs music iterations on July 14–16 burned through the entire 140k-credit quota, and the overnight build fell over with a TTS error — balance zero. Root cause: music generation was used as a “free iteration tool” without checking the quota before the call. The response was structural — we physically separated the on-air TTS key (ELEVENLABS_RADIO_KEY) from the experiment key, so music experiments can never knock the broadcast offline again. New rule: music iterations run on a fixed subscription, ElevenLabs only for special cases with an estimate and a quota check before the call.
The same week surfaced a second on-air problem: an unapproved track had made it to air because a rotation gate was missing. We blocked it with do_not_air and regenerated the rundowns. On the biznesam.ai blog, the daily AI briefings continued.
Insights
- Every generative resource needs a ceiling, not hope. Both LLM calls and music credits are exhaustible. A pre-check and a hard limit are the only thing that stops one iteration from draining the whole quota.
- One key for two purposes eventually collapses. Separate the on-air and experiment paths physically. Good intentions (“I won’t use it on air”) are not architecture.
- A breaker can be more dangerous than the problem. A fail-closed breaker that blocks everything on failure killed the admin chat. Decide deliberately: when it’s safer to block, and when it’s safer to let through.
Mistakes and lessons
A quota nobody checked. ElevenLabs music iterations ate the entire 140k-credit quota and the overnight build fell over. A double fault: iterating without a quota check, and one key serving both broadcast and experiments with no separation. Both roots are now closed — key separated, iterations gated behind an estimate and CEO sign-off.
A fail-closed breaker that bit itself. The cost breaker’s first version, on failing to read the spend ledger, blocked everything — and took down Anita’s admin chat (P0 regression). Reworked to fail-open. Lesson: before you install a breaker, decide what’s more dangerous — letting something through, or halting everything.