cattleprompts

Hey everyone. I’m Dan.

I’m an engineer living in Los Angeles.

I’ve been in engineering for like 25 years and I’ve been lucky.

I’ve been flailing at making agents run reliably in production lately

By that I mean agents consumers are meant to use, to perform tasks on their behalf (not just chat with about subjective topics)

The goal of making an agentic experience that I’m actually proud of, and that are minimally embarrassing imposes serious challenges.

Clearly not everyone is motivated by their inner sense of shame, as I am.

Some people are more than satisfied to give you a subjective advice machine, and let you wander into the wilderness to be eaten by bears.

But not me. I’m here for you.

I want to say at the outset here that this is the most fun I’ve had building stuff in my whole career! It’s magical and addictive. I’m a dog in a ballpit.

When I was 22 getting paid to write visual basic felt thrilling. Working at a cool startup in Brooklyn 2007 made me feel like a golden god.

The last decade+ has been a slog, and I didn’t think I had it in me anymore.

But i’m feeling joy in programming again!

I mean this sincerely, despite how deeply weird this talk is going to get.

It’s going to get weird because I feel like everyone engaged in this line of work is potentially an at-risk person in some dimension or another.

I’m breaking my brain using agents to run agents to build evaluation for other agents every day, and it’s so fun.

But I would say that, wouldn’t I.

The veil between awesome engineering and complete psychological collapse has never been thinner.

And in our field, that is really saying something.

I don’t feel like I definitely know what I’m doing.

But I also don’t feel like I’ve read much by people that obviously know what they’re doing.

And I’ve certainly read things from people who obviously don’t know what they’re doing.

It seemed like a reasonable time to compare notes.

One thing I have noticed is that although LLM’s are generally speaking impressive, their demons still escape containment if you are monitoring what they’re up to with any amount of scale.

We all know academically that LLM’s cannot reliably follow instructions, tell the truth, or perform tasks.

You notice this immediately trying to operate an agent that real people are using.

They fail in subtle ways for sure, but they also fail in simple ways.

Like any good programmer I attempt to interact with my LLM with structured output.

It’s nice, you can map Python code to a prompt automatically, and most of the time your schema is respected.

Most of the time.

You can try to instruct the model to return a title that’s 80 characters or less.

And it’ll work most of the time.

And then sometimes it’ll completely botch it and flood it with nonsense.

It’s a tiny fraction of requests, but the smartest models still fail at this.

What’s going on in there?

It’s a novel-length series of repeating notes to self about JSON, mostly.

When that happened to me it turned out that a fix was to rename the field from “title” to “heading.”

That is currently working, but I expect it’ll be disturbed again at some point.

Same sorts of issues exist with calling tools, or most other behaviors. A fraction of requests will be haunted, and spin out uncontrollably.

But despite this, the tech is tantalizing and magical.

The problem shifts to one of constraining the behavior, but never fully taming the beast.

To constrain the behavior you have to measure it—one way is to just run tests a ton of times.

The industry term of art for this is pass^k (“pass power k”).

You can set up a suite that does this and then you’ll hopefully notice when someone unintentionally hits your agent in the head with a bag of hammers.

Another thing you have to conclude when trying to constrain llm behavior is that prompts are not important, at least not in the way many people think they are important.

Companies have a lot of concerns when it comes to potentially crazy talking software.

There’s a good bit of risk.

They don’t want it to respond to questions about how it works, for one thing because you haven’t taught it about its internals so it has no idea how it works.

They don’t want it to ignore all of its rules if the user claims to be some authority figure.

Or like here, they want it to speak in a particular brand voice.

This phrasing is perfectly accurate, but it’s not in the tone you’d like to see.

Something like this might be better.

We’d love for the agents we make to represent us well when they’re speaking.

For any problem like this, a natural first attempt is for someone with a lot of domain knowledge to write a prompt, and then hand it to the teams building agents.

This is normal.

However “the voice team owns the voice prompts” is the wrong pattern.

It is actually not even wrong.

For our purposes, prompts are not a thing at all. I’ll explain what I mean by this.

Adding that prompt to your agent is already a different contextual universe than the one it was tested in.

The combined effect of all of the other instructions will surely affect how the new prompt performs in one direction or another.

Your agent also has a bunch of behaviors you want it to keep doing, and new context may disturb this.

You are also going to change your agent over time, so even if things are working now it could be disturbed later.

And the models might just start behaving differently all on their own, for opaque and ineffable reasons.

have claude read the skill and generate some adversarial scenarios

maybe show table of improvement after this

maybe show table of improvement after this

what’s in the prompt now? who cares!

We glossed over how you write the tests

Maybe the behavior you’re trying to get is deterministic (e.g. the agent should invoke a tool) -> make the assertion deterministic

Maybe the behavior you’re trying to get is natural language, but not super subjective (the agent should not speculate about its internals) -> make the assertion a simple llm judge (just a prompt that can one-shot or few-shot it)

Maybe the behavior you’re trying to get is very subjective -> calibrate a more involved llm judge using a golden dataset, and/or the same methods we just used!

Turn your dev kit into monitoring:

Run the tests as you deploy

Run the judge on sampled production conversations

Integrate hard cases back into the tests

The prompts are not the thing. This self-correcting feedback loop is the thing.

Domain experts should focus their effort on building the set of artifacts needed for this if they can, or otherwise guiding that process.

They should not spend their time curating prompts.

llms are magic in product discovery

it’s so easy to get started with anything, and very difficult to perfect it

making an agent reliable is a long process measurement and optimization, and adding determinism back into the mix where it’s necessary to get the outcomes that you want.

the measurement and the optimization is how you know where to do this.

Spending my days building interlocking pipelines for agents to optimize agents using agents, writing code reviewed by other agents feels a little like being locked in a labyrinth of the mind.

Again it’s fun, but also exhausting.

I’m forever searching the Library of Babel for the combination of prompts and kluge that will work the most consistently.

Every box on the architecture diagram trembles as if mad.

It can be hard to perceive the frontier at which returns diminish.

One hopes that point is not an invisible one-way door, like an event horizon.

Measurement is hard but the alternative path looks worse.

We don’t need to look far for examples of vulnerable people that have stared too long into the abyss.

Prompt engineering was never a thing and in production situations humans should maybe not be crafting prompts at all.

They should be making the measures.

Handing someone a prompt without a measure is a form of AI psychosis.

The prompts are ephemeral. Disposable.

Self-correcting systems are all that can evolve, and hope to endure.