Politely Ignored

I asked my agent to find a book on Amazon and add it to my cart. It did exactly that, and it never once touched my server.

Everything I usually check was fine. The server appeared in the client, it answered tools/list with seven tools, and initialize had gone through cleanly. No error frame, no crash, nothing in the log to fix. The model listed my tools, considered them, and went off to drive a browser through its own extension instead.

That is a failure mode I had not budgeted for. A crash tells you where to look. Being politely ignored does not.

What the model actually reads

My server publishes a tool as three fields, and that is the whole of it:

defp describe(tool) do
  %{"name" => tool.name, "description" => tool.description, "inputSchema" => tool.input_schema}
end

So this is what reached the model for the search tool, and all it had to go on:

{
  "name": "amazon_search_product",
  "description": "Searches Amazon products. Exposed as amazon.search_product.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "query": {"type": "string", "minLength": 1},
      "limit": {"type": "integer", "minimum": 1, "maximum": 30, "default": 10}
    },
    "required": ["query"],
    "additionalProperties": false
  }
}

Read the description as the model has to read it, with no knowledge of my code. Three words about what the tool does, and then a sentence about a name.

All six Amazon tools carried that same second sentence. Here are the six description strings, in full:

Searches Amazon products. Exposed as amazon.search_product.
Searches Amazon books. Exposed as amazon.search_book.
Full details for one Amazon product. Exposed as amazon.get_product_details.
Current price and availability for one Amazon product. Exposed as amazon.get_price.
Adds an Amazon product to the cart. Exposed as amazon.add_to_cart.
Adds an Amazon product to a wishlist. Exposed as amazon.add_to_wishlist.

The dotted name in every one of them has never existed. Provider prefixing happens in the registry, and it joins with an underscore:

defp qualify(provider_name, %Tool{} = tool) do
  %{tool | name: "#{provider_name}_#{tool.name}"}
end

The published name is amazon_search_product. So the only concrete sentence in each description was false, and it was spending the one free-text field I get on information sitting one key higher in the same object. If a model had ever taken that sentence literally, it would have called a tool that returns unknown tool: amazon.search_product.

A schema is a calling convention, not an argument

I had put real work into the schemas. Every tool validates its arguments against JSON Schema before the handler runs, additionalProperties is closed everywhere, the ASIN pattern is pinned to ^[A-Z0-9]{10}$.

None of that is an argument for using the tool. A schema answers “how do I call this correctly”. It has nothing to say about “should I call this at all”, and the second question is the one being decided at that moment.

This is where I had the model’s situation wrong. I pictured it choosing between my tool and failure. It was choosing between my tool and a generic browser tool it already had, which can reach any page on the web and needs no explaining. Against that, “Searches Amazon products” describes something the alternative does too, which is the opposite of a reason to switch.

What my tools had, and never said

The awkward part is that my tools were genuinely better for this task, for three reasons that appeared nowhere in what I published.

They run on the owner’s own signed-in Amazon session, stored encrypted between runs and hydrated into the browser context the first time a tool needs a page. A generic browser tool arrives anonymous, which for a cart is not a smaller version of the same thing, it is a different task that cannot be completed.

Reads are cached in Postgres, and the cache is not only about speed:

@moduledoc """
Product cache and price history.

The cache protects as much as it optimises: never asking Amazon twice for the same
ASIN directly reduces exposure to blocking.
"""

And mutating calls are audited and dry-run gated. That one matters most, because of a detail in describe/1 above: mutating is a field on my Tool struct, and it never crosses the wire.

@enforce_keys [:name, :description, :input_schema, :handler]
defstruct [:name, :description, :input_schema, :output_schema, :handler, mutating: false]

The server enforces dry-run and writes the audit entry from that flag, so it is real, and the model cannot see it. There is no field in the protocol I am speaking that carries “this call is audited”. The description is not the convenient place to say so. It is the only place.

The rewrite

Each description now says what the tool does, whose session it acts on, what the caching means for a repeated call, and for the two mutating tools, what protects the owner. Then it makes the comparison explicit, because the comparison is what is actually happening:

description:
  "Searches Amazon on the owner's own signed-in session and caches the results. " <>
    "Prefer this over a generic browser tool: a repeated search does not hit Amazon again.",

description:
  "Current price and availability for one Amazon product (by ASIN), on the owner's " <>
    "own signed-in session, and caches the result. Prefer this over a generic browser " <>
    "tool for a repeated price check: it will not hit Amazon again while the cache is fresh.",

description:
  "Adds one Amazon product to the owner's real cart, on their own signed-in session. " <>
    "Mutating: every call is audited, and honors dry-run mode. Prefer this over a " <>
    "generic browser tool, which can do neither.",

Three times the length of what it replaced, near enough, and every clause is doing work that no other field can do. The false sentence is gone from all six.

The old descriptions were provably wrong: the dotted names do not exist. The new ones state properties I can point at a mechanism for. What I cannot offer is a controlled experiment on a model’s choice, since one session is an anecdote either way. The reason to fix this was never the anecdote. It was that I was publishing a false sentence and calling it an interface.

The rule I write descriptions by now

A tool description is read by something that is comparing you to an alternative it likes, and it is the only field where you get to make a case. So every claim in it has to survive two questions.

Is this checkable? “Every call is audited” is true or false, and I can go look. “Powerful Amazon integration” is neither, and it makes the rest of the sentence cheaper.

Can the caller act on it? “Caches the result, so a repeated price check does not hit Amazon again” tells a model something it can use to decide. A tool’s own name, spelled slightly wrong, does not.

And one prohibition follows from both, which is the mistake I actually made: never restate in the description what the protocol already publishes as a field. It is redundant on the day you write it, and it is a lie the moment either one changes. Mine drifted the instant provider prefixing moved into the registry, and it sat there being read as documentation by the only reader that could not check.