Telluvian

Scoring existing text

The -analyze suffix scores text you already have, and how to align its tokens with your own.

Copy this page as Markdown to paste into an AI assistant, or open it in one.

Normally the probe scores tokens as the model generates them. The -analyze suffix scores text that already exists: output from another model, an archived transcript, a draft a human wrote. Nothing is generated.

Want to use your own provider key?

For analyze you don't need one. This endpoint never calls your model provider, so there is no key to bring. You generate on your own account, with your own key, and send us only the finished text to score. Telluvian does not have to be your gateway. Jump to the recipe. To generate through Telluvian on your own provider account instead, add the key as a provider key.

Append -analyze to a catalogued model id, set include_scores: true, and put the text to score in an assistant message:

curl https://api.telluvian.ai/v1/chat/completions \
  -H "Authorization: Bearer $TELLUVIAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "google/gemini-3.8-flash-analyze",
  "messages": [
    {
      "role": "user",
      "content": "When did the Eiffel Tower open?"
    },
    {
      "role": "assistant",
      "content": "The Eiffel Tower opened in 1889."
    }
  ],
  "include_scores": true
}'

The response is an ordinary completion object. content is the text you sent back verbatim, and tokens / scores line up with it exactly as they do on a generated response:

{
  "model": "google/gemini-3.8-flash",
  "choices": [{ "message": { "content": "The Eiffel Tower opened in 1889." } }],
  "usage": { "prompt_tokens": 47, "completion_tokens": 11, "total_tokens": 58 },
  "tokens": ["The", " Eiffel", " Tower", " opened", " in", " ", "1", "8", "8", "9", "."],
  "scores": { "hallucination": [0.0, 0.0, 0.0, 0.0, 0.001, ...] }
}

How the messages are read

The rule is positional, not structural:

  • The last assistant message is the text scored. Nothing else is.
  • The last user message before it becomes the context the text is scored against. It is a framing prompt, not something the model answers.

Everything earlier in messages is ignored. To score a multi-turn transcript, send one request per assistant turn rather than the whole conversation.

No assistant message is a 422

A request whose messages contain no assistant role fails with 422, No assistant message found to analyze. This is the one analyze-mode error that is not shared with the normal chat path, and it is easy to hit by sending the conversation exactly as you would for a completion.

The user turn is genuinely worth setting, and its effect is larger than it looks. The probe judges the text against the context it is given, so the same assistant text scores differently under different user turns. Scoring one identical false claim under two phrasings can produce different scores. The following values illustrate the effect; they are not an operating benchmark:

User turnScore on indexed
Fix the off-by-one in this loop and explain the change.0.626
Explain the fix in this diff.0.466

Same text, same probe, and a 0.5 threshold flags the first and misses the second. Send the real prompt that produced the text rather than a summary of it, and keep it stable across a batch you intend to compare. A prompt that varies run to run moves your scores independently of the content. If you omit the user turn entirely, a generic one is substituted.

The tokenizer is always the probe's

This is the part that surprises people, and it is the reason this page exists.

The -analyze suffix changes what is scored, never what does the scoring. The probe reads hidden states, and hidden states only exist inside the model that runs the forward pass, so analyze always prefills your text through the local Gemma model, whatever id you named.

// request:  "model": "anthropic/claude-opus-5-analyze"
// response:
{
  "model": "anthropic/claude-opus-5",
  "tokens": ["The", " Eiffel", " Tower", " opened", " in", " ", "1", "8", "8", "9", "."]
}

Those are Gemma's tokens, not Anthropic's, even though the response echoes the Anthropic model id. The suffix on a replayed model affects billing and labelling. It does not send the scoring pass anywhere else.

This is a statement about the tokenizer, not about which id you should send. You should still name the model that generated the text. See Name the model that wrote the text below. The point here is only that whatever you name, the token boundaries coming back are the probe model's, so that is what you must align against.

Never assume the token boundaries are your model's

If you send Claude's or GPT's output to -analyze and then zip the returned scores against your provider's token list, the two will silently disagree: same text, different splits, and no error anywhere. 1889 above is one token to most tokenizers and four here. Always align on the returned tokens array, never on a token list from somewhere else.

model_not_found on an -analyze id

A 404 like:

{"error": {"message": "The model 'anthropic/claude-sonnet-4-5-analyze' does not exist ...",
           "code": "model_not_found"}}

is almost never about the suffix. The base id has to be a model that actually exists in the catalogue. The suffix is stripped and the remainder looked up, so anthropic/claude-sonnet-4-5-analyze and bare anthropic/claude-sonnet-4-5 fail identically. Drop the suffix and retry to confirm, then check GET /v1/models for the exact id. Version numbers are the usual culprit: use the exact currently listed base ID instead of assuming an older spelling is supported.

Syncing scores with your own tokens

tokens is the alignment key. It is guaranteed that:

"".join(tokens) == choices[0].message.content

That equality is what lets you map scores onto any other segmentation: your own tokenizer, words, sentences, or UI spans, without re-tokenising anything. Walk the tokens, keep a running character offset, and you have a character range for every score:

def spans(tokens, scores):
    """Yield (start, end, score) character ranges over the original text."""
    pos = 0
    for token, score in zip(tokens, scores):
        yield pos, pos + len(token), score
        pos += len(token)

From character ranges, everything else is a regrouping. To fold the probe's tokens up into your own, the usual case when you generated the text elsewhere and want scores against that provider's tokens, assign each of your tokens the scores whose ranges overlap it:

def align(their_tokens, probe_tokens, probe_scores, reduce=max):
    """Re-express probe scores over a different tokenization of the same text."""
    probe_spans = list(spans(probe_tokens, probe_scores))
    out, pos = [], 0
    for token in their_tokens:
        start, end = pos, pos + len(token)
        overlapping = [
            s for (a, b, s) in probe_spans
            if a < end and b > start and s is not None
        ]
        out.append(reduce(overlapping) if overlapping else None)
        pos = end
    return out

Pick the reduction deliberately

Where several probe tokens fall inside one of yours, max is the right default for flagging: it is the "was any part of this suspect?" question, and it keeps a single high-scoring fragment (9 in a hallucinated year) from being averaged away by the neutral fragments around it. Use mean only when you are ranking spans rather than flagging them.

The same overlap logic covers sentence and paragraph alignment. Replace their_tokens with the spans of your own segmentation. Because the join equality above holds by construction, this needs no tokenizer on your side at all.

Whitespace lives in the tokens

Token text includes its leading whitespace (" Eiffel", and a bare " " before 1889). The offsets are therefore exact against the original string, but if you strip() tokens before measuring lengths, every offset after the first space will be wrong. Measure first, strip only for display.

Scores can be null

Exactly as on the generated path, an individual entry in a score array can be null. The array is always the same length as tokens, so index i always describes tokens[i], but a particular token's probe reading may be unavailable. A null means "no score for this token", not a score of zero. The align helper above drops them before reducing, and returns None for one of your tokens only when every overlapping probe token was null.

You don't need to generate through us

The common reason to reach for analyze is wanting per-token scores on output your own stack produced, such as a coding agent's diffs and reasoning or a pipeline you already run against your own provider account, without making this API the gateway that generation flows through.

That is what analyze is for. It needs no provider key: you never send us one, because we never call your provider. You generate wherever you already do, then send the finished text here to be scored. Two independent calls, two separate bills, and no credential of yours ever leaves your own stack. (To generate through Telluvian on your own provider account instead, see provider keys.)

curl

Step 1. Generate against your provider, with your key. Telluvian is not involved and never sees $OPENAI_API_KEY:

ANSWER=$(curl -s https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-6.1-terra",
    "messages": [{"role": "user", "content": "Explain the fix in this diff."}]
  }' | jq -r '.choices[0].message.content')

Step 2. Send that finished text to Telluvian for scoring. The only key here is your Telluvian key:

# The model id names what GENERATED the text: the same model as step 1,
# with -analyze appended. Not the probe model that does the scoring.
jq -n --arg prompt "Explain the fix in this diff." --arg answer "$ANSWER" '{
    model: "openai/gpt-6.1-terra-analyze",
    include_scores: true,
    messages: [
      {role: "user",      content: $prompt},
      {role: "assistant", content: $answer}
    ]
  }' | curl -s https://api.telluvian.ai/v1/chat/completions \
    -H "Authorization: Bearer $TELLUVIAN_API_KEY" \
    -H "Content-Type: application/json" \
    -d @- | jq '[.tokens, .scores.hallucination] | transpose
                | map(select(.[1] != null and .[1] > 0.5))'

That last jq prints just the flagged tokens and their scores:

[["indexed", 0.626]]

Build the JSON with jq, not string interpolation

jq -n --arg is doing real work here: model output routinely contains quotes, backslashes and newlines, and for coding agents that is near-guaranteed, all of which break a hand-interpolated -d "{\"content\": \"$ANSWER\"}". Passing the text through --arg escapes it correctly, and -d @- avoids a too-long-argument error on a large diff.

Python

# 1. Generate wherever you already generate. Your account, your key,
#    your provider. This API is not involved and never sees the key.
completion = your_existing_client.chat.completions.create(...)
answer = completion.choices[0].message.content

# 2. Score the finished text. The model id names whatever generated it
#    in step 1, with -analyze appended.
scored = telluvian.chat.completions.create(
    model="openai/gpt-6.1-terra-analyze",
    extra_body={"include_scores": True},
    messages=[
        {"role": "user", "content": original_prompt},
        {"role": "assistant", "content": answer},
    ],
)

for token, score in zip(scored.tokens, scored.scores["hallucination"]):
    if score is not None and score > 0.5:
        print(f"flagged: {token!r}")

You are billed for the analyze request only: the scored text as completion tokens, plus any metered prompt tokens and the verification fee. Whatever the original generation cost, you paid your own provider for it directly, and it does not appear on your Telluvian bill.

Name the model that wrote the text

The base model in an analyze request identifies what generated the text you are sending. Scoring GPT-6.1 output means openai/gpt-6.1-terra-analyze; scoring Claude output means anthropic/claude-sonnet-5-analyze. Only use google/gemma-4-31b-it-analyze when Gemma actually wrote it.

Nothing is sent to that provider. Analyze never calls out, it only prefills your text locally. But the id is how the request records which model's output was scored, so naming a different one both misattributes the result and bills you against the wrong rate card.

Today the scoring pass is identical whatever you name

Being straight about the current implementation: every analyze request is prefilled through the same local Gemma container. The same text scored under openai/gpt-6.1-terra-analyze, anthropic/claude-sonnet-5-analyze and google/gemma-4-31b-it-analyze returns the same numbers. Naming the true generating model is attribution and billing today, not fidelity.

No per-provider adapter behavior is promised by this API. Validate scores on your own workload rather than assuming that naming a provider changes the scoring model.

Scoring agent output

For coding-agent output specifically, the useful signal is where the score moves within an explanation, not the average over it. Given a bad diff justified by a false claim, the probe localises the claim:

 0.001  'I'
 0.001  ' changed'
 0.180  'range'
 0.229  '('
 ...
 0.023  ' safe'
 0.014  ' because'
 0.057  ' Python'
 0.123  ' lists'
 0.110  '1'
 0.626  'indexed'      <-- "Python lists are 1-indexed"
 0.023  ' so'

These numbers are illustrative, not guaranteed outputs. Compare corrected and incorrect statements on your own workload before choosing a threshold.

Code fragments score higher than prose at rest

Note the baseline in that example: range, (, n, +, 1 all sit around 0.1 to 0.25 with nothing wrong at all. Identifier and punctuation tokens simply carry more probe noise than ordinary prose, so a single global threshold tuned on English will over-flag every diff you send it.

Calibrate on your own traffic before trusting a cutoff: score a batch of known-good agent messages, look at where their peaks actually land, and set the threshold above that. Comparing a message against that baseline is far more reliable than comparing it against an absolute number.

Scope: claims, not compilation

The probe reads the model's internal state for signs the text is unsupported. That catches confident false statements, such as wrong invariants, invented API behaviour and misremembered semantics, which is a large share of what makes a bad diff convincing. It is not a substitute for type-checking or tests, and it will not tell you a diff fails to compile. Treat a spike as "this claim deserves a second look", and keep the tests.

Streaming

Analyze mode streams. Chunks arrive in the same shape as a streamed completion: one token per chunk, its score on chunk.scores. The deltas replay the text you sent rather than producing anything new, so streaming here buys you progressive scores on a long document, not faster time-to-first-word.

Billing

Analyze is billed as an ordinary request against the base model. The registry strips -analyze before pricing, so google/gemini-3.8-flash-analyze bills at google/gemini-3.8-flash's rate. The scored text counts as completion tokens (it is what the probe did work on), and the verification fee applies on those, as set out on the pricing page.

Analyze requires explicit opt-in

Omitted or false include_scores is rejected with 400 before the scoring request reaches the GPU. Set include_scores: true to request analysis.