Skip to content

Prediction Hooks

Hooks let you observe or shape every decision Laya makes, without forking it.

They are the extension seam for the things every real deployment needs: audit logging, PII redaction before inference, caching, metrics, confidence gating, routing overrides, and forwarding a decision to an external service. They are opt-in: with no hooks configured the behaviour of Agent, Router and ONNXAgent is unchanged.

This folder is the full reference. Start here, then dive into the page you need:

page what is in it
API reference every class, field, parameter and default
Lifecycle exactly when each hook runs, with flowcharts
Errors hooks_raise, on_error, exception chaining, failure matrix
Patterns and anti-patterns what to do, what to avoid, and why
Examples copy-paste recipes for every use case
Tracing run_id, span correlation, OpenTelemetry

Quick start

import laya

def log(ctx):
    print(ctx.model, ctx.results[0]["answers"], ctx.elapsed_ms)

agent = laya.load("convaiinnovations/laya", on_predict_end=log)
agent.system_one("I was charged twice.", {"urgent": {"type": "noul", "instructions": "Urgent?"}})

An object can implement any subset of the lifecycle events:

class Audit:
    def on_predict_start(self, ctx):
        print("start", ctx.run_id)

    def on_predict_end(self, ctx):
        print("end", ctx.run_id, ctx.usage, ctx.elapsed_ms)

    def on_error(self, ctx):
        print("failed", ctx.run_id, ctx.error)

laya.load("convaiinnovations/laya", hooks=[Audit()])

Hooks can also be added later or scoped to a block:

agent.add_hook(tracer)               # attach at runtime
with agent.hooks_installed(debug):   # installed for the block, removed on exit
    agent.system_one(state, questions)

See runtime registration. For a hook that should apply everywhere without threading it through every call, register it once with process-wide defaults:

from laya import hooks

hooks.set_default_hooks(hooks=[Tracer()])

The mental model

There are three ideas.

  1. A hook is a callable or an object. A plain function is convenient for one event; an object is convenient for several. Both are passed to hooks= / on_predict_start= / on_predict_end=.

  2. Every hook of one call shares one mutable PredictContext. It carries the states, questions, results, routing decision, model name, usage, timing and any error. Because it is mutable, a hook can shape the call, not only watch it: redact the state, rewrite the questions, replace the result, or skip inference with a cached answer.

  3. There are two scopes. Agent hooks wrap a forward pass; Router hooks wrap routing plus inference and can also see model lifecycle (on_route, on_load, on_evict). This mirrors the "run hooks" vs "agent hooks" split in other agent frameworks.

                             Router.predict(state, questions)
   ┌──────────────────────────────────────────────────────────────────────────┐
   │  route()                                                                 │
   │    ├─ detect language / workflow                                         │
   │    └─ on_route          ctx.decision  (a hook may replace it)            │
   │                                                                          │
   │  load(decision.model)                                                    │
   │    ├─ build checkpoint on first use ──► on_load    ctx.model, ctx.agent  │
   │    └─ evict LRU checkpoint ───────────► on_evict   ctx.model             │
   │                                                                          │
   │  on_predict_start       ctx.states, ctx.questions, ctx.decision          │
   │    │                                                                     │
   │    ├── ctx.skip(results)? ──► skip the forward pass                      │
   │    │                                                                     │
   │    └── Agent.system_one(...)  ──►  Agent-level hooks run here            │
   │           on_predict_start  ─►  forward  ─►  on_predict_end              │
   │                                                                          │
   │  result["routing"] = decision                                            │
   │  on_predict_end         ctx.results, ctx.usage, ctx.elapsed_ms           │
   └──────────────────────────────────────────────────────────────────────────┘
                 any failure on the way ──► on_error, then on_predict_end

Scope at a glance

Agent / ONNXAgent Router
on_predict_start yes yes
on_predict_end yes yes
on_error yes yes
on_route no yes
on_load no yes
on_evict no yes

laya.serve and the MCP server call Router.predict, so Router hooks fire for them automatically. Agent hooks fire whenever the Router runs an attached or built agent.

Compatibility

  • No hooks configured means no behavioural change. The unset path is regression-tested.
  • All hook parameters are keyword arguments with defaults, so existing calls keep working.
  • laya/hooks.py is pure Python: import laya does not pull in torch because of it.
  • Hooks are synchronous. Keep them fast and non-blocking; see errors and patterns for the consequences on laya.serve.

See also