API reference¶

Everything on this page is importable from laya (the common names) or laya.hooks (the whole surface).

from laya import PredictContext, PredictHook, Hook, OptionStabilityHook
from laya.hooks import HOOK_EVENTS, normalise_hooks, dispatch, aggregate_usage

• PredictContext
• Hook protocol
• OptionStabilityHook
• Convenience types
• Configuration surface
• Event payloads
• Validation
• Advanced helpers

PredictContext¶

A PredictContext is created once per public call and passed to every hook of that call. Router.predict_batch is the one exception: it creates one per request, as a Router.predict call for that request would, so its Router-level hooks fire once per request with one state each. It is mutable: hooks may rewrite states, questions and results, and on_route may rewrite decision. It uses identity equality (eq=False), so a context is hashable and two contexts are never equal.

@dataclass(eq=False)
class PredictContext:
    states: List[Any]
    questions: Dict[str, Any]
    run_id: str = <uuid4 hex>
    results: Optional[List[Dict[str, Any]]] = None
    decision: Optional[Dict[str, Any]] = None
    model: Optional[str] = None
    agent: Any = None
    router: Any = None
    max_len: Optional[int] = None
    head_max_len: Optional[int] = None
    usage: Optional[Dict[str, int]] = None
    started_at: float = <perf_counter()>
    elapsed_ms: Optional[float] = None
    error: Optional[BaseException] = None

PredictContext.skip(results)¶

Short-circuits inference. Called from on_predict_start, it sets ctx.results so the forward pass is skipped; on_predict_end still runs and the supplied results are returned.

The key has to cover everything the answer depends on, and the hook has to cover everything the call carries: hooks fire once per call, and predict_batch calls them with every state at once.

import hashlib, json

CACHE = {}

def key(ctx, index):
    # Not sort_keys=True: criteria order is positional, so two orders are two questions,
    # and the checkpoint and token budget change the answer too.
    payload = json.dumps([ctx.states[index], ctx.questions, ctx.model,
                          ctx.max_len, ctx.head_max_len], default=str)
    return hashlib.sha256(payload.encode()).hexdigest()

def cache_read(ctx):
    hits = [CACHE.get(key(ctx, i)) for i in range(len(ctx.states))]
    if all(hit is not None for hit in hits):
        ctx.skip(hits)   # one entry per state, same shape as predict_batch's return

def cache_write(ctx):
    for i, result in enumerate(ctx.results or []):
        CACHE[key(ctx, i)] = result

laya.load("convaiinnovations/laya", on_predict_start=cache_read, on_predict_end=cache_write)

tests/test_hooks_api.py execs this block, examples/hooks/cache.py, and the caching blocks of docs/hooks/patterns.md and docs/hooks/examples.md, and asserts all four keys the same way, so a page cannot teach a key the example has moved on from.

On the Router, a skipped payload gets a routing key added (without overwriting one it already has), so Router.predict keeps its documented return shape.

Hook protocol¶

Hook is a typing.Protocol. Implement any subset of the methods; the rest are skipped.

class Hook(Protocol):
    def on_predict_start(self, ctx: PredictContext) -> None: ...
    def on_predict_end(self, ctx: PredictContext) -> None: ...
    def on_route(self, ctx: PredictContext) -> None: ...
    def on_load(self, ctx: PredictContext) -> None: ...
    def on_evict(self, ctx: PredictContext) -> None: ...
    def on_error(self, ctx: PredictContext) -> None: ...

A hook is free to define extra attributes and methods; only the six event names are consulted. If a hook defines one of the six as a non-callable, configuration fails fast (see Validation).

BaseHook¶

BaseHook is the concrete counterpart to the protocol: a class with a no-op body for every event. Subclass it and override only the events you need.

from laya import BaseHook

class Audit(BaseHook):
    def on_predict_end(self, ctx):
        ship(ctx.run_id, ctx.results)

Hook is best when you want structural typing (any object with the right methods); BaseHook is best when you want an explicit base to subclass and call super() on.

OptionStabilityHook¶

OptionStabilityHook evaluates prediction reliability across meaning-preserving option permutations and letter renamings (addressing issue #635). It executes variants in the same single-forward-pass batch without multiple network calls or serial invocations.

Layout-aware (#635, #951): on checkpoints using parallel option layouts (option_layout="parallel"), options cannot attend to each other so reordering cannot move decisions by construction. The hook automatically detects layout from the agent config, executing rename-only probes on parallel checkpoints while running both rename and reorder probes on sequential checkpoints.

from laya import OptionStabilityHook

# Required accept and escalate thresholds; pass None to disable categorical triage:
stability_hook = OptionStabilityHook(accept=0.85, escalate=0.60, variants="fast")
agent = laya.load("convaiinnovations/laya", hooks=[stability_hook])

Thresholds accept and escalate are required keyword arguments: categorical gating decisions (ACCEPT, VERIFY, ESCALATE) depend directly on application error tolerance, option count, and temperature calibration (Issue #1245). Pass accept=None, escalate=None to evaluate continuous stability signals without applying categorical gating.

It attaches a reliability diagnostic block to each choice question's answer: - probes: List of probe kinds that executed (e.g. ["rename"], ["reorder", "rename"], or []). - option_layout: Checkpoint layout ("parallel" or "sequential"). - soft_stability: Mean probability assigned to the chosen option across all evaluated variants, without additional rounding (None if unprobed). This is the exact value used for decision. - stability: Fraction of variants whose argmax choice agrees with the original decision (None if unprobed). - rename_stability: Stability across rename variants (None if no rename probe ran). - reorder_stability: Stability across reorder variants (None if no reorder probe ran). - decision: Categorical classification (ACCEPT, VERIFY, or ESCALATE, or None if unprobed). - variant_choices: Mapping of variant name to decoded choice. - variant_support: Mapping of variant name to probability mass on the primary choice.

Calibrating and replaying decisions¶

Collect soft_stability with accept=None, escalate=None on labelled evaluation examples, then pass those numeric values and correctness labels to laya.hooks.calibrate_reliability. Only fit or replay probed, numeric scores; exclude unprobed reports with soft_stability=None and keep their decision=None. The fitter requires equal, nonempty, sized score and label sequences; generators raise TypeError. Scores and both accuracy targets must be finite numbers.Real values in [0, 1], including NumPy scalars such as numpy.float32 and numpy.int64, but excluding bool and numpy.bool_; invalid lengths or values raise ValueError. Correctness labels must be bool or numpy.bool_ values (as produced by comparing NumPy arrays), otherwise it raises TypeError. It does not drop unmatched rows or coerce labels such as 0/1 or "false". Returned cutoffs are plain Python float values, including when fitting NumPy inputs, so they can be saved in JSON. It chooses cutoffs from the distinct input scores at their supplied precision; equal scores stay in the same group. ACCEPT includes scores equal to accept, while ESCALATE requires a score strictly below escalate. If no cutoff meets the requested accuracy, accept=1.01 accepts nothing. These are fits to the supplied examples, not a guarantee on unseen data.

Keep the numeric scores and thresholds unchanged when saving them (for example, in JSON). Replaying decide_reliability(report["soft_stability"], accept, escalate) then agrees with the hook for the same evaluated variants and decoded probabilities. Format a separate display string if fewer decimal places are wanted; do not fit or gate on that string. The other stability diagnostics and variant_support retain their four-decimal formatting and are not substitutes for soft_stability when replaying the gate.

Earlier reports rounded soft_stability to four decimals even though the gate used the unrounded mean. Previously saved reports cannot recover that lost precision: recollect them before refitting thresholds. Fixed thresholds still use the same runtime comparison; the reported score and newly fitted cutoffs may now contain more decimal places.

What it costs¶

One forward pass, not several, but a wider one: the variants are extra rows in the same batch. A four-option choice costs 3 rows under variants="fast" and 7 under "full", against 1 with no hook. On a GPU or MPS those extra rows are largely absorbed by the batch; on a CPU they are not, and the cost is close to linear in rows. Measured here on one 4-option choice, CPU, median of 15 calls after 4 warmups:

So on CPU this is a diagnostic to enable deliberately, per request or on a sample of traffic, rather than everywhere by default. Scale the row counts by option count from the table in #908.

Convenience types¶

PredictHook = Callable[[PredictContext], None]

PredictHook is the type of a plain callable used with on_predict_start= / on_predict_end=. Pass a single callable or a sequence of them; each is wrapped into a minimal hook.

Runtime registration¶

Every runtime mixes in HookRegistry, so hooks can be added, removed or scoped after construction. Mutation is thread-safe; a call reads a snapshot of the list, so adding or removing a hook never disturbs a call in flight.

agent.add_hook(tracer)              # one hook or a sequence; returns self for chaining
agent.remove_hook(tracer)           # by identity; True if it was installed

with agent.hooks_installed(debug):  # installed for the block, removed on exit
    agent.system_one(state, questions)

add_hook accepts the same objects as hooks= (not plain callables). hooks_installed takes any number of hook objects or sequences and, on exit -- including when the block raises -- removes exactly what it installed: one occurrence of each hook it added, the most recent. It does not restore a snapshot of the list, so a hook the application had already installed stays installed, and a hook added inside the block with add_hook is left in place.

Process-wide defaults¶

laya.hooks keeps a small process-wide registry, so a tracer, metrics hook or tenant tagger does not have to be threaded through every Agent and Router. Defaults run first, then the hooks installed on the instance, then per-call hooks.

from laya import hooks

hooks.set_default_hooks(hooks=[Tracer()])          # replaces the set, accepts the hooks= arguments
hooks.add_default_hook(Metrics())                  # appends
hooks.clear_default_hooks()                        # removes everything
hooks.default_hooks()                              # a copy of the current list

hooks.compose_hooks(agent.hooks)                   # defaults + installed (advanced)

Defaults apply to every event, including the Router lifecycle events on_load and on_evict. The registry is read at call time, so hooks set after an Agent or Router is built still apply. There is no per-instance opt-out; call clear_default_hooks() to turn the process-wide set off.

Async hooks¶

An event may be a coroutine. Wrap the hook in AsyncHook and its async def methods run to completion in the sync core:

from laya import AsyncHook

class Remote:
    async def on_predict_end(self, ctx):
        await ship(ctx.results)

agent = laya.load("convaiinnovations/laya", hooks=[AsyncHook(Remote())])

A plain async callable passed to on_predict_start= / on_predict_end= also works, because dispatch runs any awaitable a hook returns.

Where the coroutine runs:

• If the calling thread has no running loop, it is run with asyncio.run.
• If it already has one (a caller inside an async function), it runs on a dedicated background loop, so the calling thread can block without deadlocking. Pass AsyncHook(hook, loop=...) to funnel onto a specific loop; it must be running, and must not be the calling thread's own loop. Both are checked: a stopped loop and the caller's own loop each raise ValueError instead of blocking forever.

A hook with no async methods is unaffected.

Configuration surface¶

Every entry point accepts the same hook parameters. hooks takes an object or a sequence of objects; on_predict_start / on_predict_end take a callable or a sequence.

Agent¶

Agent(
    model_id_or_path="convaiinnovations/laya",
    device=None, token=None, subfolder=None, fast=False, compile=False,
    hooks=None, on_predict_start=None, on_predict_end=None,
    hooks_raise=True, hooks_concurrent=True, hooks_timeout=None,
)

load(..., hooks=None, on_predict_start=None, on_predict_end=None,
     hooks_raise=True, hooks_concurrent=True, hooks_timeout=None)

agent.predict_batch(states, questions, batch_size=None,
                    hooks=None, on_predict_start=None, on_predict_end=None, hooks_raise=None,
                    hooks_timeout=None, max_len=None, head_max_len=None, sort_by_length=False)

agent.system_one(state, questions,
                 hooks=None, on_predict_start=None, on_predict_end=None, hooks_raise=None,
                 hooks_timeout=None, max_len=None, head_max_len=None)

agent.predict_long(state, questions, window=None, stride=None, aggregate="auto",
                   batch_size=None, lang=None,
                   hooks=None, on_predict_start=None, on_predict_end=None, hooks_raise=None,
                   hooks_timeout=None)

agent.predict(...)          # alias of system_one

• hooks_raise and hooks_timeout on a per-call method default to None, meaning "use the instance value".
• hooks_concurrent is instance-level only.
• On predict_long the hooks wrap the inference that answers the state, which for a document needing several windows is the single shared predict_batch over them: on_predict_start fires once, and ctx.states holds the decoded window texts in scan order rather than the caller's state, which was tokenized to produce them. states is mutable from a start hook, so the scan that reaches inference need not be the split predict_long computed. What changes is what the answer can claim:

usage["windows"] is total over every path, including the one where a state already fit a single window, so a cached answer never reads as a window the model read.

End hooks may reorder result rows: an unchanged answer keeps the index and token span of its original window. Attribution requires the original answer object and unchanged prediction fields (type, choice/noul/score, probabilities, legend and confidence). Cloned, replaced or edited answers omit answer["window"]; reliability and action annotations are retained. Adding the window to the returned answer does not mutate the per-window answer a hook may have cached.

Router¶

Router(
    models=None, device=None, token=None, max_loaded=2, default="multilingual",
    auto_task_detection=False, standalone_repos=False, preload=False, lang_guess=None,
    hooks=None, on_predict_start=None, on_predict_end=None,
    hooks_raise=True, hooks_concurrent=True, hooks_timeout=None,
)

router.route(state, questions=None, model=None, task=None, lang=None, lang_guess=None,
             hooks=None, hooks_raise=None, hooks_timeout=None)

router.predict(state, questions, model=None, task=None, lang=None, lang_guess=None,
               hooks=None, on_predict_start=None, on_predict_end=None, hooks_raise=None,
               hooks_timeout=None, max_len=None, head_max_len=None)

router.predict_batch(requests, batch_size=None, hooks_timeout=None, min_confidence=None,
                     sort_by_length=False, hooks=None, on_predict_start=None, on_predict_end=None,
                     hooks_raise=None, diagnostics=None)

router.route_batch(requests, hooks_timeout=None, *, hooks=None, hooks_raise=None)

router.system_one(...)      # alias of predict
router.load(name)           # builds on first use; fires on_load
router.preload(names=None)  # builds several; fires on_load per build
router.unload(name=None)    # frees one or all; fires on_evict
router.attach(name, agent)  # registers an existing agent; does not fire on_load
router.loaded               # list of resident checkpoint names

• route_batch calls route once per request, so it dispatches on_route once per request with the same per-call hooks. It is the one surface whose hook arguments differ in shape: hooks and hooks_raise are keyword-only and there is no on_predict_start= / on_predict_end=, because it never predicts.
• Per-call hooks= on route, route_batch, predict and predict_batch apply to the whole call, including on_route. On predict_batch the list is composed the same way as on predict (installed hooks first, then the per-call list; None and [] add nothing) and runs once per request.
• route() is public: calling it dispatches on_route with the installed hooks plus any per-call hooks.

ONNXAgent¶

ONNXAgent(model_id_or_path, onnx_path="laya.onnx", subfolder=None,
          hooks=None, on_predict_start=None, on_predict_end=None,
          hooks_raise=True, hooks_concurrent=True, hooks_timeout=None)

onnx_agent.system_one(state, questions,
                      hooks=None, on_predict_start=None, on_predict_end=None, hooks_raise=None,
                      hooks_timeout=None, max_len=None, head_max_len=None)

onnx_agent.predict(...)     # alias of system_one

ONNXAgent has no Router, so it exposes only the predict-level events.

Event payloads¶

Which fields are populated, per event and runtime:

Timing details:

• on_predict_end sees results on the success path. On the failure path results is None unless a start hook had set them via skip(), and usage is therefore None too (it is derived from results); elapsed_ms is always set.
• on_error runs before the finally block that computes elapsed_ms and usage, so both are None there. Read timing and usage from on_predict_end instead.
• run_id is always populated.

Validation¶

Configuration is validated when hooks are normalised, which happens at construction for installed hooks and at call time for per-call hooks. These raise TypeError:

hooks= does not accept plain callables, because a bare callable does not say which event it is for. Use on_predict_start= / on_predict_end= for those.

Advanced helpers¶

These are used internally and are stable, but most users do not need them.

HOOK_EVENTS          # tuple of the six event names; the set hooks= is validated against
normalise_hooks(hooks=None, on_predict_start=None, on_predict_end=None) -> list
dispatch(hooks, event, ctx, *, raise_errors=True, lock=None) -> None
aggregate_usage(results) -> {"input_tokens": int, "output_tokens": int}

dispatch(hooks, event, ctx, *, raise_errors=True, lock=None, timeout=None)
run_coroutine_sync(coro, loop=None)

HOOK_EVENTS is the set of names normalise_hooks and AsyncHook look for on a hook; its order is declaration order, not the order the events fire in a call -- for that see Lifecycle. normalise_hooks flattens a hooks object/sequence and the two callables into one ordered list. dispatch calls event on every hook that implements it, applying the raise policy, lock and timeout, and runs a hook's result if it is awaitable. run_coroutine_sync runs an awaitable to completion from sync code, on the caller's loop if it is free, or on a background loop if the caller already has one. aggregate_usage sums per-state usage blocks.

from laya.hooks import normalise_hooks, dispatch, PredictContext

hooks = normalise_hooks(on_predict_start=[log, redact])
ctx = PredictContext(states=["..."], questions={...})
dispatch(hooks, "on_predict_start", ctx)

See also¶

• Lifecycle: when each event runs, with flowcharts.
• Errors: the failure matrix and chaining rules.
• Patterns and anti-patterns: how to structure hooks well.
• Examples: recipes for every use case.