Router¶

laya.Router detects the language of each state and sends the request to the matching checkpoint, loading checkpoints on first use.

Router ¶

Router(
    models: Optional[Dict[str, str]] = None,
    device: Optional[str] = None,
    token: Optional[str] = None,
    max_loaded: int = 2,
    default: str = "english",
    auto_task_detection: bool = False,
    standalone_repos: bool = False,
    preload: bool = False,
    lang_guess: Optional[Any] = None,
    hooks=None,
    on_predict_start=None,
    on_predict_end=None,
    hooks_raise: bool = True,
    hooks_concurrent: bool = True,
)

Bases: HookRegistry

Lazily loads Laya checkpoints and sends each request to the right one.

from laya import Router

r = Router()
r.predict({"message": "Mein Konto wurde zweimal belastet"}, questions)   # -> multilingual
r.predict({"message": "I was charged twice"}, questions)                 # -> english
r.predict(state, questions, model="typed-decisions")                     # explicit

Models are downloaded and built on first use. max_loaded caps how many stay resident (least-recently-used is evicted), because all three together are ~1.16B parameters.

The default is 2, because automatic routing only ever chooses between english and multilingual: a cap of one rebuilds the checkpoint it just evicted on every script switch, which is seconds per request on exactly the traffic the Router exists for. Traffic that only ever sees one language never builds the second checkpoint, so the default costs it nothing. Lower it to 1 for a memory-constrained host, and raise it to 3 (or preload) when auto_task_detection, an explicit model= or an explicit task= can reach typed-decisions as well.

For a server or a demo, preload instead: a cold load costs seconds, while detection costs microseconds, so even the default still pays a load the first time a language appears.

r = Router(preload=True)                    # all three resident, routing is free
r = Router(preload=True, device="cuda")
r.preload(["english", "multilingual"])      # or just the two you serve

Hooks are opt-in and run at the Router level: on_route sees the routing decision, on_load / on_evict see model lifecycle, and on_predict_start / on_predict_end wrap the whole route+infer call. See laya.hooks.

load ¶

load(name: str)

Return the Agent for name, downloading and building it on first use.

Concurrent callers share a single Agent instead of building duplicates.

attach ¶

attach(name: str, agent: Any)

Register an already-built Agent under name instead of loading a second copy.

Useful when the process has a checkpoint loaded for other reasons: a demo that already built convaiinnovations/laya can hand it to the router rather than pay for -- and hold in memory -- a duplicate 421M parameters.

preload ¶

preload(names: Optional[List[str]] = None)

Download and build checkpoints up front so no request ever pays a model load.

A cold load costs seconds; language detection costs microseconds. With every checkpoint resident, routing is effectively free -- which is what you want in a server or a demo. max_loaded is raised to fit both the requested checkpoints and all already-resident agents, so incremental preloading does not evict either.

unload ¶

unload(name: Optional[str] = None)

Free one model, or all of them.

route ¶

route(
    state: Union[str, dict, list, None],
    questions: Optional[Dict[str, Any]] = None,
    model: Optional[str] = None,
    task: Optional[str] = None,
    lang: Optional[str] = None,
    lang_guess: Optional[Any] = None,
    hooks=None,
    hooks_raise: Optional[bool] = None,
) -> RouteDecision

Decide which checkpoint to use, then let on_route hooks observe or replace it.

ctx.decision is the RouteDecision; a hook may replace it (for example to pin a checkpoint) and the replacement is what gets returned and used. hooks are per-call hooks, appended after any installed on the Router.

predict ¶

predict(
    state: Union[str, dict, list],
    questions: Dict[str, Any],
    model: Optional[str] = None,
    task: Optional[str] = None,
    lang: Optional[str] = None,
    lang_guess: Optional[Any] = None,
    hooks=None,
    on_predict_start=None,
    on_predict_end=None,
    hooks_raise: Optional[bool] = None,
    max_len: Optional[int] = None,
    head_max_len: Optional[int] = None,
) -> Dict[str, Any]

Route, then answer every question in one forward pass on the chosen checkpoint.

The result is the usual system_one payload plus a routing key recording the decision. Router-level on_predict_start / on_predict_end hooks wrap the whole route+infer call and see ctx.decision; see laya.hooks. max_len / head_max_len override the agent token budget for this call (a start hook may set ctx.max_len / ctx.head_max_len).

decide ¶

decide(
    state: Union[str, dict, list],
    schema: Any = None,
    *,
    questions: Optional[Dict[str, Any]] = None,
    return_details: bool = False,
    **predict_kwargs,
) -> Any

Answer state against a schema (JSON schema or pydantic model) and return typed values.

See laya.structured. Pass exactly one of schema or questions; extra keyword arguments (for example model=, task=, hooks=) are forwarded to predict.

route_batch ¶

route_batch(
    requests: Sequence[Dict[str, Any]],
) -> List[RouteDecision]

Route a heterogeneous request batch without loading any checkpoints.

Each request is a mapping with state and questions plus the same optional routing overrides accepted by :meth:route: model, task, lang and lang_guess. The returned decisions preserve input order.

This is intentionally separate from inference so callers can inspect or aggregate routing decisions before paying model-load cost.

predict_batch ¶

predict_batch(
    requests: Sequence[Dict[str, Any]],
    batch_size: Optional[int] = None,
) -> List[Dict[str, Any]]

Route and execute a heterogeneous request batch with minimal model churn.

Requests are routed first and grouped by checkpoint. Within each checkpoint, requests that share the same question schema are passed to Agent.predict_batch so their states can share forward passes. Results are then restored to the original request order.

Requests may independently specify model, task, lang or lang_guess and may use different question schemas.

Router-level predict hooks run per request, as predict runs them: each request gets its own PredictContext, so on_predict_start can replace that request's state, questions or token budget, or ctx.skip(...) it, and on_predict_end sees and may replace its result. Requests are grouped for the forward pass after their start hooks have run. If the batch fails, every request whose start hook ran and that has no result gets on_error, then every started request gets on_predict_end, before the exception propagates.

Parameters:

requests (Sequence[Dict[str, Any]]) –

Sequence of request dictionaries. Every item requires state and questions and may include model, task, lang or lang_guess overrides.
batch_size (Optional[int], default: None ) –

Optional maximum number of states per Agent forward-pass batch.

Returns:

List[Dict[str, Any]] –

One normal Router prediction result per request, in the same order as the input.

RouteDecision ¶

Bases: dict

The routing outcome: which model, why, and what was detected.

Behaves as a dict so it serialises straight into an API response.

DEFAULT_MODELS `module-attribute` ¶

DEFAULT_MODELS = {
    "english": (BUNDLE_REPO, None),
    "multilingual": (BUNDLE_REPO, "multilingual"),
    "typed-decisions": (BUNDLE_REPO, "typed-decisions"),
}