Agent¶

laya.Agent loads one checkpoint and answers typed questions about a state. laya.load is a shortcut for Agent(...), and laya.RLAgent is an alias of Agent. ONNXAgent runs an exported ONNX model on CPU; import it from laya.onnx_agent.

Agent ¶

Agent(
    model_id_or_path: str = "convaiinnovations/laya",
    device: Optional[str] = None,
    token: Optional[str] = None,
    subfolder: Optional[str] = None,
    fast: bool = False,
    compile: bool = False,
    lang_temperatures: Optional[Dict[str, Dict[str, Any]]] = None,
    hooks=None,
    on_predict_start=None,
    on_predict_end=None,
    hooks_raise: bool = True,
    hooks_concurrent: bool = True,
)

Bases: HookRegistry

System 1 decision model runtime: fast, non-autoregressive, calibrated decisions.

Load a Laya checkpoint.

fast=True swaps the encoder/head forward for the TileLang fast path (CUDA only, needs pip install laya[fast]); see Agent.accelerate.

subfolder selects one checkpoint from a repo that bundles several, e.g. Agent("convaiinnovations/laya", subfolder="multilingual"). Only that subfolder is downloaded, so bundling does not cost every user the whole family.

hooks / on_predict_start / on_predict_end observe or shape every prediction; see laya.hooks. hooks_raise=False warns and continues when a hook fails, and hooks_concurrent=False serialises hooks that are not safe to run in parallel.

accelerate ¶

accelerate(use_graphs: bool = True, strict: bool = False)

Replace the model forward with the TileLang fast path (fused GEMM/GEGLU/LayerNorm/RoPE kernels, sliding-window flash attention, bf16 resident weights, CUDA graphs per shape bucket).

Same numerics as the stock bf16 autocast path (see benchmarks/bench_fast.py). Returns True if enabled. With strict=False any failure (no CUDA, tilelang missing) leaves the stock path in place.

deaccelerate ¶

deaccelerate()

Restore the stock forward.

predict_batch ¶

predict_batch(
    states: List[Union[str, dict, list]],
    questions: Dict[str, Dict[str, Any]],
    batch_size: Optional[int] = None,
    lang: Optional[str] = None,
    hooks=None,
    on_predict_start=None,
    on_predict_end=None,
    hooks_raise: Optional[bool] = None,
    max_len: Optional[int] = None,
    head_max_len: Optional[int] = None,
    sort_by_length: bool = False,
) -> List[Dict[str, Any]]

Evaluate the same questions over many states, packing them into shared forward passes.

This is the throughput path. system_one/predict handle one state per forward pass; on a GPU that leaves most of the batch dimension idle. predict_batch collates several states' question rows into one tensor, so a call that would take N sequential forward passes takes one (or ceil(len(states) / batch_size)), which is several times faster per decision on GPU.

Parameters:

states (List[Union[str, dict, list]]) –

A list of states (each a text string, JSON dict, or conversation turn list). The same questions are evaluated against every state.
questions (Dict[str, Dict[str, Any]]) –

Question definitions, exactly as accepted by system_one.
batch_size (Optional[int], default: None ) –

Optional cap on states per forward pass. None sends them all in one pass; set it to bound peak memory when batching many or long states.
hooks, on_predict_start, on_predict_end –

Per-call hooks, appended after any installed on the Agent. on_predict_start may rewrite the state/questions or call ctx.skip(...) to short-circuit inference; on_predict_end may rewrite the results. See laya.hooks.
hooks_raise (Optional[bool], default: None ) –

Override the Agent's hooks_raise for this call.
max_len, head_max_len –

Override the agent config for this call. A start hook may also set ctx.max_len / ctx.head_max_len to shape the token budget.
sort_by_length (bool, default: False ) –

Group similarly sized encoded states within windows of eight batches to reduce padding. Requires an explicit batch_size greater than one and smaller than the number of states; otherwise it has no effect. Results retain input order. This buffers up to eight batches of tokenized states instead of one. Changing batch shapes can slightly change floating-point predictions.

Returns:

List[Dict[str, Any]] –

A list of per-state result dicts, each identical in shape to system_one's output and aligned with states by index.

system_one ¶

system_one(
    state: Union[str, dict, list],
    questions: Dict[str, Dict[str, Any]],
    lang: Optional[str] = None,
    hooks=None,
    on_predict_start=None,
    on_predict_end=None,
    hooks_raise: Optional[bool] = None,
    max_len: Optional[int] = None,
    head_max_len: Optional[int] = None,
) -> Dict[str, Any]

Evaluate typed questions across state in a single, parallel forward pass.

Parameters:

state (Union[str, dict, list]) –

Text string, JSON dict, or conversation turn list.
questions (Dict[str, Dict[str, Any]]) –

Dictionary mapping question_id -> question definition. - choice: {"type": "choice", "instructions": "...", "criteria": {"optA": "...", ...}} - score: {"type": "score", "instructions": "...", "criteria": ["lvl0", "lvl1", ...]} - noul: {"type": "noul", "instructions": "...", "criteria": {"false": "...", "true": "..."}, "labels": {"false": "B", "true": "A"}}

Noul criteria and labels are optional. Labels only control the text shown to the model; their keys retain false/true semantics, and the returned noul value is always P(true). Labels default to false/true for compatibility.

Returns:

Dict[str, Any] –

Dictionary with answers, probabilities, calibrated confidence, and token usage. Empty questions return empty answers and zero token usage without tokenization or a model forward pass.

To score many states at once, see predict_batch, which shares forward passes across them.

decide ¶

decide(
    state: Union[str, dict, list],
    schema: Any = None,
    *,
    questions: Optional[Dict[str, Any]] = None,
    return_details: bool = False,
    **predict_kwargs,
) -> Any

Answer state against a schema (JSON schema or pydantic model) and return typed values.

See laya.structured. Pass exactly one of schema or questions; extra keyword arguments are forwarded to predict / system_one.

load ¶

load(
    model_id_or_path: str = "convaiinnovations/laya",
    device: Optional[str] = None,
    token: Optional[str] = None,
    subfolder: Optional[str] = None,
    fast: bool = False,
    lang_temperatures: Optional[Dict[str, Dict[str, Any]]] = None,
    hooks=None,
    on_predict_start=None,
    on_predict_end=None,
    hooks_raise: bool = True,
    hooks_concurrent: bool = True,
) -> Agent

Load a Laya agent.

subfolder picks one checkpoint out of a repo that bundles several:

laya.load("convaiinnovations/laya")                           # English (repo root)
laya.load("convaiinnovations/laya", subfolder="multilingual")
laya.load("convaiinnovations/laya", fast=True)                # TileLang GPU fast path

hooks / on_predict_start / on_predict_end observe or shape every prediction; see laya.hooks.

ONNXAgent ¶

ONNXAgent(
    model_id_or_path: str,
    onnx_path: str = "laya.onnx",
    subfolder: Optional[str] = None,
    hooks=None,
    on_predict_start=None,
    on_predict_end=None,
    hooks_raise: bool = True,
    hooks_concurrent: bool = True,
)

Bases: HookRegistry

System 1 decision model runtime via ONNX: fast CPU-optimized decisions.

Load a Laya agent backed by ONNX Runtime.

Parameters:

model_id_or_path (str) –

HuggingFace Hub ID or local path to the original PyTorch checkpoint (used to load the tokenizer and config).
onnx_path (str, default: 'laya.onnx' ) –

Path to the exported .onnx file.
subfolder (Optional[str], default: None ) –

Optional subfolder if downloading from a repo bundle.
hooks, on_predict_start, on_predict_end –

Opt-in prediction hooks; see laya.hooks.
hooks_raise (bool, default: True ) –

When False, a failing hook warns and inference continues.
hooks_concurrent (bool, default: True ) –

When False, hooks are serialised with a lock.

system_one ¶

system_one(
    state: Union[str, dict, list],
    questions: Dict[str, Dict[str, Any]],
    hooks=None,
    on_predict_start=None,
    on_predict_end=None,
    hooks_raise: Optional[bool] = None,
    max_len: Optional[int] = None,
    head_max_len: Optional[int] = None,
) -> Dict[str, Any]

Evaluate typed questions, running any opt-in hooks around the inference.

decide ¶

decide(
    state: Union[str, dict, list],
    schema: Any = None,
    *,
    questions: Optional[Dict[str, Dict[str, Any]]] = None,
    return_details: bool = False,
    **predict_kwargs,
) -> Any

Answer state against a schema (JSON schema or pydantic model) and return typed values.

See laya.structured. Pass exactly one of schema or questions; extra keyword arguments are forwarded to predict / system_one.