Skip to content

Routing

Router chooses a checkpoint for each request and loads its Agent when prediction needs it. The default router sends English text to the English checkpoint and other supported languages to the multilingual checkpoint. Text whose language it cannot place follows default, which is the multilingual checkpoint since 0.4.0. You can override that choice, supply your own language hint, or select the typed-decisions checkpoint explicitly.

This guide covers model selection and lifecycle. For the question types accepted by prediction, see Structured decisions; for lifecycle callbacks, see Prediction hooks.

Quick start

from laya import Router

router = Router()

questions = {
    "department": {
        "type": "choice",
        "instructions": "Which department should handle this request?",
        "criteria": {
            "billing": "invoices, payments, refunds",
            "technical": "bugs, outages, system errors",
            "other": "everything else",
        },
    }
}

result = router.predict("We were billed twice. Please refund the duplicate charge.", questions)
print(result["answers"]["department"]["choice"])
print(result["routing"]["model"])

Creating Router() does not download checkpoints by default. predict() routes the request, then loads the selected checkpoint on first use. The first prediction can therefore take longer while files are downloaded and the model is initialized; later predictions reuse the loaded agent.

How a checkpoint is selected

Router.route(state, questions, ...) returns a RouteDecision without loading a checkpoint or running inference. The decision contains the selected model, a human-readable reason, and language detection details when built-in detection was used.

Routing checks inputs in this order:

  1. model= selects a checkpoint directly.
  2. task= selects a checkpoint for the named task.
  3. If auto_task_detection=True, an exact match against one of the known typed-decisions question ID sets selects typed-decisions.
  4. A recognized lang= value selects English or multilingual.
  5. A per-call lang_guess= or the router's configured lang_guess is consulted.
  6. Built-in script and language analysis selects a checkpoint. If there is no reliable language signal, the router uses its configured default (multilingual since 0.4.0).

The first matching rule wins. For example, model="multilingual" overrides lang="en". Invalid model names raise ValueError instead of falling through to detection.

decision = router.route(
    "La aplicación se cierra cada vez que abro la configuración.",
    questions,
)
print(decision.model)   # multilingual
print(decision.reason)  # why that checkpoint was selected

RouteDecision is dict-compatible, so its fields are also available with keys such as decision["model"] and decision["reason"]. Router.predict() includes the same decision under the result's "routing" key.

The vega lane

Since 0.4.25 a Router has a fourth lane beside the three encoder checkpoints: vega, served by laya.Agent(vegaml=True), the frozen 0.8B universal reader with vegaml's decision engine. It is on by default when the vegaml package is installed (it ships with laya 0.4.23 and newer) and built on first use, so a Router costs nothing more to construct than before. Text the encoders can read keeps going to the encoders, with the same decisions and the same reasons as before. The lane only takes what the encoders cannot:

input routed to vega because
image=, audio= or video= instead of a state the encoders have no vision or speech path
a state past vega_over_tokens (8,192, the most any encoder reads) the encoders would truncate it; vega reads up to 73,728 tokens whole
a question the vega agent has a fitted head for the head was fitted on your rows for that question
from laya import Router

router = Router()                                    # the lane is on if vegaml is installed
router.predict(ticket, questions)                    # English text: english, as before
router.predict(seventy_page_contract, questions)     # about 40,000 tokens: vega, read whole
router.predict(None, questions, image="invoice.png") # vega
router.predict(None, questions, audio="call.wav")    # vega, transcribed then decided
router.route(long_text, questions).reason            # "state is about 40000 tokens, past the 8192 ..."

Router(vega=False) turns the lane off and the Router behaves exactly as it did before, including refusing media with a message that names the lane. Router(vega=True) insists on it and raises at construction when vegaml is missing. Router(vega="4b") names a different reader (any name laya.Agent(vegaml=...) takes), and Router(vega=agent) attaches an agent you already built, which is how fitted heads reach the Router: fit on the agent, hand it over, and the questions it serves route to it. model="vega" selects the lane explicitly like any other checkpoint name.

Two things stay on the encoders on purpose. predict_batch never routes to vega, because a batch shares one forward pass on one checkpoint; send media and long states one at a time. And the HTTP server refuses image, audio and video in a body with a message saying where media goes, since the JSON surface has no media encoding yet.

Images and video through the lane need pip install 'laya[vision]' (Pillow and torchvision) and audio files pip install 'laya[audio]' (soundfile); text and a numpy waveform need nothing more.

Answers from the lane carry vega's fields on top of laya's: prediction_set, abstain, unbound and route; see vega inside laya.

Explain a detection result

Use explain=True to inspect why built-in detection reached its result, without loading a checkpoint. The existing detection fields and routing rules stay unchanged; only the optional diagnostic field is added.

from laya import Router
from laya.lang import analyse

detection = analyse("Turn lights off", explain=True)
print(detection["diagnostic"])
# {"stage": "whole_state", "reason": "insufficient_words", "source": None}

router = Router(default="english")
decision = router.route("Turn lights off", explain=True)
assert decision["model"] == "english"
assert decision["detection"]["language_undecided"] is True

The detector's reason is separate from the router's policy: an unidentified language follows the configured default, which need not be multilingual. Reasons describe heuristic branches, not calibrated language confidence. Explicit model/task/language overrides, an accepted lang_guess, and automatic workflow selection still skip detection and return detection=None. As before, an on_route hook can replace the whole decision.

For structured input, a later value can change the result even when a long earlier value has filled the joined-text window:

state = {
    "note": "This is an English note about the order. " * 140,
    "payload": {"customer": "Ich kann mich nicht anmelden und brauche Hilfe"},
}
detection = analyse(state, explain=True)
assert detection["language"] == "de"
assert detection["mixed_segment"] is None  # its existing meaning is unchanged
assert detection["diagnostic"] == {
    "stage": "value",
    "reason": "named_value",
    "source": {"leaf_index": 1, "path": ["payload", "customer"], "line": 1},
}

stage is whole_state for the initial joined sample, segment for the first qualifying line in the shared scan window, or value for the subsequent per-value fallback. Only the selected result is reported; this is not a trace of every candidate. An override changes only the fields it changed before: for example, a late non-Latin value can make is_english=False while script_profile still describes the initial Latin sample. The diagnostic identifies the override's source; it does not relabel those older measurements.

Reason Meaning
no_letters The initial sample contains no alphabetic characters.
non_latin_script The script check selected a non-Latin script, including mixed-script promotion.
insufficient_words The Latin language guess returned before general scoring because it had fewer than four words.
no_language_evidence General word scoring did not identify a language.
language_evidence Word evidence identified a language; see the existing language field.
short_swedish The existing short Swedish phrase rule identified the language.
caps_veto The all-capitals evidence check rejected the language name.
caps_filtered The language name was reconsidered with acronym-shaped capitals removed.
non_english_markers No language was named, but diacritics or a shared Nordic marker still made the text non-English. This takes precedence over an inconclusive word/caps reason.
named_segment A named non-English prose segment overrode the joined result.
named_value The selected fallback line passed the named-language prose check.
non_latin_value The selected fallback line passed the non-Latin word/letter check.
diacritic_value The selected fallback line passed the diacritic and word-count check.

For whole_state, source is None: the joined sample need not correspond to a single field. Other stages report a zero-based leaf_index in the existing string-leaf traversal and a one-based line within that decoded leaf. path is an array of mapping keys and sequence indices, so keys containing dots or slashes remain unambiguous; a root string has path []. The detector's existing depth and character limits still apply. Invalid UTF-8 byte leaves and non-text values do not contribute a leaf index.

A path retains string keys of at most 128 characters and integer keys with at most 63 bits of magnitude, including negative integers. If a key anywhere on that path is unsupported or exceeds those bounds, the entire path is None; the leaf index and line remain available. No arbitrary key is stringified, and the diagnostic adds no input text. The existing mixed_segment can still contain text from the segment scan, as it did before.

Router.route_batch(requests, explain=True) applies the same option to the whole routing batch, preserving order. The CLI exposes it in route-only mode:

laya "Turn lights off" --explain --json
laya --batch tickets.txt --explain --json

The flag also works in interactive mode and ordinary text output. Combining it with --predict, --preset or --questions is rejected: diagnostics belong to the routing call, not a separate detector pass attached to an inference result.

Override language routing

Use lang= when the application already knows the request language. Language tags such as "en", "en-US", and "en_US.UTF-8" are accepted. English routes to english; other recognized language codes route to multilingual.

result = router.predict(state, questions, lang="de")
assert result["routing"]["model"] == "multilingual"

If the application has its own language detector, pass its result as a language code with lang_guess=. A callable receives the state and can return a code or None to abstain:

def detect_request_language(state):
    # Replace this with the application's detector.
    return "en" if "invoice" in str(state).lower() else None

router = Router(lang_guess=detect_request_language)

An abstaining hint does not force a checkpoint; routing continues to the next rule. That covers None, a blank string, and the codes that name no language (C, POSIX, C.UTF-8, und, zxx, mul), which is what a detector returns when it has nothing to say, so an abstention cannot silently pin requests to the wrong model.

An unrecognized hint is not an abstention. Any other value, including "xx", False and 0, is read as "not English" and routes to the multilingual checkpoint. So a detector that returns a garbage code rather than None does pick a checkpoint, and if that matters, map its unknown case to None before passing it on.

Built-in detection is a lightweight script and language heuristic, not a general-purpose language identification model. It analyzes string values in text, dict, and list states; dictionary keys are ignored because they are often field names in English. Short or ambiguous text can use the default checkpoint. For known workloads, an explicit language or application-provided hint is more predictable.

Selecting typed-decisions

The typed-decisions checkpoint is not selected automatically by default. Select it explicitly:

result = router.predict(state, questions, model="typed-decisions")
# `task="typed_decisions"` is also accepted.

Alternatively, set auto_task_detection=True. The router then checks whether the question IDs are an exact match for one of its known typed-decision workflows. It does not infer the task from the question wording, and adding unrelated question IDs prevents an exact match.

router = Router(auto_task_detection=True)

Registering your own checkpoints

Router(models=...) and Router.register(name, source, description=None) accept any name beside the three built-in ones. A source is a Hub repo id, a (repo, subfolder) pair or a local directory. A registered checkpoint loads, is evicted and unloads like a built-in, and is named in model= or task= like one. Router.registered lists the registered checkpoints and Router.unregister(name) removes one. A name one edit from a built-in, such as englsh for english, is still registered as its own checkpoint and a warning names the built-in, which is left unchanged.

router = Router(models={"papers": "/models/laya-papers"})
router.register("tone", ("acme/laya-tone", None), description="tone of voice")
result = router.predict(state, questions, model="papers")

normalise_name still knows only the built-in names, while Router.resolve also knows the registry.

Inspect routing without loading models

Use route() to inspect one decision or route_batch() to inspect a sequence. Neither method loads checkpoints, so both are useful for debugging routing rules before running inference.

requests = [
    {"state": "Please refund the duplicate charge.", "questions": questions},
    {"state": "Necesito ayuda con mi factura.", "questions": questions},
]

decisions = router.route_batch(requests)
for decision in decisions:
    print(decision.model, decision.reason)

Every route_batch() item needs state and questions; optional routing overrides (model, task, lang, and lang_guess) are specified per item. Decisions stay in input order.

Loading and memory

By default, the router loads a checkpoint the first time it is needed and keeps at most two agents resident. Automatic language routing normally needs only the English and multilingual checkpoints. If requests can also select typed-decisions, a small max_loaded may evict another agent and cause it to be loaded again the next time it is needed.

# Load only the checkpoints this process serves, before accepting requests.
router = Router()
router.preload(["english", "multilingual"])

print(router.loaded)  # currently resident checkpoint names
router.unload("multilingual")

Router(preload=True) preloads all configured checkpoints. Preloading raises the resident-model limit to fit the requested set. To control a three-checkpoint workload without preloading, set max_loaded=3. Use unload() to release one agent or all agents (router.unload()). A router used as a context manager unloads its agents when the block exits:

with Router(preload=True) as router:
    result = router.predict(state, questions)

You can also pass device="cpu", device="cuda", or another supported PyTorch device when constructing the router. Availability and memory determine which devices can run a given model.

Mixed batches

predict_batch() accepts requests with different states, models, languages, and question schemas. The router first makes a decision for each request, groups work by checkpoint and compatible question schema, then restores the results to the original input order.

requests = [
    {"state": "Please refund the duplicate charge.", "questions": questions},
    {"state": "Mi cuenta fue cobrada dos veces.", "questions": questions},
    {"state": "A third request", "questions": questions, "model": "typed-decisions"},
]

results = router.predict_batch(requests, batch_size=8)

Each request requires state and questions; it may also include the routing overrides model, task, lang and lang_guess, or the token budgets max_len and head_max_len. Those are the same arguments predict() takes, applied per item, so a wide question does not shrink the batch's other states to the same window. Requests sharing a checkpoint, question schema and token budget can share an Agent batch forward pass; a different schema, checkpoint or budget is grouped separately, since one Agent batch carries one budget for all its states. batch_size limits the number of states passed together to the Agent; results still correspond to the request order.

Choosing an entry point

  • Use route() or route_batch() when you need to inspect decisions without loading models.
  • Use predict() for a single request and predict_batch() for multiple possibly heterogeneous requests.
  • Use Agent directly when the application already chose and loaded one checkpoint and does not need automatic routing.

See the Router API reference for constructor and method details.