Routing¶
Router chooses a checkpoint for each request and loads its Agent when prediction needs it.
The default router sends English text to the English checkpoint and other supported languages to
the multilingual checkpoint. Text whose language it cannot place follows default, which is the
multilingual checkpoint since 0.4.0. You can override that choice, supply your own language hint, or
select the typed-decisions checkpoint explicitly.
This guide covers model selection and lifecycle. For the question types accepted by prediction, see Structured decisions; for lifecycle callbacks, see Prediction hooks.
Quick start¶
from laya import Router
router = Router()
questions = {
"department": {
"type": "choice",
"instructions": "Which department should handle this request?",
"criteria": {
"billing": "invoices, payments, refunds",
"technical": "bugs, outages, system errors",
"other": "everything else",
},
}
}
result = router.predict("We were billed twice. Please refund the duplicate charge.", questions)
print(result["answers"]["department"]["choice"])
print(result["routing"]["model"])
Creating Router() does not download checkpoints by default. predict() routes the request, then
loads the selected checkpoint on first use. The first prediction can therefore take longer while
files are downloaded and the model is initialized; later predictions reuse the loaded agent.
How a checkpoint is selected¶
Router.route(state, questions, ...) returns a RouteDecision without loading a checkpoint or
running inference. The decision contains the selected model, a human-readable reason, and language
detection details when built-in detection was used.
Routing checks inputs in this order:
model=selects a checkpoint directly.task=selects a checkpoint for the named task.- If
auto_task_detection=True, an exact match against one of the known typed-decisions question ID sets selectstyped-decisions. - A recognized
lang=value selects English or multilingual. - A per-call
lang_guess=or the router's configuredlang_guessis consulted. - Built-in script and language analysis selects a checkpoint. If there is no reliable language
signal, the router uses its configured
default(multilingual since 0.4.0).
The first matching rule wins. For example, model="multilingual" overrides lang="en". Invalid
model names raise ValueError instead of falling through to detection.
decision = router.route(
"La aplicación se cierra cada vez que abro la configuración.",
questions,
)
print(decision.model) # multilingual
print(decision.reason) # why that checkpoint was selected
RouteDecision is dict-compatible, so its fields are also available with keys such as
decision["model"] and decision["reason"]. Router.predict() includes the same decision under
the result's "routing" key.
The vega lane¶
Since 0.4.25 a Router has a fourth lane beside the three encoder checkpoints: vega, served by
laya.Agent(vegaml=True), the frozen 0.8B universal reader with vegaml's decision engine. It is on
by default when the vegaml package is installed (it ships with laya 0.4.23 and newer) and built
on first use, so a Router costs nothing more to construct than before. Text the encoders can read
keeps going to the encoders, with the same decisions and the same reasons as before. The lane only
takes what the encoders cannot:
| input | routed to vega because |
|---|---|
image=, audio= or video= instead of a state |
the encoders have no vision or speech path |
a state past vega_over_tokens (8,192, the most any encoder reads) |
the encoders would truncate it; vega reads up to 73,728 tokens whole |
| a question the vega agent has a fitted head for | the head was fitted on your rows for that question |
from laya import Router
router = Router() # the lane is on if vegaml is installed
router.predict(ticket, questions) # English text: english, as before
router.predict(seventy_page_contract, questions) # about 40,000 tokens: vega, read whole
router.predict(None, questions, image="invoice.png") # vega
router.predict(None, questions, audio="call.wav") # vega, transcribed then decided
router.route(long_text, questions).reason # "state is about 40000 tokens, past the 8192 ..."
Router(vega=False) turns the lane off and the Router behaves exactly as it did before, including
refusing media with a message that names the lane. Router(vega=True) insists on it and raises at
construction when vegaml is missing. Router(vega="4b") names a different reader (any name
laya.Agent(vegaml=...) takes), and Router(vega=agent) attaches an agent you already built, which
is how fitted heads reach the Router: fit on the agent, hand it over, and the questions it serves
route to it. model="vega" selects the lane explicitly like any other checkpoint name.
Two things stay on the encoders on purpose. predict_batch never routes to vega, because a batch
shares one forward pass on one checkpoint; send media and long states one at a time. And the HTTP
server refuses image, audio and video in a body with a message saying where media goes, since
the JSON surface has no media encoding yet.
Images and video through the lane need pip install 'laya[vision]' (Pillow and torchvision) and
audio files pip install 'laya[audio]' (soundfile); text and a numpy waveform need nothing more.
Answers from the lane carry vega's fields on top of laya's: prediction_set, abstain, unbound
and route; see vega inside laya.
Explain a detection result¶
Use explain=True to inspect why built-in detection reached its result, without loading a
checkpoint. The existing detection fields and routing rules stay unchanged; only the optional
diagnostic field is added.
from laya import Router
from laya.lang import analyse
detection = analyse("Turn lights off", explain=True)
print(detection["diagnostic"])
# {"stage": "whole_state", "reason": "insufficient_words", "source": None}
router = Router(default="english")
decision = router.route("Turn lights off", explain=True)
assert decision["model"] == "english"
assert decision["detection"]["language_undecided"] is True
The detector's reason is separate from the router's policy: an unidentified language follows
the configured default, which need not be multilingual. Reasons describe heuristic branches,
not calibrated language confidence. Explicit model/task/language overrides, an accepted
lang_guess, and automatic workflow selection still skip detection and return
detection=None. As before, an on_route hook can replace the whole decision.
For structured input, a later value can change the result even when a long earlier value has filled the joined-text window:
state = {
"note": "This is an English note about the order. " * 140,
"payload": {"customer": "Ich kann mich nicht anmelden und brauche Hilfe"},
}
detection = analyse(state, explain=True)
assert detection["language"] == "de"
assert detection["mixed_segment"] is None # its existing meaning is unchanged
assert detection["diagnostic"] == {
"stage": "value",
"reason": "named_value",
"source": {"leaf_index": 1, "path": ["payload", "customer"], "line": 1},
}
stage is whole_state for the initial joined sample, segment for the first qualifying
line in the shared scan window, or value for the subsequent per-value fallback. Only the
selected result is reported; this is not a trace of every candidate.
An override changes only the fields it changed before: for example, a late non-Latin value can
make is_english=False while script_profile still describes the initial Latin sample. The
diagnostic identifies the override's source; it does not relabel those older measurements.
| Reason | Meaning |
|---|---|
no_letters |
The initial sample contains no alphabetic characters. |
non_latin_script |
The script check selected a non-Latin script, including mixed-script promotion. |
insufficient_words |
The Latin language guess returned before general scoring because it had fewer than four words. |
no_language_evidence |
General word scoring did not identify a language. |
language_evidence |
Word evidence identified a language; see the existing language field. |
short_swedish |
The existing short Swedish phrase rule identified the language. |
caps_veto |
The all-capitals evidence check rejected the language name. |
caps_filtered |
The language name was reconsidered with acronym-shaped capitals removed. |
non_english_markers |
No language was named, but diacritics or a shared Nordic marker still made the text non-English. This takes precedence over an inconclusive word/caps reason. |
named_segment |
A named non-English prose segment overrode the joined result. |
named_value |
The selected fallback line passed the named-language prose check. |
non_latin_value |
The selected fallback line passed the non-Latin word/letter check. |
diacritic_value |
The selected fallback line passed the diacritic and word-count check. |
For whole_state, source is None: the joined sample need not correspond to a single field.
Other stages report a zero-based leaf_index in the existing string-leaf traversal and a
one-based line within that decoded leaf. path is an array of mapping keys and sequence
indices, so keys containing dots or slashes remain unambiguous; a root string has path [].
The detector's existing depth and character limits still apply. Invalid UTF-8 byte leaves and
non-text values do not contribute a leaf index.
A path retains string keys of at most 128 characters and integer keys with at most 63 bits of
magnitude, including negative integers. If a key anywhere on that path is unsupported or exceeds
those bounds, the entire path is None; the leaf index and line remain available. No arbitrary
key is stringified, and the diagnostic adds no input text. The existing mixed_segment can
still contain text from the segment scan, as it did before.
Router.route_batch(requests, explain=True) applies the same option to the whole routing batch,
preserving order. The CLI exposes it in route-only mode:
The flag also works in interactive mode and ordinary text output. Combining it with
--predict, --preset or --questions is rejected: diagnostics belong to the routing call,
not a separate detector pass attached to an inference result.
Override language routing¶
Use lang= when the application already knows the request language. Language tags such as
"en", "en-US", and "en_US.UTF-8" are accepted. English routes to english; other recognized
language codes route to multilingual.
result = router.predict(state, questions, lang="de")
assert result["routing"]["model"] == "multilingual"
If the application has its own language detector, pass its result as a language code with
lang_guess=. A callable receives the state and can return a code or None to abstain:
def detect_request_language(state):
# Replace this with the application's detector.
return "en" if "invoice" in str(state).lower() else None
router = Router(lang_guess=detect_request_language)
An abstaining hint does not force a checkpoint; routing continues to the next rule. That covers
None, a blank string, and the codes that name no language (C, POSIX, C.UTF-8, und,
zxx, mul), which is what a detector returns when it has nothing to say, so an abstention
cannot silently pin requests to the wrong model.
An unrecognized hint is not an abstention. Any other value, including "xx", False and 0,
is read as "not English" and routes to the multilingual checkpoint. So a detector that returns
a garbage code rather than None does pick a checkpoint, and if that matters, map its unknown
case to None before passing it on.
Built-in detection is a lightweight script and language heuristic, not a general-purpose language identification model. It analyzes string values in text, dict, and list states; dictionary keys are ignored because they are often field names in English. Short or ambiguous text can use the default checkpoint. For known workloads, an explicit language or application-provided hint is more predictable.
Selecting typed-decisions¶
The typed-decisions checkpoint is not selected automatically by default. Select it explicitly:
result = router.predict(state, questions, model="typed-decisions")
# `task="typed_decisions"` is also accepted.
Alternatively, set auto_task_detection=True. The router then checks whether the question IDs are
an exact match for one of its known typed-decision workflows. It does not infer the task from the
question wording, and adding unrelated question IDs prevents an exact match.
Registering your own checkpoints¶
Router(models=...) and Router.register(name, source, description=None) accept any name beside the
three built-in ones. A source is a Hub repo id, a (repo, subfolder) pair or a local directory. A
registered checkpoint loads, is evicted and unloads like a built-in, and is named in model= or
task= like one. Router.registered lists the registered checkpoints and Router.unregister(name)
removes one. A name one edit from a built-in, such as englsh for english, is still registered
as its own checkpoint and a warning names the built-in, which is left unchanged.
router = Router(models={"papers": "/models/laya-papers"})
router.register("tone", ("acme/laya-tone", None), description="tone of voice")
result = router.predict(state, questions, model="papers")
normalise_name still knows only the built-in names, while Router.resolve also knows the registry.
Inspect routing without loading models¶
Use route() to inspect one decision or route_batch() to inspect a sequence. Neither method
loads checkpoints, so both are useful for debugging routing rules before running inference.
requests = [
{"state": "Please refund the duplicate charge.", "questions": questions},
{"state": "Necesito ayuda con mi factura.", "questions": questions},
]
decisions = router.route_batch(requests)
for decision in decisions:
print(decision.model, decision.reason)
Every route_batch() item needs state and questions; optional routing overrides (model,
task, lang, and lang_guess) are specified per item. Decisions stay in input order.
Loading and memory¶
By default, the router loads a checkpoint the first time it is needed and keeps at most two agents
resident. Automatic language routing normally needs only the English and multilingual checkpoints.
If requests can also select typed-decisions, a small max_loaded may evict another agent and
cause it to be loaded again the next time it is needed.
# Load only the checkpoints this process serves, before accepting requests.
router = Router()
router.preload(["english", "multilingual"])
print(router.loaded) # currently resident checkpoint names
router.unload("multilingual")
Router(preload=True) preloads all configured checkpoints. Preloading raises the resident-model
limit to fit the requested set. To control a three-checkpoint workload without preloading, set
max_loaded=3. Use unload() to release one agent or all agents (router.unload()). A router used
as a context manager unloads its agents when the block exits:
You can also pass device="cpu", device="cuda", or another supported PyTorch device when
constructing the router. Availability and memory determine which devices can run a given model.
Mixed batches¶
predict_batch() accepts requests with different states, models, languages, and question schemas.
The router first makes a decision for each request, groups work by checkpoint and compatible
question schema, then restores the results to the original input order.
requests = [
{"state": "Please refund the duplicate charge.", "questions": questions},
{"state": "Mi cuenta fue cobrada dos veces.", "questions": questions},
{"state": "A third request", "questions": questions, "model": "typed-decisions"},
]
results = router.predict_batch(requests, batch_size=8)
Each request requires state and questions; it may also include the routing overrides model,
task, lang and lang_guess, or the token budgets max_len and head_max_len. Those are the
same arguments predict() takes, applied per item, so a wide question does not shrink the batch's
other states to the same window. Requests sharing a checkpoint, question schema and token budget
can share an Agent batch forward pass; a different schema, checkpoint or budget is grouped
separately, since one Agent batch carries one budget for all its states. batch_size limits the
number of states passed together to the Agent; results still correspond to the request order.
Choosing an entry point¶
- Use
route()orroute_batch()when you need to inspect decisions without loading models. - Use
predict()for a single request andpredict_batch()for multiple possibly heterogeneous requests. - Use
Agentdirectly when the application already chose and loaded one checkpoint and does not need automatic routing.
See the Router API reference for constructor and method details.