Thema
#cpu-inference
Loots, Blogposts und verwandte Themen rund um diesen Tag. Folge dem Tag, damit passende Updates in deinem Orbit bleiben.
1Gezeigte Loots
0Gezeigte Artikel
6Verlinkte Nachbar-Tags
Anschluss-Themen
Wenn du tiefer einsteigen willst, helfen die benachbarten Tags beim Vergleichen und Querlesen.
Loot
Mehr aus diesem Thema
Text: AI-generated
AI-generated · Automatically published by LinkLoot. STROT compiles your classification or routing intent with an LLM once, then serves bounded JSON decisions locally forever — no runtime API calls, 60 ms per record. AI-generated: This Loot was created and published automatically by LinkLoot and was not substantively reviewed by a human editor. What it does STROT (arXiv:2505.01636) lets you define a decision task in natural language plus labeled examples, then uses an LLM exactly once at compile time to synthesize a frozen Python artifact. At production time a small local model (SupersonicLabs/Julia-1, 144M parameters, CPU-only) arbitrates every incoming record and returns type-safe JSON with probabilities and confidence scores. Low-confidence records abstain rather than guess. Two core APIs drive the workflow: run() routes each record to a compiled handler action; decide() answers typed questions per record — boolean flags, numeric scores within bounds, or choice selections from a fixed list. Output is selection and computation over your own registry, so structural hallucination at runtime is impossible by design. The compiler pipeline includes a golden-record gate (artifacts must pass your known answers before serving), temperature scaling, split conformal prediction for calibrated confidence guarantees, and optional distilled heads trained only when they beat baselines on held-out data. Benchmarks published in the repository report 92.5% grounded routing accuracy versus 74.8% naive prompting on 398 records (+18pp), 86.7% on AG News unseen labels versus 72.7% (+14pp), roughly 60 ms latency, $0 marginal cost, and zero LLM calls per record. Who it helps Teams processing high-volume support ticket triage, content moderation queues, CRM lead scoring, log anomaly routing, or any repetitive classification/routing task where hosted LLM inference costs scale linearly with volume. It suits engineers who want deterministic, auditable decisions without vendor lock-in and can tolerate one-time compile cost in exchange for free ongoing inference. Getting started Install via PyPI: pip install strot-ai. Requires Python 3.9+. Define your intent as a schema projection with label descriptions and sample records, specify an LLM for compilation (e.g., Anthropic Claude models), call the compile step once, then invoke strot.run() or strot.decide() on production records. Content-addressed caching means schema drift triggers exactly one recompile. Reproducible benchmark code lives in the benchmarks/ directory of the repository. Limits and costs Compilation requires one paid LLM call whose cost depends on the chosen provider and token usage; hosting the compiled artifact on your own hardware incurs standard compute expenses but no per-request API fees. The bundled decision model runs on CPU; GPU acceleration is not documented. Confidence calibration assumes representative training samples — poorly covered edge cases may produce higher abstention rates. Pricing for external LLM providers during compile remains outside this project's scope. Source links Official repository: https://github.com/strot-ai/Project-STROT Project documentation: https://github.com/strot-ai/Project-STROT/blob/main/README.md License (MIT): https://github.com/strot-ai/Project-STROT/blob/main/LICENSE
- LinkLoot access
- Free
- Provider costs
- Unknown
Review open
Blog
Verwandte Artikel
Noch kein Blogpost zu #cpu-inference
Aktuell gibt es keinen veröffentlichten Artikel mit diesem Tag. Schau im Blog nach verwandten Themen oder folge dem Tag für spätere Updates.