Model and Inference Services

The platform hosts controlled model execution and qualified runtime profiles for installed ModelPackage and ToolPackage capabilities. It does not require every installation to transcribe meetings, detect contractual scope or generate professional recommendations.

Runtime boundary

The Model Gateway selects an authorized, qualified capability by task contract, budget, tenancy and environment lock. Exact model, runtime, quantization and adapter versions remain traceable. Domain taxonomies, output meaning and evaluation datasets belong to the consuming package.

Local Language-Model Inference

Language models are served behind the versioned, vendor-neutral Telynt inference gateway. The gateway also exposes an internal OpenAI-compatible adapter, while agents depend only on stable task and capability contracts. The standard GPU installation uses vLLM. Qualified profiles use SGLang for benchmarked shared-prefix and branching workloads, TensorRT-LLM for NVIDIA Hopper/Blackwell optimization, llama.cpp/GGUF for CPU and Apple Silicon, OpenVINO GenAI for Intel CPU/GPU/NPU, ONNX Runtime GenAI for Windows/DirectML/QNN/ARM, and llm-d with the Kubernetes Gateway API Inference Extension for large distributed models.

The runtime capability contract declares chat, responses, embeddings, reranking, multimodal input, structured JSON, grammar, tool calls, log probabilities, prefix caching, cache salting, KV offload, speculative decoding, disaggregated prefill/decode, LoRA, quantization, and parallelism. Each deployment records only the capabilities proven for its exact engine, model, hardware, driver, and configuration.

Deployed profiles:

Every weight is pinned by repository, revision, checksum, license, quantization, tokenizer, chat template, runtime image, attention backend, grammar backend, LoRA adapter, cache configuration, and hardware matrix. The router selects a profile according to sensitivity, language, length, complexity, SLA, evaluated quality, cache locality, and available memory. There is no automatic fallback to a public API: an external route runs only when tenant policy explicitly authorizes it.

Production profiles use continuous batching and runtime-qualified paged attention. Prefix/KV caching is partitioned by tenant trust domain, purpose, policy epoch, model, tokenizer, template, prompt, schema, LoRA adapter, evidence-set hash, modality hashes, and cryptographic salt. Cross-tenant KV reuse is prohibited. Speculative decoding, FP8/FP4/INT4 quantization, KV offload, multi-LoRA, and disaggregated prefill/decode are enabled only after representative Telynt traces prove a quality-preserving latency or throughput improvement.

Prompts are versioned templates containing:

• purpose and taxonomy;
• positive and negative definitions;
• JSON schema;
• numbered sources;
• abstention rules;
• a prohibition on filling in missing information;
• approved examples and counterexamples;
• input and output language;
• severity and recipient boundaries.

Responses use structured outputs or JSON grammars. XGrammar is the preferred constrained-decoding backend on compatible qualified runtimes; llama.cpp and ONNX Runtime use their native grammar layers. Grammars compile and warm before production traffic. A validator rejects out-of-schema fields, nonexistent references, unsupported citations, incoherent dates, and subjects absent from the input. Grammar constraints guarantee syntax, not truth; semantic validation, citation entailment, authorization, and business rules always run afterward. Semantic validation failures are retried once with a repair prompt and then enter NEEDS_REVIEW.

Execution Profiles

The selected profile is fixed in the ModelRun, so every result remains reproducible after the default profile changes.

Confidential Computing and Federated Learning

The advanced-privacy module is implemented as a restricted profile:

• Nodes compatible with confidential computing generate hardware and workload attestation; the KMS releases a tenant key only when the measurement matches an approved image.
• Data remains encrypted outside the enclave and is decrypted only during authorized execution.
• Federated learning exchanges updates to adapters, classifiers, or calibrators, never audio, transcripts, or literal examples.
• Secure aggregation prevents the coordinator from observing an individual contribution.
• Tenants participate through a separate opt-in and limit tasks, categories, and periods through policy.
• Aggregated artifacts undergo the full evaluation again before promotion.
• Differential privacy is applied to metrics or updates for profiles whose use case tolerates its quality impact.

Federated learning is not used as a shortcut around authorization: an update derived from data remains processing and requires policy, traceability, and evaluation.

Performance and Scalability

• Sizing by audio hours per day and peak concurrency.
• Priority queues and backpressure.
• Horizontal scaling of independent processes.
• Checkpoint resume.
• Incremental and selective processing.
• Latency budget by stage.
• Limits by organization, customer, and source.

Resilience

• Idempotent ingestion.
• Bounded retries and error queue.
• Graceful degradation: retain the transcript if later analysis fails.
• Worker or GPU failure recovery.
• Backup and disaster-recovery tests.
• RPO/RTO objectives by data type.

Portability and Exit

• Export of data and metadata in documented formats.
• Deletion or return at contract end.
• Avoidance of irreversible dependency on one model.
• Schema versioning and auditable migrations.

---

Package implementation profiles

Speech processing is documented in Meeting Intelligence, ingestion and OCR in Evidence, finding contracts and calibration in Findings, and cited question answering in Search. Operational qualification and monitoring interfaces belong to AI & Platform Operations.

Authoring

See agent and model contracts for creating replaceable packages and evaluation profiles.