AI, Platform, Audit, and Security Operations — Technical reference

This reference owns the implementation contracts of BND-AIOPS. Read the package guide for usage, configuration, dependencies and upgrades. Generic execution, isolation and admission remain governed by Platform Documentation.

Data model and interfaces

DataModelPackage: DM-AIOPS

Owned entities: model and agent registry record, evaluation suite and run, deployment, canary, health finding, capacity snapshot, incident, quarantine.

InterfacePackage contributions: IF-PLATFORM-ADMIN.

Each business object and source reference remains tenant- and purpose-scoped. The package owns schema evolution and declared migration behavior; the platform enforces schema registration, authorization and versioned installation. Consumers reference this model rather than creating competing canonical copies.

Workflows

WorkflowPackage: WFP-AI-PLATFORM-OPERATIONS

• Package Qualification
• Model and Agent Evaluation
• Deployment and Canary
• Quality and Drift Monitoring
• Pipeline Health
• Capacity and Cost Monitoring
• Automatic Hard-Gate Suspension
• Authorized Rollback
• Processing-Job Recovery
• Quarantine Resolution
• Security-Event Investigation
• Audit Reconstruction

These are named definitions within the workflow package, not separate invented workflow IDs. Workflow runs resolve package versions, actor and purpose, authorized inputs, retries, review gates and output schemas through the environment lock. Temporal provides durable execution and Kafka carries declared lifecycle/domain events.

Policies, evaluations and tools

POL-PACKAGE-ADMISSION, POL-AI-DEPLOYMENT, EVAL-PACKAGE-QUALIFICATION, telemetry, registry, sandbox, deployment, rollback, and quarantine tools.

Qualification covers the selected model/runtime profile, languages, access scope, failure handling and the package's domain-specific outcomes. Tool execution uses the platform gateway; package code cannot increase its own permissions or write directly to external systems.

Owned agents

P06 — Model and Agent Qualification Agent

Agent ID: effority.extension-qualification

Purpose: Executes the documented evaluation suite for Effority and customer ModelPackage and AgentPackage versions and produces a promotion recommendation.

Triggers: Package registration, new version, dependency change, evaluation-set update, policy change, or scheduled reevaluation.

Inputs: Signed package, manifest, SBOM, evaluation suites, baselines, compatible schemas, runtime profile, and tenant-specific thresholds.

Outputs: Integrity, security, compatibility, quality, permission, leakage, latency, resource, and rollback test results with a pass, fail, or restricted recommendation.

Tools: package.inspect, sandbox.execute, evaluation.run, baseline.compare, vulnerability.scan, and registry.write_evaluation.

Review and limits: It does not approve its own package or promote a production deployment. An authorized administrator records the deployment decision.

Primary metrics: Gate coverage, reproducibility, escaped regressions, false gate failures, evaluation duration, and baseline deltas.

P07 — AI Operations Agent

Agent ID: effority.ai-operations

Purpose: Detects operational and quality anomalies across models, agents, queues, hardware, canaries, and review outcomes and proposes a bounded operational response.

Triggers: Metric threshold, drift signal, canary comparison, run failure pattern, GPU anomaly, quality regression, or scheduled health analysis.

Inputs: Aggregated telemetry, evaluation history, deployment state, tenant budgets, review metrics, baseline, and incident rules.

Outputs: Health finding, probable cause, affected scope, recommended pause, rollback, capacity change, evaluation, or investigation draft.

Tools: telemetry.query, deployment.read, evaluation.read, run.sample_metadata, incident.draft, and rollback.request.

Review and limits: It does not read literal content through operations telemetry. Automatic suspension occurs only for preapproved hard safety gates; production rollback otherwise requires authorized confirmation.

Primary metrics: Detection lead time, false alarm rate, mean time to diagnosis, quality-regression escape rate, and rollback effectiveness.

Implementation profiles

MLOps and Model Lifecycle

The model registry retains:

• logical name and purpose;
• origin, owner, OCI artifact digest, cosign signature, in-toto/SLSA provenance, SPDX bill of materials, and tenant visibility;
• exact repository and revision;
• weight and container hash;
• license and restrictions;
• runtime, quantization, and hardware;
• evaluated languages and domains;
• global and segment-level metrics;
• limitations card;
• date, approvers, and authorized tenants;
• compatible prompts, schemas, rules, and calibrators.

The same supply-chain contract applies to ModelPackage, AgentPackage, and their referenced tool, prompt, policy, skill, taxonomy, schema, and evaluation OCI artifacts. The supporting artifacts are not additional ExtensionPackage variants. Effority production artifacts use isolated ephemeral builds with SLSA 1.2 Build Level 3 provenance. Customer artifacts remain quarantined until signature, provenance, bill-of-materials, vulnerability, license, isolation, and evaluation checks pass. Public runtime downloads, floating tags, arbitrary remote code, and unrestricted dynamic LoRA loading are disabled.

The four isolated promotion nodes are development → testing → staging → production. Evaluation and qualification run in Testing, shadow validation runs in Staging, and canary is the first controlled rollout state inside Production. These are lifecycle stages within the four-node topology, not additional environments. A version does not advance if it degrades a critical threshold, increases uncited claims, breaks a language, or fails permission tests. The same signed artifact digest moves between nodes; it is never rebuilt after qualification. Rollback restores the prior approved environment lock, including weights, prompt, schema, calibrator, agents, interfaces, policies, tools, and compatible data migrations as one traceable release graph.

Every change is represented by an integrated, versioned ChangeSet linked to the Git-compatible source revision, immutable OCI artifacts, dependency locks, evaluation evidence, approvals, node-by-node promotion history, before-and-after environment locks, and rollback target. A rollback creates a new immutable audit record and never removes the reverted release or its evidence.

The system never retrains automatically on production content. Corrections enter a separate feedback store; only specifically authorized data enters a curated, reviewed, versioned dataset, with deidentification where appropriate.

Expand screen ↗

Production AI operations with model registry, pipeline health, evaluation gates, canary rollout, rollback, GPU utilization, recent runs, and audit log.

Automated AI Evaluation

Every build runs:

• WER/CER, timestamp, and language-detection tests;
• DER/JER and speaker-identity accuracy;
• precision, recall, F1, and calibration by label;
• entity-linking accuracy;
• citation faithfulness and groundedness;
• coverage of actions, decisions, and commitments;
• contradiction and longitudinal tests;
• ethics false-positive and false-negative tests;
• cross-tenant leakage and permission tests;
• prompt injection and adversarial-document tests;
• latency, real-time factor, memory, and throughput tests;
• template and export nonregression tests.

The evaluation service uses Telynt's canonical EvalSuite, EvalCase, and EvalRun contracts. Inspect AI is the primary evaluation/sandbox adapter; PyRIT, garak, and AgentDojo provide red-team, vulnerability, and indirect prompt-injection campaigns. Metamorphic, counterfactual, difficult-negative, multilingual, replay, dependency-compromise, forged-handoff, exfiltration, and adaptive defense-aware tests supplement static benchmarks. An LLM judge is a secondary, human-calibrated metric and never the sole promotion gate.

Hard gates require zero cross-tenant leakage, unauthorized writes, policy bypasses, and missing mandatory approvals. Quantized variants, LoRA adapters, inference optimizations, authoring adapters, and retrieval profiles pass the same task, evidence, permission, calibration, latency, resource, resilience, and adversarial evaluation as their reference deployment.

The corpus is partitioned by meeting so excerpts from the same conversation never appear in both training and test sets. Results are linked to the model artifact and available from the operations panel.

AI Observability and Operations

OpenTelemetry SDKs, Collector, OTLP, and W3C Trace Context propagate trace_id and correlation_id across UI, API, Temporal workflow, event bus, agent, retrieval, model, and tool workers. Telynt pins a compatibility profile for the evolving OpenTelemetry GenAI semantic conventions. Technical metrics are exported to Prometheus and Grafana dashboards; structured logs reach the SIEM without literal text. OpenInference is an optional export adapter, not the internal telemetry schema.

Operational telemetry excludes prompts, transcripts, retrieved excerpts, tool arguments, credentials, and model output by default. Tenant-authorized diagnostic content is separately encrypted, redacted, access-controlled, and time-limited. Tail sampling retains all denials, extraordinary access, cross-tenant probes, restricted actions, and failures.

Operational alerts include:

• queue depth and age;
• errors and retries by stage;
• GPU OOM, temperature, and utilization;
• ASR and diarization real-time factor;
• time to first token and tokens per second;
• invalid JSON and repair rate;
• coverage or confidence drops;
• changes in label distribution;
• increased human-rejection rate;
• unsupported or blocked queries;
• external synchronization failures.

The autoscaler uses queue depth, waiting and running requests, RTF, input/output token rate, KV utilization, time to first token, time per output token, and accelerator memory. Batch workers fill GPUs through continuous batching; real-time workers have reserved capacity and priority. Frequently used models remain warm, while occasional-use models load in separate pools. Large-model profiles use inference-aware endpoint selection, cache-aware routing, tiered KV, and independently scaled prefill/decode pools only when production traces justify their overhead.

Integration boundaries

P06 and P07 propose and assess; deterministic policy and authorized humans control admission and deployment. Platform administration is separate from solution business administration.

Owned screen contracts: SCR-ADM-05, SCR-ADM-06, SCR-ADM-07.

See solution workflows for cross-package behavior. Screens, agents and workflows keep one canonical owner even when other bundles invoke or enrich them.