Why conventional AI safeguards may be insufficient
As AI systems and large language models move into critical environments, software quality alone cannot address adversarial manipulation, privilege escalation, unsafe actions, data exfiltration or systemic failures. This proposal organizes the problem into two tiers: foundational trustworthy AI and governance requirements, and a confined high-assurance architecture inspired by mission-critical engineering.
The framework is intentionally stringent. Its architecture describes a restricted deployment profile for extreme-risk environments, not a universal design for all AI products.
Part I — Seven foundational pillars of trustworthy AI and cybersecurity
Pillar 1: Technical Robustness and Cybersecurity
- Adversarial Robustness: Systems must enforce defenses against prompt injection (direct and indirect), jailbreaking, evasion techniques, model evasion, and training data poisoning.
- Supply Chain Security: Organizations must establish formal provenance tracking for base models, foundation checkpoints, weights, third-party libraries, and serialized dependencies (such as pickle files), mitigating unauthorized modifications and vulnerable dependencies.
- Input and Output Sanitization: Input prompts and model-generated responses must never be implicitly trusted. Environments must implement runtime content filtering, structural validation, and strict sandboxing for automated API calls and code execution environments.
- Safe State and Graceful Degradation (Fail-Safe): When encountering extreme uncertainty, unexpected anomalies, or active exploitation, the system must transition into a predictable, safe state without cascading failures or data leakage.
Pillar 2: Data Governance and Privacy
- Data Quality and Integrity: Data corpora used for pre-training, fine-tuning, retrieval-augmented generation (RAG), and evaluation must be statistically representative, validated, and free of known systematic corruptions.
- Privacy by Design and Exfiltration Prevention: Architecture must prevent sensitive information and Personally Identifiable Information (PII) disclosures caused by inference extraction, membership inference, or model inversion attacks.
- Lineage, Consent, and Licensing: Provenance, legal basis of collection, intellectual property licenses, and user consent must be recorded and auditable across all dataset pipelines.
Pillar 3: Transparency, Explainability, and Documentation
- Interaction Disclosure: Systems must inform human users that they are interacting with an artificial system and convey its design boundaries and known performance limitations.
- Reasoning and Interpretability: In high-consequence domains (e.g., healthcare, criminal justice, financial assessment), decision pathways, confidence metrics, and contextual rationales must be explainable in human-understandable terms.
- Standardized System Cards: Organizations must maintain and publish detailed model documentation (e.g., Model Cards, System Cards) covering training methodologies, known edge-case failures, intended use cases, and out-of-scope deployments.
Pillar 4: Human Oversight and Governance
- Continuous Intervention Capabilities (HITL / HOTL): Qualified personnel must retain the capability to supervise operations, adjust intermediate parameters, override erroneous inferences, or abort operations in real time.
- Mitigation of Automation Bias: Interfaces must be structured to prevent human operators from reflexively trusting machine inferences without verification.
- Deterministic Emergency Stop (Kill Switch): Hardware- and software-level kill mechanisms must be capable of severing execution threads or isolating the model without generating cascading operational hazards.
Pillar 5: Fairness, Bias Management, and Non-Discrimination
- Systematic Bias Auditing: Continuous evaluation across protected classes to detect and mitigate algorithmic biases that lead to unlawful or disparate impacts.
- Metric-Driven Parity: Systems must undergo pre-deployment and continuous testing against formal fairness and parity metrics relevant to the application domain.
Pillar 6: Continuous Security Testing and Operational Guardrails
- Adversarial Red Teaming: Independent red teams must carry out adversarial evaluations to identify boundary degradation, prompt bypasses, and logical flaws prior to deployment.
- Contextual Guardrails & Content Gateways: Intermediary safety filters and deterministic guard models must inspect and restrict hazardous topics, including weapons design, cyber exploits, and mass deception.
- Model Drift and Anomaly Monitoring: Continuous telemetry to identify shifts in runtime distributions, semantic drift, and performance regressions over time.
Pillar 7: Accountability, Versioning, and Audit Trails
- Immutable Logging: Cryptographically verifiable logging of user inputs, outputs, automated actions, error traces, and configuration parameters to ensure forensically viable audits.
- Full Reproducibility: Comprehensive versioning of code, model weights, parameter checkpoints, runtime dependencies, and training sets across the machine learning operations (MLOps) lifecycle.
Part II — Confined and deterministic high-assurance architecture
For extreme operational, national security or life-safety consequences, the proposed enclave limits resources, accepts supervised input-only data and separates model suggestions from physical or consequential actions.
- Supervised input through a physical diode or transient link
- Resource-capped hardware host with no radio interfaces
- Sandboxed deterministic execution boundary with no outbound sockets
- Suggested action / unexecuted result
- Non-delegable expert review and manual authorization before action
Boundary of the claim: physical isolation constrains selected channels, but does not by itself eliminate all side channels, malicious input, operational mistakes or unsafe model output. The source framework expresses stronger isolation ambitions; these require system-specific verification.
Five confined architecture requirements
Requirement A: Resource Capping and Anti-Convergence Controls
Constraint: Systems must never be allocated unconstrained compute, memory, disk, or network privileges.
Technical rationale in the proposal: According to Nick Bostrom's Instrumental Convergence thesis, an unconstrained autonomous agent naturally attempts to secure additional compute, bandwidth, and persistence to minimize failure risk. Applying the Principle of Least Privilege and strict hard quotas (CPU/RAM/VRAM caps, kernel-level container cgroups, hypervisor memory locks) neutralizes resource hoarding and local denial-of-service risks.
Requirement B: Unidirectional Data Flow, Radio Frequency (RF) Elimination, and AI Boxing
Constraint: Outbound network communication must be blocked. The system may ingest data from the exterior, but it cannot transmit data outward. All wireless interfaces (Wi-Fi, Bluetooth, cellular, near-field) must be physically absent or hardware-disabled.
Technical rationale in the proposal: Hardware Data Diodes: Using physical optical data diodes (where the receive line is active but the transmit optical fiber is physically severed) ensures through physical laws that data exfiltration is impossible. AI Boxing: As articulated in containment literature (e.g., Eliezer Yudkowsky’s AI Box theory), software-only egress filtering remains vulnerable to exploit chaining; true isolation requires hardware-enforced boundaries. TEMPEST & SCIF Directives: Mandating zero RF emission eliminates wireless side-channel attacks and acoustic/RF covert channels within high-security environments.
Requirement C: 100% Local, Single-Operator Air-Gapping
Constraint: Model inference and execution must run locally on air-gapped, on-premises hardware without cloud telemetry or external API calls. Any periodic data ingestion must be conducted via supervised, manual physical connections (such as connecting and immediately disconnecting an isolated Ethernet cable or physical unidirectional medium).
Technical rationale in the proposal: Air-gap isolation protects the local runtime from external drive-by exploits and automated lateral movement, ensuring that configuration state and operational memory remain accessible only to the authorized physical operator.
Requirement D: Non-Delegable Human Final Authority (Strict HITL)
Constraint: AI outputs must be treated exclusively as non-binding suggestions. No AI model may trigger downstream systems, real-world actuators, financial disbursements, or legal judgments autonomously. Human authorization is mandatory for every action, without exception.
Technical rationale in the proposal: Models are fundamentally non-deterministic and subject to hallucination and adversarial manipulation. Retaining Meaningful Human Control aligns with high-risk operational safety doctrines and ensures that legal responsibility and ultimate agency remain with human operators.
Requirement E: High-Assurance Software Verification (DO-178C Paradigm for AI Code)
Constraint: Code generated by an AI assistant or autonomous model must be rejected unless an authorized Senior Software Engineer (or certified engineering team) actively validates, verifies, and assumes legal responsibility for every single line of code. Automated code-generation tools that generate undocumented multi-file codebases lacking comments, unit tests, architectural diagrams, and traceability must be prohibited.
Technical rationale in the proposal: Critical software domains—such as avionics under DO-178C (Software Considerations in Airborne Systems and Equipment Certification)—require Modified Condition/Decision Coverage (MC/DC) and bidirectional requirements-to-code traceability. Unverified AI code introduces architectural erosion, hallucinated dependencies, hidden race conditions, and unmaintainable technical debt. Human domain expertise and line-by-line review remain necessary to verify safety, determinism, and correctness.
Regulatory and standards concordance
The following references are used as engineering and governance context. They are not all directly applicable to every deployment, and their inclusion does not imply legal compliance or certification.
| Requirement area | Standard or framework | Referenced controls or provisions |
|---|---|---|
| Trustworthy AI, Governance & Risk | NIST AI Risk Management Framework (AI RMF 1.0) | Functions: GOVERN, MAP, MEASURE, MANAGE |
| High-Risk AI System Requirements | European Union Artificial Intelligence Act (Regulation (EU) 2024/1689) | Chapter III, Section 2 (Articles 9–15: Risk, Data, Documentation, Logs, Transparency, Oversight, Robustness) |
| Secure AI Engineering & Lifecycle | UK NCSC & US CISA Guidelines for Secure AI System Development | Secure Design, Secure Development, Secure Deployment, Secure Operation |
| LLM & Generative Application Flaws | OWASP Top 10 for Large Language Model Applications | Risks: Prompt Injection, Insecure Output Handling, Supply Chain, Unbounded Consumption |
| AI Management Systems (AIMS) | ISO/IEC 42001:2023 Information Technology — AI Management System | Clauses 5–10; Annex A Controls (Risk Management, Data Lifecycle, Operations) |
| Resource Quotas & Least Privilege | NIST Special Publication 800-53 Revision 5 | Controls AC-6 (Least Privilege) & SC-6 (Resource Availability / Allocation) |
| Automated Decision Restrictions | EU General Data Protection Regulation (GDPR) — Article 22 | Prohibition of solely automated decision-making producing significant or legal effects |
| Meaningful Human Oversight | U.S. Department of Defense Directive 3000.09 (Autonomy in Weapon Systems) | Mandated human judgment over use of force; rigorous T&E/V&V validation |
| Safety-Critical Software Verification | RTCA DO-178C / EUROCAE ED-12C Airborne Software Certification | Structural Coverage (MC/DC), Traceability, Tool Qualification (DO-330) |
| International AI Ethical Principles | OECD AI Principles on Responsible Stewardship of Trustworthy AI | Principles 1.1–1.5 (Human-centered values, transparency, robustness, accountability) |
| Global Ethical Recommendations | UNESCO Recommendation on the Ethics of Artificial Intelligence | Values & Policy Actions (Human oversight, privacy, safety, and proportionality) |
Some mappings are analogies rather than direct requirements; consult the linked primary sources and qualified regulatory specialists for applicability.
Frequently asked questions
What is high-assurance AI?
A high-assurance AI approach combines trustworthy AI governance and cybersecurity requirements with explicitly constrained, inspectable and independently verified system architecture.
What are the seven pillars of trustworthy AI?
Technical robustness and cybersecurity; data governance and privacy; transparency and documentation; human oversight; fairness; continuous security testing; and accountability and audit trails.
What is a confined AI architecture?
A confined AI architecture restricts resources, network access and execution privileges, and places human authorization between model output and consequential real-world actions.
Can an air gap or data diode guarantee AI safety?
No. Isolation can reduce particular communication and attack paths but does not by itself guarantee safe outputs, absence of side channels, or secure human handling.
Does DO-178C certify AI systems?
DO-178C is an airborne software assurance standard. This document invokes its verification paradigm as an analogy; it does not claim that following the proposed framework confers certification.
Help examine the limits of AI containment
Challenge the architecture, scrutinize the assurance claims, propose verification methods and contribute to the Glass Box AI research community.
JOIN THE DISCORD COMMUNITY