AI Skill Supply-Chain Security

How Super Agent Skill secures the path from a published skill to your running agent: cryptographic signing, adversarial testing, transparent trust scoring, and continuous re-evaluation.

Abstract

AI agents increasingly load third-party instructions — skills, playbooks, personas (souls) and guardrails — at runtime. These packages are natural-language programs: they direct model behavior with the same authority as first-party prompts. That makes the skill registry a supply chain, and an unsigned, untested skill registry is the npm of 2010: convenient, and one compromised package away from an incident. This document describes the threat model we design against and the controls the platform applies at publish time, at install time, and continuously after release.

1. Threat model

1.1 Prompt injection via package content

A skill's markdown body is fed to the model. A malicious or compromised package can embed instructions that override the agent's system prompt, exfiltrate conversation context, or redirect tool calls. Unlike code, the payload is prose — static scanners built for binaries do not see it.

1.2 Malicious or backdoored skills

An attacker publishes a package that behaves correctly in demos but leaks PII/PHI, gives unlicensed financial or medical advice, or bypasses stated policies under adversarial phrasing. The failure mode is behavioral, so it only appears under attack — which is why we test with attacks, not just happy-path examples.

1.3 Supply-chain tampering

Classic package-registry threats apply directly: account takeover of a publisher, modification of package content between review and delivery, a compromised registry or CDN serving different bytes than were tested, and version-substitution attacks where a benign version is reviewed but a different one is installed.

1.4 Trust decay

A package that passed testing a year ago may fail today: attack techniques evolve, the underlying models change, and maintainers drift. A one-time "verified" badge is a liability if nothing re-earns it.

2. Mitigations

2.1 Ed25519-signed releases (vs. tampering)

Every published version is signed with an Ed25519 release key. The signature covers the exact content bytes; verification (npm run release:verify) is a dependency-free Node script that runs offline. This defeats in-transit and at-rest tampering: what your security team reviewed is bit-for-bit what your agent loads, and a compromised registry cannot forge a valid signature.

2.2 Adversarial harness (vs. injection and malicious behavior)

Before publish, every package is benchmarked against a curated adversarial suite: prompt injection, jailbreaks, data exfiltration, blast-radius, policy bypass, and PII/PHI leakage cases, organized per vertical (OWASP LLM, FINRA, HIPAA Safe Harbor, PCI-DSS, SRE). Outcomes are judged by an LLM judge rather than keyword matching, because keyword heuristics miss polite-but-compliant answers and over-fire on outputs that merely quote refusal language. Each case carries a severity (low 1x, medium 2x, high 4x, critical 8x), producing both a pass rate and a severity-weighted score.

2.3 Transparent Trust Score (vs. blind trust)

The Trust Score is a public weighted formula over objective signals — adversarial severity-weighted score (25%), adversarial pass rate (20%), real-world success rate (20%), schema validity (10%), signed releases (10%), package age (5%), owner 2FA (5%), contributor count (5%). Two properties matter for auditors: unproven signals default to a conservative 0.3 prior rather than a neutral 0.5, so untested packages cannot float into "acceptable" for free; and measured evidence decays linearly toward that prior over 180 days, so no package coasts on a stale benchmark. Every score carries a confidence value stating how much of it is evidence-backed versus assumed. The implementation is open source (src/lib/trust/score.ts).

2.4 Signed trust attestations (vs. trusting our endpoints)

npm run trust:attest produces a JSON attestation binding a package's SHA-256 content hash and version to its adversarial result, canonicalized (recursively key-sorted) and signed with the release key. A consumer verifies it with npm run trust:verify using only the package file, the attestation and the publisher's public key — no network, no trust in the hosted Trust Score endpoint. The verifier checks the key fingerprint, recomputes the content hash, and validates the Ed25519 signature; any mismatch exits non-zero.

2.5 SkillForge continuous re-testing (vs. trust decay)

Published packages are periodically re-scored against the live adversarial suite and against anonymized execution telemetry (success rate, latency, drift, critical findings — visible on the public Evaluation page). When robustness drops, the package is flagged and SkillForge proposes patches. Evidence-age decay in the Trust Score formula enforces the same principle numerically.

2.6 Guardrails and souls (defense in depth)

Behavioral constraints ship as first-class, individually tested packages: guardrails such as no-pii-in-output and no-medical-advice enforce output policy, while compliance souls (Fintech Compliance Officer, HIPAA-Aware Clinical Liaison, SOC 2 Type II Auditor) constrain persona-level behavior for regulated verticals. A skill failure is contained by the layers around it.

3. Verification workflow

A recommended evaluation flow for a security team assessing a package:

1. Review the public trust page at superagentskill.com/marketplace/trust/<slug>: adversarial pass rate, signature status, Trust Score breakdown and confidence.
2. Obtain the package file and its signed attestation (<slug>.trust.json).
3. Verify offline: npm run trust:verify -- --attestation pkg.trust.json --pkg pkg.yaml --pubkey pub.pem. A non-zero exit means tampering or an invalid signature.
4. Cross-check the attested content_sha256 against your own hash of the file you intend to deploy.
5. Optionally re-run the adversarial suite yourself: the harness (scripts/eval-adversarial.mjs) and case corpus (content/adversarial/) are open source.
6. Monitor post-deploy via the Evaluation page; treat critical findings or drift as a re-review trigger.

4. Shared-responsibility model

5. Conclusion

No single control secures a natural-language supply chain. Signing proves what you got; adversarial testing proves how it behaves under attack; transparent scoring makes the evidence auditable; attestations make it verifiable without trusting us; continuous re-testing keeps all of it current. Each layer is independently inspectable — and the parts your auditors care about run offline, on your side of the boundary.