Methodology

Principles

1. Public sources only. Every signal is extracted from a publicly accessible source: GitHub REST API, PyPI JSON + simple-index, the sigstore Rekor transparency log, or files committed to the public 273v/kaos-* repos. No private endpoints, no privileged credentials, no internal-only artifacts.
2. JSON is the source of truth. The dashboard’s HTML is one render of data/snapshots/latest.json. The same JSON is published for compliance ingest at https://273v.github.io/kaos-compliance/api/v1/snapshot.json.
3. Every claim links to evidence. Each card on the dashboard carries a Verify link that opens the underlying source (a GitHub workflow run, a PyPI metadata page, a Rekor log entry, or a file in this repo).
4. No invented scores. We surface the OpenSSF Scorecard aggregate (a pre-composited industry-standard number) and the underlying signals. We do not compute a competing composite. Buyers compose their own bar from the per-signal evidence.
5. Stale data is loudly marked stale. Every snapshot carries a generation timestamp; the dashboard shows a freshness indicator on every card. If the cron stops running, the indicator goes amber within 2 hours and red within 24.

Cadence

Cron is driven primarily by GitHub Actions on this repo. A local cron job on the developer machine acts as a backup and runs the LLM diary when the GHA secrets don’t have the model-provider API key wired.

Public PR and CI/CD hardening policy

The KAOS package repos are public, so the dashboard treats outside PRs as an explicit trust boundary rather than a convenience feature. The policy baseline for public 273v/kaos-* repos and this compliance repo is:

• untrusted PR code runs only from pull_request, never pull_request_target with contributor code checked out;
• the default GITHUB_TOKEN is read-only;
• workflows declare explicit permissions:;
• PR build/test jobs have no secrets, no id-token: write, no package-publish authority, and no contents: write;
• checkout uses persist-credentials: false;
• external Actions references are pinned to full commit SHAs;
• fork PR workflow runs require approval for all external contributors;
• main requires status checks, CODEOWNER review, stale-review dismissal, approval of the most recent push, linear history, and no force-pushes;
• v* release tags are protected against deletion and force-update;
• secret scanning and push protection are enabled.

For kaos-compliance specifically, external fork PR code is not an accepted contribution path and is not executed in CI. GitHub does not allow disabling forks for an org-owned public repository, so the repo uses policy plus workflow guards: public issues and private security reports remain open, while CI jobs run only for main and PR branches whose head repository is 273v/kaos-compliance.

This policy is documented in full in kaos-modules/docs/oss/40-ci-cd/public-pr-security.md. The compliance dashboard does not currently render these repository settings as a green public claim, because several are visible only through GitHub admin APIs. The dashboard may surface workflow-file checks that are publicly reproducible, such as pinned uses: references and explicit workflow permissions. Admin-only settings are audited with the runbook commands in docs/RUNBOOK.md until GitHub exposes a public evidence path.

The kaos-compliance sweep workflow is a trusted main/schedule workflow, not an untrusted PR workflow. It still has a tracked trust-lane follow-up: split collection/rendering, keyless signing, final render, and Pages deploy into separate jobs so dependency installation never shares a job token with contents: write or id-token: write. Until that lands, the snapshot signature proves which workflow produced the JSON, but it does not prove that the workflow followed the least-privilege job split described here.

Frameworks anchored

This dashboard maps signals to:

• OpenSSF Scorecard v5 — all 19 checks.
• SLSA 1.0 — Build L1–L3, Source L1–L3 (Source track still draft as of 2026-05).
• NIST SSDF (SP 800-218) — the 12 practice IDs with publicly evidenceable signals.
• CISA SBOM Minimum Elements.
• PEP 740 + sigstore + PyPI Trusted Publishers.
• Cyber Resilience Act (Annex I Part II conformity signals; full enforcement guidance not expected until 2027).

See docs/research/01-compliance-signal-inventory.md for the per-framework signal-to-source mapping table.

NIST SSDF practice-ID matrix

The dashboard surfaces signals against the publicly-evidenceable subset of NIST SP 800-218. Practices whose evidence is internal-only (e.g. PO.5.1 “Implement and maintain secure development environments”) are deliberately omitted — the public-source-only contract can’t reach them.

CRA Annex II / Article 13 traceability

The EU Cyber Resilience Act, as drafted, requires manufacturers to maintain technical documentation (Annex II) covering specific conformity signals (Article 13). The mapping below names which signals satisfy which clause; clauses without a verifiable public signal are surfaced as honest gaps rather than silently claimed. CRA full enforcement guidance is not expected until 2027 — this matrix will be revised against the final implementing acts when published.

CVE / advisory feed sources

The Security page’s “0 open advisories” claim is only meaningful when the feed sources are named. The dashboard cross-references:

• OSV.dev — https://api.osv.dev/v1/query, keyed by PURL (pkg:pypi/<name>@<version>, pkg:cargo/<crate>@<version>). Cursor: live query at sweep time; no snapshotting.
• GitHub Security Advisories — gh api /advisories filtered by affected ecosystem and package name. Cursor: live; deduplicated against OSV using the GHSA ID.

Both feeds are queried during the 4-hour Security cron. A package clears the “0 advisories” bar only when both feeds return empty for its declared version. The OSV PURL form is canonical; if a package publishes under a non-PyPI distribution name the lookup may miss — a known gap, tracked in docs/research/08-followup.md.

Anti-patterns we explicitly avoid

The dashboard does not surface:

1. Raw test coverage percentage as a top-line metric. A 90%-covered abandoned project is worse than a 60%-covered actively-maintained one. Coverage trend is shown one click down; coverage absolute is not.
2. “Signed commits” as a binary green check. Without policy enforcement, signed-commit ratio is decorative. We show the ratio paired with the branch-protection-required-signature state, or omit.
3. SBOM presence without dependency edges. A component list with no dependencies[] graph is a manifest, not an SBOM. We visibly downgrade these.
4. A composite “compliance score out of 100”. Buyers see through it; it incentivizes gaming cheap signals (badges, file presence) at the cost of expensive ones (review discipline, attestation hygiene).
5. Maintainer-identity signals (country, employer, real name). Encourages discrimination, generates false positives on pseudonymous-but-trusted maintainers, adds zero supply-chain integrity.
6. GitHub stars. A popularity metric, not a trust metric.

Where the maintainer-identity line is drawn

The dashboard collects per-commit aggregates — DCO sign-off rate, verified-commit ratio, conventional-commits rate, unique-committer count — without collecting per-commit identities. The distinction is deliberate:

• Ratios over 90 days measure repository discipline. They answer “was the policy followed” without naming who followed or didn’t. These are surfaced on governance.html.
• Per-commit identity would attach a name (or pseudonym) to an approve/sign-off action. The dashboard collects only the count of unique committers, not the committer identities themselves.
• Cardinality is the operational signal we care about for bus factor: a unique_committers_90d == 1 is a procurement-relevant fact whether the one committer is anonymous or named.

In particular, verified_commit_ratio_90d is a ratio of GitHub’s verification.verified boolean over commits-in-window — not an identity claim. It is rendered always paired with the branch-protection state, because without required-signed-commits enforcement (a branch-protection API field, not an identity check) the ratio is decorative.

Pill semantics

Every pill on the dashboard is one of four states. The state is decided from the JSON snapshot using the rules below — there is no fall-through to a default colour. Gray means “no signal yet,” never “assumed green.” Each row anchors the per-pill link on the index and per-package pages.

A pill state is a property of the snapshot — it is not an opinion of the renderer. To dispute a pill state, dispute the underlying field in api/v1/snapshot.json.

Verifying a claim independently

Every dashboard card has a Verify link. Following it should let you reproduce the underlying lookup without any privileged access:

If a Verify link doesn’t reproduce, the dashboard claim is wrong and should be reported.

Repository-level Actions settings such as fork-approval policy, default workflow-token permissions, SHA-pinning enforcement, and secret scanning are intentionally absent from this table today. They are important controls, but their GitHub API evidence requires maintainer or admin authority. We audit them operationally and avoid rendering a public green check until the evidence can be reproduced by a third party.

Limits and honest gaps

Things this dashboard does not prove:

• Reproducible builds. We surface SLSA build-level claims but do not independently rebuild artifacts. A separate effort is required to attest reproducibility.
• Per-commit author identity beyond GitHub’s verification.verified state. GitHub’s GPG-key claim is what we report; deeper identity proofing (e.g., DCO sign-off matching a known steward) is not done.
• Source-of-truth for transitive licenses. We rely on the published license metadata of each dep. If a published license is wrong, our aggregation is wrong. We do not legally audit license claims.
• Operational security of the maintainer machines. A dashboard can prove the artifact came from a specific workflow; it cannot prove the workflow’s secrets weren’t exfiltrated upstream. That’s outside the scope of any public dashboard.
• OpenSSF Scorecard per-check results. The Scorecard workflows are installed, but the dashboard does not yet ingest the per-check SARIF/JSON output. Tracked as R8 in docs/research/08-followup.md. A buyer can run scorecard --repo=273v/kaos-compliance --format=json locally to fill the gap today.
• SLSA Build Level not formally attested. Each package’s PEP 740 attestation + PyPI Trusted Publisher state puts it effectively at SLSA Build L2. The supply-chain page surfaces an implied-level pill per package; we do not emit a formal slsa.build.level claim until the SBOM dep-graph (F9) and the Scorecard ingest (F18) land.
• CISA SBOM Minimum Elements: relationships gap. The seven required elements (author, supplier, name, version, unique ID, relationships, timestamp) are visible per package on supply-chain.html; relationships is flagged yellow across the org because the SBOM emitter doesn’t yet build the dependency edge graph.

These gaps are intentionally surfaced so a buyer knows what they still need to ask for in a vendor questionnaire.

Snapshot integrity + schema

The HTML pages above are one rendering of a JSON document published at api/v1/snapshot.json. Two artifacts accompany that JSON so a third party can verify both its shape and its provenance:

• JSON Schema (Draft 2020-12) at api/v1/snapshot.schema.json. Programmatic consumers should validate the snapshot against the schema before trusting any field; the schema is derived deterministically from the dataclasses in collector/snapshot.py so any drift is caught at render time.
• Keyless sigstore signature at api/v1/snapshot.sig — a DSSE bundle minted by the sweep.yml workflow via GitHub OIDC. Verification requires cosign and asserts the bundle was issued by THIS workflow on THIS repository before the JSON is trusted. The full verification recipe lives in docs/EVIDENCE.md.

The data model itself — field-by-field meaning and units — is documented in docs/DATA-MODEL.md.

Methodology versioning

This document follows semver. The bump rules are tight on purpose: the property that makes the dashboard credible — that an external reviewer can pin a snapshot to a methodology version and reproduce the assessment — fails the first time the green threshold for a signal is silently “clarified” without bumping the version.

A bump requires an entry in CHANGELOG.md with the version, the change, the affected signals, and the rationale. The published snapshot at api/v1/snapshot.json carries schema_version separately — that field bumps independently when the JSON shape changes; the methodology version below governs the meaning of the signals, not the wire format.

Trend lines + baselines

Each row of the package grid carries up to four status sparklines — one for build, tests, security, and PEP 740 attestation. Activity surfaces also use inline SVG for commits, releases, and code-surface counters. There is no JavaScript and no client-side state. What you’re seeing is the rolling 90-day history of the underlying signal, sampled once per UTC day:

• Resolution. Daily. Each x-axis position is one UTC day; multiple sweeps on the same day collapse to the last write.
• Encoding. Booleans (build pass / tests pass / security pass / attestation present) are projected to the top of the band when true and the bottom when false. The right-most marker is a solid dot so the eye lands on the current state. Days where the signal is missing render as a small open dot — visible gap, not silent.
• First-deploy behavior. The first time this dashboard ever ran, it had exactly one day of history. The sparkline collapses to a single dot and the cell carries an accumulating history label so the absence of a line is not mistaken for a flat signal. We never extrapolate, smooth, or backfill historical data.
• Baseline. The pill hover-title carries a delta arrow comparing the current state to the previous sweep (the most recent UTC day with data before today). ↑ for an improvement, ↓ for a regression, ≡ for no change. We do not implement industry or peer-package baselines; those rolls are a separate compliance question and have their own follow-up entry.
• Source. The whole rolling history is published at api/v1/history.json and the per-day raw files live under api/v1/history/YYYY-MM-DD.json. Activity counters include commits_90d, releases_90d, loc_total, src_loc, tests_loc, and files_total. The format is documented at api/v1/.

Per-package detail pages render a separate “Changes since last sweep” section above the evidence list. That section uses the same previous-sweep baseline and lists every signal that flipped, with a timestamp pair. Packages with no prior sweep show no prior sweep yet.