Assigner scorecard

Assigner scorecard#

Vulnerability-Lookup ranks the numbering authorities whose records it holds by how well they fill them. Every CVE Numbering Authority (CNA) of the CVE Program and every GCVE Numbering Authority (GNA) gets a score, a grade and a badge, computed from its own publications against the rules on this page.

The scorecard is at /assigners/scorecard (CNAs) and /assigners/scorecard?kind=gna (GNAs); each assigner has a page at /assigners/<name> and a badge at /assigners/<name>/badge.svg. The same figures are served by /api/stats/scorecard/*.

What is scored#

A record is scored on its CNA container only: the part of a CVE JSON 5 record the assigner wrote. A CVSS or a CWE added afterwards by an Authorized Data Publisher (ADP) such as CISA’s vulnrichment is not the assigner’s work. It is counted separately, as enrichment dependency, and shown next to the score: a CNA whose severity is always supplied by someone else is a different fact from one that states none, and the two must stay tellable apart.

Both CNAs and GNAs publish CVE JSON 5 records, so one engine scores both. A CNA is identified by the record’s assignerShortName; a GNA is a source of its own (gna-<id>) whose every record is its own work, whatever the record says.

The rules#

Each rule returns a credit between 0 and 1. A record’s score is the weighted sum, out of 100; an assigner’s score is the mean over its records in the period. Four pillars, 25 points each:

Describe – can a reader tell what the vulnerability is?

Rule

Points

Credit

Description

10

An English description that is not a placeholder: 1.0 from 200 characters, 0.7 from 80, 0.4 from 40, 0 below.

Title

5

A title.

CWE

10

A weakness as a CWE identifier in problemTypes; a free-text problem type without one earns 0.25.

Scope – can a reader tell whether they are affected?

Rule

Points

Credit

Affected

10

An affected entry naming both a vendor and a product; one side missing or n/a earns 0.5.

Versions

8

An affected entry stating versions, as a version or a bound; a defaultStatus alone earns 0.5.

CPE

7

A CPE on an affected entry, or a cpeApplicability block.

Assess – can a reader tell how bad it is?

Rule

Points

Credit

Severity

15

A severity statement in the CNA container: a CVSS v3 or v4 vector, an SSVC decision, or a vendor rating as an other metric. A CVSS v2 vector alone earns 0.5.

Modern severity

5

A CVSS v4 vector or an SSVC decision.

Impacts

5

An impacts entry: a CAPEC identifier or an impact description.

Act – can a reader tell what to do about it?

Rule

Points

Credit

References

7

At least one reference.

Reference tags

5

At least one reference carrying tags.

Remediation

8

A reference tagged patch; one tagged vendor-advisory, mitigation or release-notes earns 0.6.

Solutions

5

A solution or a workaround.

Grades from the mean score: A from 90, B from 75, C from 60, D from 45, E below.

Three choices behind these rules are deliberate:

  • Severity is any severity statement, not CVSS. Several active CNAs (curl, OpenSSL, Go, Apple, Mozilla, Chrome…) publish no CVSS as a matter of policy. A scorecard reading that as negligence would rank the strength of an opinion about CVSS rather than the care put into a record. A record that states no severity of any kind still earns nothing here.

  • Rules are graded, not binary. A description of forty characters and one of four hundred are not the same record, and a check that only asks whether the field is non-empty awards full marks for one-line descriptions.

  • Timeliness is displayed, never scored. The delay between a reservation and its publication is a coordinated-disclosure embargo as often as it is a backlog, and no rule can tell the two apart. The median and 90th percentile are shown on each assigner’s page as context.

Ranking, window and threshold#

The ranking is built on a rolling window of the last 180 days of publications, so it reflects current practice rather than a decade’s. An assigner needs 10 publications in the window to be ranked; below that its score is shown and marked unranked, because three perfect records are not evidence of a practice and a leaderboard of them would be won by whoever publishes least. Both figures are options of the preparing pass and are reported with every answer.

Each assigner’s page also shows its all-time tallies and its score per publication year, which is where a CNA’s progress (or the opposite) is visible.

Context figures#

Shown beside the score and never in it:

  • Severity / weakness supplied by an ADP: the share of records whose CVSS or CWE was added by an ADP because the CNA container carries none.

  • No severity statement: the share of records with no severity of any kind.

  • Descriptions under 100 characters.

  • Known exploited: records listed by a known-exploited catalogue, and how many of them were published without a severity statement.

  • Reservation to publication: median and 90th percentile, in days.

  • Rejected records in the period.

  • For GNAs: how many records state the CVE they correspond to (or a relationship to one), and how many still use the legacy identifier placement of GCVE-BCP-05.

Hall of fame, wall of shame, badges#

The hall of fame lists the ranked assigners graded A.

The wall of shame does not list the bottom of the ranking, which would name nothing to change. It lists named practices with an objective threshold, over the window, among the ranked assigners:

  • no severity statement on any record;

  • no CWE on any record;

  • severity supplied by an ADP for more than half the records;

  • descriptions under 100 characters on more than 30% of the records;

  • affected products missing or n/a on more than half the records;

  • known-exploited vulnerabilities published without a severity statement.

It is off by default: naming publishers is a stance an instance takes deliberately. Enable it with WEB_MODULES["wall_of_shame"] = True in config/website.py. The ranking, the grades, the badges and the per-assigner pages are shown either way, and the signals stay readable from /api/stats/scorecard/signals.

Each assigner has an embeddable badge at /assigners/<name>/badge.svg showing its grade and score over the window. It reads “unranked” under the threshold rather than failing, so a README embedding it does not break the day its publisher goes quiet. The assigner’s page gives the Markdown to paste.

The API#

Endpoint

Answers

GET /api/stats/scorecard/ranking?source=cvelistv5[&period=window|all|YYYY][&limit=N][&output=markdown]

Every assigner of the source, ranked ones first, best first, with the preparation metadata.

GET /api/stats/scorecard/gnas[&period=...]

The GNAs ranked against each other, one entry per GNA source with a prepared scorecard.

GET /api/stats/scorecard/assigner?source=cvelistv5&assigner=<name>

One assigner: credit rate and points per rule, pillars, context figures, the per-year trend.

GET /api/stats/scorecard/signals?source=cvelistv5

The wall of shame signals and the ranked assigners tripping each.

For a GNA, source is its source name and assigner the same name, e.g. ?source=gna-1988&assigner=gna-1988. Like every prepared figure, an unprepared scorecard answers available: false rather than an empty leaderboard, and the methodology carries a version: figures prepared under older rules are reported as unavailable rather than read against rules they were not computed with.

Preparing the figures#

Nothing on the read side walks a record. The figures are prepared by

# the CVE list and every GNA source with records, whole history
$ poetry run scorecard

# only the rolling window the ranking is built on: a few tens of thousands of
# records rather than the corpus, the cadence to run hourly
$ poetry run scorecard --window-only

# one source, and the two knobs every answer reports
$ poetry run scorecard --source cvelistv5 --window-days 180 --min-records 10

index_vulnerabilities runs the full pass as part of every reindex, so an instance that reindexes nightly needs only the hourly refresh. Both replace the previous figures in one transaction, so a page reading them mid-run sees the previous run’s figures rather than none. The keys are stats:<source>:scorecard:*, documented in vulnerabilitylookup/scorecard.py.

Inspiration#

The idea of scoring CNAs on record completeness comes from CNA ScoreCard by RogoLabs, whose five binary checks over a six-month window showed the approach works. This implementation differs in what it scores (the CNA container only, with the ADP dependency shown apart), in grading rather than binary checks, in accepting any severity statement, in a volume threshold, in covering GNAs, and in being prepared inside the platform that already holds the records.