Command Line Interface#
Run every command from the project directory.
Core#
Services#
$ poetry run start # start everything
$ poetry run stop # stop everything
$ poetry run run_backend --start # Valkey and Kvrocks only (--stop to stop)
$ poetry run feeders_manager --start # all enabled feeders (--stop to stop)
$ poetry run start_website # the website only
$ poetry run restart_website
feeders_manager --start --reimport makes every feeder reimport its whole source.
Run an individual feeder#
$ poetry run nvd_importer
$ poetry run cvelist_importer
$ poetry run github_importer
The full list is under [project.scripts] in pyproject.toml.
Dumps#
$ poetry run dump --feed nvd # one source, as JSON
$ poetry run dump --correlations --index # identifier correlations
$ poetry run euvd_dump # CVE ↔ EUVD mapping, as CSV
--indexadds the file to the download index.dump --allalso writes the correlations.A failing auxiliary export (comments, bundles, sightings, KEV entries, correlations) is skipped and logged; the command then exits non-zero and names it.
Files are replaced atomically: a failed run keeps the previous export.
Run
poetry run dump --indexaftereuvd_dumpif the index does not list it yet.
dumps/correlations.json maps each identifier to its directly linked identifiers:
{
"cve-2024-1234": ["ghsa-abcd-efgh-ijkl", "pysec-2024-1"],
"ghsa-abcd-efgh-ijkl": ["cve-2024-1234"],
"pysec-2024-1": ["cve-2024-1234"]
}
Identifiers are lowercased;
GCVE-0-<year>-<n>is merged intocve-<year>-<n>.Self-links and identifiers with no link are left out.
Links are direct, not transitive, and not a claim of equivalence.
It is not guaranteed to be symmetric: it is read from live storage. Add the reverse edges yourself if you need them.
Daily example:
0 7 * * * cd /path/to/vulnerability-lookup && poetry run dump --correlations --index
Bootstrap from Vulnerability-Lookup dumps#
With Kvrocks running, import the NDJSON files of a trusted instance into a fresh one:
$ poetry run import_dump /path/to/dumps
$ poetry run index_vulnerabilities
The file name (without
.ndjson) becomes the source name.Existing documents are kept;
--replaceoverwrites them. Re-running is safe and never triggers notifications.Comments, bundles, sightings and KEV entries are ignored. Unknown documents are counted, not fatal.
Dumps without the
vulnerability-lookup:idandvulnerability-lookup:scoreannotations still import, but their documents look modified at import time.
Dumps are a one-time bootstrap, not a sync. Do not schedule dump downloads: an
import never updates existing documents, and full downloads cost the publisher
bandwidth. Enable the feeders afterwards, and use the API’s since= parameter.
Receive EUVD numbers from ENISA#
EUVD numbers are issued only by ENISA’s instance. Any other instance can receive them
with the enisa_euvd feeder, which never issues a number itself. It is off by default.
If this instance ever ran the former EUVD feeder, remove the numbers it issued first. The feeder refuses to start while that feeder’s counters remain:
$ poetry run python tools/purge_euvd_allocations.py # dry run $ poetry run python tools/purge_euvd_allocations.py --yes
In
config/modules.cfg, setenabled = trueand the address of ENISA’s instance in the[feeder:enisa_euvd]section:[feeder:enisa_euvd] script_name = enisa_euvd_importer level = INFO enabled = true url = https://address-of-enisas-instance
In
config/website.py, uncomment theeuvdentry ofSOURCES_TO_SHOW, so EUVD is listed under National & Regional Vulnerability Databases.Leave GCVE pulling off for GNA 2 (EUVD) in the admin GCVE registry. It would add a second, empty EUVD entry under GCVE Numbering Authorities.
Start it with the other feeders (
poetry run feeders_manager --start), and after the first load runpoetry run index_vulnerabilities --source euvd, so each record’s status follows its CVE, andpoetry run euvd --backfill-kev.
How it works:
The first pass loads ENISA’s export (
euvd.ndjsonandcve-euvd-mapping.csvunder<url>/dumps/), streamed. Each line of the export says which CVE its record was issued for; the CSV is only used for an export written before the lines did. A load that stops resumes where it stopped while the export is the same file, and a line that cannot be read is logged and skipped, then served again by the changes feed.Every later pass, hourly, asks
<url>/api/euvd/changesfor the records changed after a cursor, in ENISA’s update order. The cursor is a position in that order, not a date or a page number, so a record updated while the feeder pages is served again later rather than skipped. After the first load it starts a day before the export’s newest record, and no later than any record it could not store: a line that could not be read, or a record of an older export that the CVE mapping does not list. A record the export says has no CVE does not move it.On a 429, a 5xx or a lost connection it waits
retry_afterseconds and retries, up tomax_retriestimes; a 429’sRetry-Afteris honoured up to five minutes. Any other 4xx is not retried: the next pass asks again.A record is stored with only its own fields (number, reservation, publication and update dates). Its description, scores and products come from this instance’s own CVE record. A record that arrives before its CVE is stored at once and completed once the CVE is here; a record with no CVE is not carried. A withdrawn record stays withdrawn; any other status follows ENISA’s.
It refuses to run where
euvd_authoritativeis true, or where[feeder:euvd]is enabled: an instance either issues EUVD numbers or receives them. The purge tool, in turn, refuses to run while this feeder is enabled.poetry run enisa_euvd_importer --reimportloads the whole export again on the next pass.
Update the documentation#
$ cd docs; poetry run make html
The API reference comes from docs/_static/files/swagger.json. Regenerate it first
with dump_openapi when the API changed.
Web service#
Flask commands are prefixed with poetry run flask --app website.app.
Database#
$ poetry run flask --app website.app db_init
$ poetry run flask --app website.app db_backup # also run by `poetry run update`
$ poetry run flask --app website.app db_diagram # models diagram
$ poetry run flask --app website.app seed-rbac [--reset-defaults] # also run by `poetry run update`
seed-rbac adds the missing RBAC roles, permissions and policies. --reset-defaults
restores the built-in ones, overwriting your changes.
User management#
$ poetry run flask --app website.app create_user --login <login> --email <email> --password <password>
$ poetry run flask --app website.app create_admin --login <login> --email <email> --password <password>
$ poetry run flask --app website.app user_list
$ poetry run flask --app website.app user_delete --login <login>
$ poetry run flask --app website.app user_get_api_key --login <login>
Data management#
$ poetry run flask --app website.app update_warninglists
$ poetry run flask --app website.app update_gcve_registry
$ poetry run flask --app website.app sync_organizations [--login <login>]
$ poetry run python tools/update_cnw_assigners.py
$ poetry run flask --app website.app import_osi_approved_licenses
$ poetry run flask --app website.app import_languages
update_warninglists: MISP warning lists.update_gcve_registry: local copy of the GCVE registry and the CNA partner catalogue. Can run in cron.sync_organizations: syncs the organizations with the GCVE registry.--loginis the owner,adminby default.update_cnw_assigners.py: the EU CSIRT coordinated CNAs shown on the dashboard, from enisaeu/CNW (rows whoseCNAcolumn isYes). Can run in cron, or from the admin maintenance page. A failure keeps the previous list.
Sightings#
$ poetry run flask --app website.app clean_sightings --source <source>
$ poetry run flask --app website.app clean_sightings --source '<pattern>' --regex # PostgreSQL only
$ poetry run flask --app website.app backfill_sightings
$ poetry run flask --app website.app dedup_sightings [--yes]
$ poetry run flask --app website.app retype_sightings [--yes]
clean_sightings: deletes the sightings of a source, or of a matching pattern.backfill_sightings: creates sightings from existing comments (seen), bundles (seen, one per vulnerability) and KEV entries (exploited), with their original timestamps. Safe to re-run.dedup_sightings: removes repeated sightings of the same observation (same vulnerability, URL orTelegram/…source, type, author and origin), keeping the oldest. Label sources likeCISA-KEVare left alone.retype_sightings: by default retypes the Exploit-DB sightings pushed as exploited (source starting withhttps://www.exploit-db.com/exploits/) to published-proof-of-concept, deleting the duplicates.--source-prefix,--fromand--toselect other sightings. Safe to re-run.
Without --yes, dedup_sightings and retype_sightings are a dry run. Both only touch
sightings created on this instance: --all-origins includes synced ones (the next sync
may bring them back), --author <login> narrows to one user. Sync never updates a
sighting it already holds, so each instance runs them itself.
Move sightings between instances:
$ poetry run flask --app website.app export_sightings sightings.ndjson # source instance
$ poetry run flask --app website.app import_sightings sightings.ndjson --dry-run # target: preview
$ poetry run flask --app website.app import_sightings sightings.ndjson --author admin
Both stream, so they handle volumes the browser export times out on.
-is stdout/stdin (export_sightings - | gzip).The format is the one of
dumps/sightings.ndjson; the import also reads the browser export.--local-onlyexports only the sightings created on this instance.The import keeps uuid, origin and timestamp, and skips sightings already present. Unknown authors are replaced by
--author(adminby default). Invalid records are counted, not fatal.
Regenerate the OpenAPI specification#
$ poetry run flask --app website.app dump_openapi # write docs/_static/files/swagger.json
$ poetry run flask --app website.app dump_openapi --check # exit non-zero if it is out of date
No running instance is needed. The output is stable, and the instance UUID is replaced by an example (
--example-originsets another).It refuses to write unless
user_accounts,cnaandcpe_enabledare all on, and names the missing ones.--allow-partialwrites anyway.The
OpenAPIworkflow runs--checkon every pull request, with every module on.A pre-commit hook regenerates it when you touch
website/web/api/or the version inpyproject.toml. It skips, with a reason, on an instance that cannot produce the complete reference. When it regenerates, the commit stops so you can stage the file.A relative
--outputis taken fromVULNERABILITYLOOKUP_HOME. The command prints the path used.
Health and integrity checks#
$ poetry run flask --app website.app index_health
$ poetry run euvd --integrity-scan
index_health: reports whether the state indexes still match the corpus. A stale index shows no error anywhere else.euvd --integrity-scan: counts EUVD identifier breaches (several CVEs or none, pointer disagreements, published CVEs without EUVD) for/metrics. Exits non-zero on a breach. Run it daily. See Metrics.
Prepared statistics figures#
The per-source /api/stats/* figures are prepared in advance, not computed per
request. A figure never prepared answers available: false.
/api/stats/vulnerability/status is always live.
Run these four commands once after installing or upgrading, then in cron:
# m h dom mon dow command
30 3 * * * cd /path/to/vulnerability-lookup && poetry run index_vulnerabilities
30 4 * * * cd /path/to/vulnerability-lookup && poetry run euvd --stats-scan --stats-source cvelistv5
15 * * * * cd /path/to/vulnerability-lookup && poetry run scorecard --window-only
45 * * * * cd /path/to/vulnerability-lookup && poetry run kev_scorecard
All four are idempotent.
Source names#
A source (--source, --stats-source, ?source=) is the name records are stored
under, not the feeder’s section in config/modules.cfg. For example, [feeder:cvelist]
stores cvelistv5. List them with:
$ redis-cli -p 10002 HKEYS last_updates
GNA sources are the
gna-<id>entries, one per GNA whose records this instance pulls. A GNA with neither API nor dump (such as GNA 2, EUVD) never becomes a source.The
euvdsource holds EUVD records, which is unrelated to GNA 2. Official EUVD numbers are issued only by ENISA’s instance, by a component that is not part of Vulnerability-Lookup. Any other instance holdseuvdrecords only by receiving them from ENISA’s (see Receive EUVD numbers from ENISA).
The reindex: counts, rankings and CVSS distributions#
$ poetry run index_vulnerabilities # every source, GNAs included
$ poetry run index_vulnerabilities --source cvelistv5 --source gna-1
Derives the counts per year and status, the top vendors and assigners and the CVSS distributions, and runs the full assigner scorecard.
Without
--source, it covers every source inlast_updates. Name sources to make the nightly run shorter.Run it at least once on a new, restored or upgraded instance before trusting any figure: a stale index looks normal but is wrong.
The scorecards#
$ poetry run scorecard # assigner scorecard, whole history (also run by every reindex)
$ poetry run scorecard --window-only # only the rolling window: cheap, hourly
$ poetry run kev_scorecard # KEV catalogue scorecard, seconds
See assigner scorecard and catalogue scorecard.
Without the hourly scorecard --window-only, the pages show what the last reindex
prepared. The wall of shame has its own switch, WEB_MODULES["wall_of_shame"] (off by
default).
Upgrading from the EUVD-only figures#
The figures used to be keyed euvd:stats:*; they are now stats:<source>:*. Run once
after upgrading:
$ poetry run euvd --migrate-stats-keys
It is idempotent and does nothing on an instance that never held EUVD figures. Without
it, the time-to-exploit snapshot stays available: false.
Background services#
$ poetry run flask --app website.app notify_users # notifications
$ poetry run flask --app website.app sync # synchronization
See Notifications and Synchronization.