# Our policy
#
# Allowed:
# - Search engine indexers
# - Archival services (e.g. IA)
#
# Disallowed:
# - Marketing or SEO crawlers
# - Bots which are too aggressive by default. This is subjective, if you annoy
#   our sysadmins you'll be blocked.
# - AI bots (ChatGPT, Bard, etc.)
#
# Automated consumers: see https://vulnerability.circl.lu/.well-known/api-policy.json for the machine-readable
# access policy, including the canonical sync path (API + stream),
# expectations for a meaningful User-Agent, and rate-limit posture.
#
# Reach out to info@circl.lu if you have questions.

Policy: https://vulnerability.circl.lu/.well-known/api-policy.json

# It doesn't make sense to index these and/or it's expensive:
User-agent: *
Disallow: /*.ndjson$
Disallow: /stats

User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: rogerbot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: AspiegelBot
Disallow: /

User-agent: ZoominfoBot
Disallow: /

User-agent: Yandex
Disallow: /