GHSA-WJMF-P669-5M5P

Vulnerability from github – Published: 2026-08-28 18:43 – Updated: 2026-08-28 18:43
VLAI
Summary
Protego has exponential backtracking ReDoS in robots.txt URL wildcard matching
Details

Problem description

Protego constructs regular expressions to match URLs against robots.txt Allow: and Disallow: directives, see protego._urlpattern._URLPattern._prepare_pattern_for_regex(). Every * in the directive value is translated into a lazy .*? regex piece, thus a specially crafted directive value with many asterisks may produce a regex that freezes the parser due to exponential backtracking.

Impact

Parsing a specially crafted robots.txt with protego.Protego.parse() and then trying to match an URL with protego.Protego.can_fetch() results in the latter call not returning for a period dependent on the length of the URL.

Proof of concept

from protego import Protego

robotstxt = f"""
User-agent: *
Disallow: /{"*1" * 12}*Z
"""
rp = Protego.parse(robotstxt)
url = "/" + "1" * 60
rp.can_fetch(url, "mybot")  # freezes
Severity
7.1 (High)
CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:P/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N
Show details on source website
To clipboard 

{
  "affected": [
    {
      "database_specific": {
        "last_known_affected_version_range": "\u003c= 0.6.1"
      },
      "package": {
        "ecosystem": "PyPI",
        "name": "Protego"
      },
      "ranges": [
        {
          "events": [
            {
              "introduced": "0"
            },
            {
              "fixed": "0.6.2"
            }
          ],
          "type": "ECOSYSTEM"
        }
      ]
    }
  ],
  "aliases": [
    "CVE-2026-55520"
  ],
  "database_specific": {
    "cwe_ids": [
      "CWE-1333",
      "CWE-400"
    ],
    "github_reviewed": true,
    "github_reviewed_at": "2026-08-28T18:43:02Z",
    "nvd_published_at": null,
    "severity": "HIGH"
  },
  "details": "### Problem description\n\nProtego constructs regular expressions to match URLs against `robots.txt` `Allow:` and `Disallow:` directives, see `protego._urlpattern._URLPattern._prepare_pattern_for_regex()`. Every `*` in the directive value is translated into a lazy `.*?` regex piece, thus a specially crafted directive value with many asterisks may produce a regex that freezes the parser due to exponential backtracking.\n\n### Impact\n\nParsing a specially crafted `robots.txt` with `protego.Protego.parse()` and then trying to match an URL with `protego.Protego.can_fetch()` results in the latter call not returning for a period dependent on the length of the URL.\n\n### Proof of concept\n\n```python\nfrom protego import Protego\n\nrobotstxt = f\"\"\"\nUser-agent: *\nDisallow: /{\"*1\" * 12}*Z\n\"\"\"\nrp = Protego.parse(robotstxt)\nurl = \"/\" + \"1\" * 60\nrp.can_fetch(url, \"mybot\")  # freezes\n```",
  "id": "GHSA-wjmf-p669-5m5p",
  "modified": "2026-08-28T18:43:02Z",
  "published": "2026-08-28T18:43:02Z",
  "references": [
    {
      "type": "WEB",
      "url": "https://github.com/scrapy/protego/security/advisories/GHSA-wjmf-p669-5m5p"
    },
    {
      "type": "WEB",
      "url": "https://github.com/scrapy/protego/commit/785940181659bf440ba82f1da148fade5087e858"
    },
    {
      "type": "PACKAGE",
      "url": "https://github.com/scrapy/protego"
    },
    {
      "type": "WEB",
      "url": "https://github.com/scrapy/protego/releases/tag/0.6.2"
    }
  ],
  "schema_version": "1.4.0",
  "severity": [
    {
      "score": "CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:P/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N",
      "type": "CVSS_V4"
    }
  ],
  "summary": "Protego has exponential backtracking ReDoS in robots.txt URL wildcard matching"
}



Log in or create an account to share your comment.




Tags
Taxonomy of the tags.





Forecast uses a logistic model when the trend is rising, or an exponential decay model when the trend is falling. Fitted via linearized least squares.

Sightings

Author Source Type Date Other

Nomenclature

  • Seen: The vulnerability was mentioned, discussed, or observed by the user.
  • Confirmed: The vulnerability has been validated from an analyst's perspective.
  • Published Proof of Concept: A public proof of concept is available for this vulnerability.
  • Exploited: The vulnerability was observed as exploited by the user who reported the sighting.
  • Patched: The vulnerability was observed as successfully patched by the user who reported the sighting.
  • Not exploited: The vulnerability was not observed as exploited by the user who reported the sighting.
  • Not confirmed: The user expressed doubt about the validity of the vulnerability.
  • Not patched: The vulnerability was not observed as successfully patched by the user who reported the sighting.



Related by attack behaviour

Vulnerabilities whose description is nearest to this one in the vector space of the CIRCL/vulnerability-attack-technique-biencoder model. This is a similarity search over the bi-encoder space (plain cosine), not a classification, and it has no measured accuracy.