pysec - pysec-2021-859

PYSEC-2021-859

Vulnerability from pysec - Published: 2021-12-23 18:15 - Updated: 2022-01-04 17:38

Details

NLTK (Natural Language Toolkit) is a suite of open source Python modules, data sets, and tutorials supporting research and development in Natural Language Processing. Versions prior to 3.6.5 are vulnerable to regular expression denial of service (ReDoS) attacks. The vulnerability is present in PunktSentenceTokenizer, sent_tokenize and word_tokenize. Any users of this class, or these two functions, are vulnerable to the ReDoS attack. In short, a specifically crafted long input to any of these vulnerable functions will cause them to take a significant amount of execution time. If your program relies on any of the vulnerable functions for tokenizing unpredictable user input, then we would strongly recommend upgrading to a version of NLTK without the vulnerability. For users unable to upgrade the execution time can be bounded by limiting the maximum length of an input to any of the vulnerable functions. Our recommendation is to implement such a limit.

Impacted products

Name	purl
nltk	pkg:pypi/nltk

Aliases

JSON

To clipboard

{
  "affected": [
    {
      "package": {
        "ecosystem": "PyPI",
        "name": "nltk",
        "purl": "pkg:pypi/nltk"
      },
      "ranges": [
        {
          "events": [
            {
              "introduced": "0"
            },
            {
              "fixed": "1405aad979c6b8080dbbc8e0858f89b2e3690341"
            }
          ],
          "repo": "https://github.com/nltk/nltk",
          "type": "GIT"
        },
        {
          "events": [
            {
              "introduced": "0"
            },
            {
              "fixed": "3.6.5"
            }
          ],
          "type": "ECOSYSTEM"
        }
      ],
      "versions": [
        "0.8",
        "0.9",
        "0.9.3",
        "0.9.4",
        "0.9.5",
        "0.9.6",
        "0.9.7",
        "0.9.8",
        "0.9.9",
        "2.0.1",
        "2.0.1rc1",
        "2.0.1rc2-git",
        "2.0.1rc3",
        "2.0.1rc4",
        "2.0.2",
        "2.0.3",
        "2.0.4",
        "2.0.5",
        "2.0b4",
        "2.0b5",
        "2.0b6",
        "2.0b7",
        "2.0b8",
        "2.0b9",
        "3.0.0",
        "3.0.0b1",
        "3.0.0b2",
        "3.0.1",
        "3.0.2",
        "3.0.3",
        "3.0.4",
        "3.0.5",
        "3.1",
        "3.2",
        "3.2.1",
        "3.2.2",
        "3.2.3",
        "3.2.4",
        "3.2.5",
        "3.3",
        "3.4",
        "3.4.1",
        "3.4.2",
        "3.4.3",
        "3.4.4",
        "3.4.5",
        "3.5",
        "3.5b1",
        "3.6",
        "3.6.1",
        "3.6.2",
        "3.6.3",
        "3.6.4"
      ]
    }
  ],
  "aliases": [
    "CVE-2021-43854",
    "GHSA-f8m6-h2c7-8h9x"
  ],
  "details": "NLTK (Natural Language Toolkit) is a suite of open source Python modules, data sets, and tutorials supporting research and development in Natural Language Processing. Versions prior to 3.6.5 are vulnerable to regular expression denial of service (ReDoS) attacks. The vulnerability is present in PunktSentenceTokenizer, sent_tokenize and word_tokenize. Any users of this class, or these two functions, are vulnerable to the ReDoS attack. In short, a specifically crafted long input to any of these vulnerable functions will cause them to take a significant amount of execution time. If your program relies on any of the vulnerable functions for tokenizing unpredictable user input, then we would strongly recommend upgrading to a version of NLTK without the vulnerability. For users unable to upgrade the execution time can be bounded by limiting the maximum length of an input to any of the vulnerable functions. Our recommendation is to implement such a limit.",
  "id": "PYSEC-2021-859",
  "modified": "2022-01-04T17:38:55.854845Z",
  "published": "2021-12-23T18:15:00Z",
  "references": [
    {
      "type": "WEB",
      "url": "https://github.com/nltk/nltk/pull/2869"
    },
    {
      "type": "FIX",
      "url": "https://github.com/nltk/nltk/commit/1405aad979c6b8080dbbc8e0858f89b2e3690341"
    },
    {
      "type": "ADVISORY",
      "url": "https://github.com/nltk/nltk/security/advisories/GHSA-f8m6-h2c7-8h9x"
    },
    {
      "type": "REPORT",
      "url": "https://github.com/nltk/nltk/issues/2866"
    }
  ]
}

GHSA-F8M6-H2C7-8H9X

Vulnerability from github – Published: 2022-01-06 17:38 – Updated: 2024-09-26 14:17

Summary

Inefficient Regular Expression Complexity in nltk (word_tokenize, sent_tokenize)

Details

Impact

The vulnerability is present in PunktSentenceTokenizer, sent_tokenize and word_tokenize. Any users of this class, or these two functions, are vulnerable to a Regular Expression Denial of Service (ReDoS) attack. In short, a specifically crafted long input to any of these vulnerable functions will cause them to take a significant amount of execution time. The effect of this vulnerability is noticeable with the following example:

from nltk.tokenize import word_tokenize

n = 8
for length in [10**i for i in range(2, n)]:
    # Prepare a malicious input
    text = "a" * length
    start_t = time.time()
    # Call `word_tokenize` and naively measure the execution time
    word_tokenize(text)
    print(f"A length of {length:<{n}} takes {time.time() - start_t:.4f}s")

Which gave the following output during testing:

A length of 100      takes 0.0060s
A length of 1000     takes 0.0060s
A length of 10000    takes 0.6320s
A length of 100000   takes 56.3322s
...

I canceled the execution of the program after running it for several hours.

If your program relies on any of the vulnerable functions for tokenizing unpredictable user input, then we would strongly recommend upgrading to a version of NLTK without the vulnerability, or applying the workaround described below.

Patches

The problem has been patched in NLTK 3.6.6. After the fix, running the above program gives the following result:

A length of 100      takes 0.0070s
A length of 1000     takes 0.0010s
A length of 10000    takes 0.0060s
A length of 100000   takes 0.0400s
A length of 1000000  takes 0.3520s
A length of 10000000 takes 3.4641s

This output shows a linear relationship in execution time versus input length, which is desirable for regular expressions. We recommend updating to NLTK 3.6.6+ if possible.

Workarounds

The execution time of the vulnerable functions is exponential to the length of a malicious input. With other words, the execution time can be bounded by limiting the maximum length of an input to any of the vulnerable functions. Our recommendation is to implement such a limit.

References

The issue showcasing the vulnerability: https://github.com/nltk/nltk/issues/2866
The pull request containing considerably more information on the vulnerability, and the fix: https://github.com/nltk/nltk/pull/2869
The commit containing the fix: 1405aad979c6b8080dbbc8e0858f89b2e3690341
Information on CWE-1333: Inefficient Regular Expression Complexity: https://cwe.mitre.org/data/definitions/1333.html

For more information

If you have any questions or comments about this advisory: * Open an issue in github.com/nltk/nltk * Email us at nltk.team@gmail.com

Severity ?

7.5 (High)


                  
                    CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

8.7 (High)


                  
                    CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N

Show details on source website

JSON

To clipboard

{
  "affected": [
    {
      "package": {
        "ecosystem": "PyPI",
        "name": "nltk"
      },
      "ranges": [
        {
          "events": [
            {
              "introduced": "0"
            },
            {
              "fixed": "3.6.6"
            }
          ],
          "type": "ECOSYSTEM"
        }
      ]
    }
  ],
  "aliases": [
    "CVE-2021-43854"
  ],
  "database_specific": {
    "cwe_ids": [
      "CWE-400"
    ],
    "github_reviewed": true,
    "github_reviewed_at": "2022-01-05T17:40:20Z",
    "nvd_published_at": "2021-12-23T18:15:00Z",
    "severity": "HIGH"
  },
  "details": "### Impact\nThe vulnerability is present in [`PunktSentenceTokenizer`](https://www.nltk.org/api/nltk.tokenize.punkt.html#nltk.tokenize.punkt.PunktSentenceTokenizer), [`sent_tokenize`](https://www.nltk.org/api/nltk.tokenize.html#nltk.tokenize.sent_tokenize)  and [`word_tokenize`](https://www.nltk.org/api/nltk.tokenize.html#nltk.tokenize.word_tokenize). Any users of this class, or these two functions, are vulnerable to a Regular Expression Denial of Service (ReDoS) attack. \nIn short, a specifically crafted long input to any of these vulnerable functions will cause them to take a significant amount of execution time. The effect of this vulnerability is noticeable with the following example:\n```python\nfrom nltk.tokenize import word_tokenize\n\nn = 8\nfor length in [10**i for i in range(2, n)]:\n    # Prepare a malicious input\n    text = \"a\" * length\n    start_t = time.time()\n    # Call `word_tokenize` and naively measure the execution time\n    word_tokenize(text)\n    print(f\"A length of {length:\u003c{n}} takes {time.time() - start_t:.4f}s\")\n```\nWhich gave the following output during testing:\n```python\nA length of 100      takes 0.0060s\nA length of 1000     takes 0.0060s\nA length of 10000    takes 0.6320s\nA length of 100000   takes 56.3322s\n...\n```\nI canceled the execution of the program after running it for several hours.\n\nIf your program relies on any of the vulnerable functions for tokenizing unpredictable user input, then we would strongly recommend upgrading to a version of NLTK without the vulnerability, or applying the workaround described below.\n\n### Patches\nThe problem has been patched in NLTK 3.6.6. After the fix, running the above program gives the following result:\n```python\nA length of 100      takes 0.0070s\nA length of 1000     takes 0.0010s\nA length of 10000    takes 0.0060s\nA length of 100000   takes 0.0400s\nA length of 1000000  takes 0.3520s\nA length of 10000000 takes 3.4641s\n```\nThis output shows a linear relationship in execution time versus input length, which is desirable for regular expressions.\nWe recommend updating to NLTK 3.6.6+ if possible.\n\n### Workarounds\nThe execution time of the vulnerable functions is exponential to the length of a malicious input. With other words, the execution time can be bounded by limiting the maximum length of an input to any of the vulnerable functions. Our recommendation is to implement such a limit.\n\n### References\n* The issue showcasing the vulnerability: https://github.com/nltk/nltk/issues/2866\n* The pull request containing considerably more information on the vulnerability, and the fix: https://github.com/nltk/nltk/pull/2869\n* The commit containing the fix: 1405aad979c6b8080dbbc8e0858f89b2e3690341\n* Information on CWE-1333: Inefficient Regular Expression Complexity: https://cwe.mitre.org/data/definitions/1333.html\n\n### For more information\nIf you have any questions or comments about this advisory:\n* Open an issue in [github.com/nltk/nltk](https://github.com/nltk/nltk)\n* Email us at [nltk.team@gmail.com](mailto:nltk.team@gmail.com)\n",
  "id": "GHSA-f8m6-h2c7-8h9x",
  "modified": "2024-09-26T14:17:15Z",
  "published": "2022-01-06T17:38:45Z",
  "references": [
    {
      "type": "WEB",
      "url": "https://github.com/nltk/nltk/security/advisories/GHSA-f8m6-h2c7-8h9x"
    },
    {
      "type": "ADVISORY",
      "url": "https://nvd.nist.gov/vuln/detail/CVE-2021-43854"
    },
    {
      "type": "WEB",
      "url": "https://github.com/nltk/nltk/issues/2866"
    },
    {
      "type": "WEB",
      "url": "https://github.com/nltk/nltk/pull/2869"
    },
    {
      "type": "WEB",
      "url": "https://github.com/nltk/nltk/commit/1405aad979c6b8080dbbc8e0858f89b2e3690341"
    },
    {
      "type": "PACKAGE",
      "url": "https://github.com/nltk/nltk"
    },
    {
      "type": "WEB",
      "url": "https://github.com/pypa/advisory-database/tree/main/vulns/nltk/PYSEC-2021-859.yaml"
    }
  ],
  "schema_version": "1.4.0",
  "severity": [
    {
      "score": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H",
      "type": "CVSS_V3"
    },
    {
      "score": "CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N",
      "type": "CVSS_V4"
    }
  ],
  "summary": "Inefficient Regular Expression Complexity in nltk (word_tokenize, sent_tokenize)"
}

CVE-2021-43854 (GCVE-0-2021-43854)

Vulnerability from cvelistv5 – Published: 2021-12-23 17:55 – Updated: 2024-08-04 04:10

Summary

Severity ?

7.5 (High)


                        
                          CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

CWE

CWE-400 - Uncontrolled Resource Consumption

Assigner

GitHub_M

References

URL

Tags

	https://github.com/nltk/nltk/security/advisories/…	x_refsource_CONFIRM
	https://github.com/nltk/nltk/issues/2866	x_refsource_MISC
	https://github.com/nltk/nltk/pull/2869	x_refsource_MISC
	https://github.com/nltk/nltk/commit/1405aad979c6b…	x_refsource_MISC

Impacted products

	Vendor	Product	Version
	nltk	nltk	Affected: < 3.6.5

Show details on NVD website

JSON

To clipboard

{
  "containers": {
    "adp": [
      {
        "providerMetadata": {
          "dateUpdated": "2024-08-04T04:10:16.281Z",
          "orgId": "af854a3a-2127-422b-91ae-364da2661108",
          "shortName": "CVE"
        },
        "references": [
          {
            "tags": [
              "x_refsource_CONFIRM",
              "x_transferred"
            ],
            "url": "https://github.com/nltk/nltk/security/advisories/GHSA-f8m6-h2c7-8h9x"
          },
          {
            "tags": [
              "x_refsource_MISC",
              "x_transferred"
            ],
            "url": "https://github.com/nltk/nltk/issues/2866"
          },
          {
            "tags": [
              "x_refsource_MISC",
              "x_transferred"
            ],
            "url": "https://github.com/nltk/nltk/pull/2869"
          },
          {
            "tags": [
              "x_refsource_MISC",
              "x_transferred"
            ],
            "url": "https://github.com/nltk/nltk/commit/1405aad979c6b8080dbbc8e0858f89b2e3690341"
          }
        ],
        "title": "CVE Program Container"
      }
    ],
    "cna": {
      "affected": [
        {
          "product": "nltk",
          "vendor": "nltk",
          "versions": [
            {
              "status": "affected",
              "version": "\u003c 3.6.5"
            }
          ]
        }
      ],
      "descriptions": [
        {
          "lang": "en",
          "value": "NLTK (Natural Language Toolkit) is a suite of open source Python modules, data sets, and tutorials supporting research and development in Natural Language Processing. Versions prior to 3.6.5 are vulnerable to regular expression denial of service (ReDoS) attacks. The vulnerability is present in PunktSentenceTokenizer, sent_tokenize and word_tokenize. Any users of this class, or these two functions, are vulnerable to the ReDoS attack. In short, a specifically crafted long input to any of these vulnerable functions will cause them to take a significant amount of execution time. If your program relies on any of the vulnerable functions for tokenizing unpredictable user input, then we would strongly recommend upgrading to a version of NLTK without the vulnerability. For users unable to upgrade the execution time can be bounded by limiting the maximum length of an input to any of the vulnerable functions. Our recommendation is to implement such a limit."
        }
      ],
      "metrics": [
        {
          "cvssV3_1": {
            "attackComplexity": "LOW",
            "attackVector": "NETWORK",
            "availabilityImpact": "HIGH",
            "baseScore": 7.5,
            "baseSeverity": "HIGH",
            "confidentialityImpact": "NONE",
            "integrityImpact": "NONE",
            "privilegesRequired": "NONE",
            "scope": "UNCHANGED",
            "userInteraction": "NONE",
            "vectorString": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H",
            "version": "3.1"
          }
        }
      ],
      "problemTypes": [
        {
          "descriptions": [
            {
              "cweId": "CWE-400",
              "description": "CWE-400: Uncontrolled Resource Consumption",
              "lang": "en",
              "type": "CWE"
            }
          ]
        }
      ],
      "providerMetadata": {
        "dateUpdated": "2021-12-23T17:55:11",
        "orgId": "a0819718-46f1-4df5-94e2-005712e83aaa",
        "shortName": "GitHub_M"
      },
      "references": [
        {
          "tags": [
            "x_refsource_CONFIRM"
          ],
          "url": "https://github.com/nltk/nltk/security/advisories/GHSA-f8m6-h2c7-8h9x"
        },
        {
          "tags": [
            "x_refsource_MISC"
          ],
          "url": "https://github.com/nltk/nltk/issues/2866"
        },
        {
          "tags": [
            "x_refsource_MISC"
          ],
          "url": "https://github.com/nltk/nltk/pull/2869"
        },
        {
          "tags": [
            "x_refsource_MISC"
          ],
          "url": "https://github.com/nltk/nltk/commit/1405aad979c6b8080dbbc8e0858f89b2e3690341"
        }
      ],
      "source": {
        "advisory": "GHSA-f8m6-h2c7-8h9x",
        "discovery": "UNKNOWN"
      },
      "title": "Inefficient Regular Expression Complexity in nltk",
      "x_legacyV4Record": {
        "CVE_data_meta": {
          "ASSIGNER": "security-advisories@github.com",
          "ID": "CVE-2021-43854",
          "STATE": "PUBLIC",
          "TITLE": "Inefficient Regular Expression Complexity in nltk"
        },
        "affects": {
          "vendor": {
            "vendor_data": [
              {
                "product": {
                  "product_data": [
                    {
                      "product_name": "nltk",
                      "version": {
                        "version_data": [
                          {
                            "version_value": "\u003c 3.6.5"
                          }
                        ]
                      }
                    }
                  ]
                },
                "vendor_name": "nltk"
              }
            ]
          }
        },
        "data_format": "MITRE",
        "data_type": "CVE",
        "data_version": "4.0",
        "description": {
          "description_data": [
            {
              "lang": "eng",
              "value": "NLTK (Natural Language Toolkit) is a suite of open source Python modules, data sets, and tutorials supporting research and development in Natural Language Processing. Versions prior to 3.6.5 are vulnerable to regular expression denial of service (ReDoS) attacks. The vulnerability is present in PunktSentenceTokenizer, sent_tokenize and word_tokenize. Any users of this class, or these two functions, are vulnerable to the ReDoS attack. In short, a specifically crafted long input to any of these vulnerable functions will cause them to take a significant amount of execution time. If your program relies on any of the vulnerable functions for tokenizing unpredictable user input, then we would strongly recommend upgrading to a version of NLTK without the vulnerability. For users unable to upgrade the execution time can be bounded by limiting the maximum length of an input to any of the vulnerable functions. Our recommendation is to implement such a limit."
            }
          ]
        },
        "impact": {
          "cvss": {
            "attackComplexity": "LOW",
            "attackVector": "NETWORK",
            "availabilityImpact": "HIGH",
            "baseScore": 7.5,
            "baseSeverity": "HIGH",
            "confidentialityImpact": "NONE",
            "integrityImpact": "NONE",
            "privilegesRequired": "NONE",
            "scope": "UNCHANGED",
            "userInteraction": "NONE",
            "vectorString": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H",
            "version": "3.1"
          }
        },
        "problemtype": {
          "problemtype_data": [
            {
              "description": [
                {
                  "lang": "eng",
                  "value": "CWE-400: Uncontrolled Resource Consumption"
                }
              ]
            }
          ]
        },
        "references": {
          "reference_data": [
            {
              "name": "https://github.com/nltk/nltk/security/advisories/GHSA-f8m6-h2c7-8h9x",
              "refsource": "CONFIRM",
              "url": "https://github.com/nltk/nltk/security/advisories/GHSA-f8m6-h2c7-8h9x"
            },
            {
              "name": "https://github.com/nltk/nltk/issues/2866",
              "refsource": "MISC",
              "url": "https://github.com/nltk/nltk/issues/2866"
            },
            {
              "name": "https://github.com/nltk/nltk/pull/2869",
              "refsource": "MISC",
              "url": "https://github.com/nltk/nltk/pull/2869"
            },
            {
              "name": "https://github.com/nltk/nltk/commit/1405aad979c6b8080dbbc8e0858f89b2e3690341",
              "refsource": "MISC",
              "url": "https://github.com/nltk/nltk/commit/1405aad979c6b8080dbbc8e0858f89b2e3690341"
            }
          ]
        },
        "source": {
          "advisory": "GHSA-f8m6-h2c7-8h9x",
          "discovery": "UNKNOWN"
        }
      }
    }
  },
  "cveMetadata": {
    "assignerOrgId": "a0819718-46f1-4df5-94e2-005712e83aaa",
    "assignerShortName": "GitHub_M",
    "cveId": "CVE-2021-43854",
    "datePublished": "2021-12-23T17:55:11",
    "dateReserved": "2021-11-16T00:00:00",
    "dateUpdated": "2024-08-04T04:10:16.281Z",
    "state": "PUBLISHED"
  },
  "dataType": "CVE_RECORD",
  "dataVersion": "5.1"
}

Sightings

Author	Source	Type	Date

Nomenclature

Seen: The vulnerability was mentioned, discussed, or observed by the user.
Confirmed: The vulnerability has been validated from an analyst's perspective.
Published Proof of Concept: A public proof of concept is available for this vulnerability.
Exploited: The vulnerability was observed as exploited by the user who reported the sighting.
Patched: The vulnerability was observed as successfully patched by the user who reported the sighting.
Not exploited: The vulnerability was not observed as exploited by the user who reported the sighting.
Not confirmed: The user expressed doubt about the validity of the vulnerability.
Not patched: The vulnerability was not observed as successfully patched by the user who reported the sighting.

Detection rules are retrieved from Rulezet.

Action not permitted

PYSEC-2021-859

GHSA-F8M6-H2C7-8H9X

Impact

Patches

Workarounds

References

For more information

CVE-2021-43854 (GCVE-0-2021-43854)

Tags

Sightings

Nomenclature