-
-
Notifications
You must be signed in to change notification settings - Fork 631
Memory leak: rotate_tls_ciphers creates new CipherSuiteAdapter on every request without closing the old one #328
Description
Bug Description
rotate_tls_ciphers=True (the default) causes a memory leak by creating a new CipherSuiteAdapter on every HTTP request without closing the previous one.
Root Cause
In cloudscraper/__init__.py, the request() method calls _rotate_tls_cipher_suite() on every request when rotate_tls_ciphers=True. This method creates a new CipherSuiteAdapter (which wraps a urllib3.PoolManager + ssl.SSLContext) and mounts it via self.mount('https://', ...).
The problem: requests.Session.mount() simply replaces the dict entry — it does not call .close() on the old adapter. The orphaned PoolManager objects hold references to sockets, threading.Condition, collections.deque, and threading.RLock, which prevent garbage collection.
Impact
- ~1 leaked PoolManager per HTTP request
- ~56 MB/min RSS growth under moderate load (~150 req/min)
- OOM kill within minutes in containerized environments (e.g., 512MB → OOM in ~7 min)
Reproduction
import cloudscraper
import os
def get_rss_mb():
# Linux
with open('/proc/self/status') as f:
for line in f:
if line.startswith('VmRSS:'):
return int(line.split()[1]) / 1024
return 0
scraper = cloudscraper.create_scraper(rotate_tls_ciphers=True) # default
rss_before = get_rss_mb()
for i in range(500):
scraper.get('https://httpbin.org/get')
if i % 100 == 0:
print(f"Request {i}: RSS = {get_rss_mb():.0f} MB (delta: +{get_rss_mb() - rss_before:.0f} MB)")RSS will grow linearly and never stabilize.
Suggested Fix
Close the old adapter before mounting the new one in _rotate_tls_cipher_suite():
def _rotate_tls_cipher_suite(self):
# Close old adapter to release PoolManager resources
old_adapter = self.get_adapter('https://')
if old_adapter:
old_adapter.close()
self.mount(
'https://',
CipherSuiteAdapter(...)
)Alternatively, reuse the existing adapter and only rotate the cipher list without creating a new PoolManager each time.
Workaround
Pass rotate_tls_ciphers=False and manage session rotation externally:
scraper = cloudscraper.create_scraper(rotate_tls_ciphers=False)Possibly Related
Issue #323 ("3.0内存溢出" / memory overflow in 3.0) may be caused by this same leak.
Environment
- cloudscraper 3.0.0
- Python 3.14
- Linux (Docker container with jemalloc)