Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ERROR: Task was destroyed but it is pending! #188

Open
ma-pony opened this issue Mar 31, 2023 · 10 comments
Open

ERROR: Task was destroyed but it is pending! #188

ma-pony opened this issue Mar 31, 2023 · 10 comments
Labels
upstream issue

Comments

@ma-pony
Copy link
Contributor

ma-pony commented Mar 31, 2023 •
edited by elacuesta
Loading

I sometimes get this error when i use scrapy-pilaywright

2023-03-31 09:33:35 [asyncio] ERROR: Task was destroyed but it is pending!
source_traceback: Object created at (most recent call last):
  File "/Users/rccpony/PycharmProjects/bid-spiders/packages/universal_spiders/main.py", line 5, in <module>
    cmdline.execute(
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/scrapy/cmdline.py", line 154, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/scrapy/cmdline.py", line 109, in _run_print_help
    func(*a, **kw)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/scrapy/cmdline.py", line 162, in _run_command
    cmd.run(args, opts)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/scrapy/commands/crawl.py", line 27, in run
    self.crawler_process.start()
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/scrapy/crawler.py", line 363, in start
    reactor.run(installSignalHandlers=False)  # blocking call
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/twisted/internet/asyncioreactor.py", line 257, in run
    self._asyncioEventloop.run_forever()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
    self._run_once()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1901, in _run_once
    handle._run()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 249, in run
    await self._transport.run()
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_transport.py", line 155, in run
    self.on_message(obj)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 210, in <lambda>
    self._transport.on_message = lambda msg: self.dispatch(msg)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 356, in dispatch
    object._channel.emit(
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/base.py", line 176, in emit
    handled = self._call_handlers(event, args, kwargs)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/base.py", line 154, in _call_handlers
    self._emit_run(f, args, kwargs)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/asyncio.py", line 48, in _emit_run
    coro: Any = f(*args, **kwargs)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_browser_context.py", line 131, in <lambda>
    lambda params: self._on_request(
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_browser_context.py", line 485, in _on_request
    page.emit(Page.Events.Request, request)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/base.py", line 176, in emit
    handled = self._call_handlers(event, args, kwargs)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/base.py", line 154, in _call_handlers
    self._emit_run(f, args, kwargs)
  File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/asyncio.py", line 57, in _emit_run
    fut: Any = ensure_future(cast(Any, coro), loop=self._loop)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/tasks.py", line 615, in ensure_future
    return _ensure_future(coro_or_future, loop=loop)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/tasks.py", line 636, in _ensure_future
    return loop.create_task(coro_or_future)
task: <Task pending name='Task-386' coro=<_make_request_logger.<locals>._log_request() running at /Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/scrapy_playwright/handler.py:606> wait_for=<Future pending cb=[Task.task_wakeup()] created at /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py:429> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/asyncio.py:65, ProtocolCallback.__init__.<locals>.cb() at /Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_connection.py:168] created at /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/tasks.py:636>

[scrapy.utils.log] INFO: Scrapy 2.7.0 started (bot: universal_spiders)
[scrapy.utils.log] INFO: Versions: lxml 4.9.2.0, libxml2 2.9.13, cssselect 1.2.0, parsel 1.7.0, w3lib 2.1.1, Twisted 21.7.0, Python 3.10.10 (v3.10.10:aad5f6a891, Feb  7 2023, 08:47:40) [Clang 13.0.0 (clang-1300.0.29.30)], pyOpenSSL 22.1.0 (OpenSSL 3.0.7 1 Nov 2022), cryptography 38.0.4, Platform macOS-13.2.1-arm64-arm-64bi

(Edit) Reproducible example at #233
@stefanondisponibile
Copy link

Encountering the same issue here.

2023-05-24 09:10:35 [scrapy.core.engine] INFO: Spider closed (finished)
2023-05-24 09:10:35 [scrapy-playwright] INFO: Closing download handler
2023-05-24 09:10:35 [scrapy-playwright] INFO: Closing browser
2023-05-24 09:10:35 [scrapy-playwright] INFO: Closing download handler
2023-05-24 09:10:35 [scrapy-playwright] INFO: Closing browser
2023-05-24 09:10:35 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-5112' coro=<_make_request_logger.<locals>._log_request() running at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/scrapy_playwright/handler.py:606> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/pyee/asyncio.py:65, ProtocolCallback.__init__.<locals>.cb() at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/playwright/_impl/_connection.py:168]>
2023-05-24 09:10:35 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-5113' coro=<_make_request_logger.<locals>._log_request() running at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/scrapy_playwright/handler.py:606> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/pyee/asyncio.py:65, ProtocolCallback.__init__.<locals>.cb() at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/playwright/_impl/_connection.py:168]>
2023-05-24 09:10:35 [asyncio] ERROR: Task was destroyed but it is pending!
...[many more here]

Logs seem to point here (using scrapy-playwright 0.0.26) but when I read this question I started wondering if this could be related too.

A nice starting point would be understanding if this is a scrapy-playwright or playwright kind of issue. What do you think?

@auxsvr
Copy link

auxsvr commented May 28, 2023

This seems to be related to jfhbrook/pyee#120. Keeping a reference to the tasks prevents this in my tests.

@elacuesta
Copy link
Member

Please share a minimal, reproducible example.

@ma-pony
Copy link
Contributor Author

ma-pony commented Jul 25, 2023

Please share a minimal, reproducible example.

This was an occasional issue, the code that caused the problem was removed by me at the time, and i will try to reproduce the problem recently

@elacuesta elacuesta added the Stale label Sep 4, 2023
@elacuesta elacuesta mentioned this issue Sep 12, 2023
Closed
@elacuesta
Copy link
Member

Using the example from #233 I'm seeing much less of these messages after upgrading pyee from 9.0.4 to 11.0.0.

@elacuesta elacuesta mentioned this issue Sep 12, 2023
Closed
@mgrist
Copy link

mgrist commented Sep 15, 2023 •
edited
Loading

@elacuesta Any luck on a solution? I have a long-running scraper with this issue as well. It's not consistent and happens intermittently. Sometimes errors in 10 minutes, sometimes after 4 hours.

I just took your advice and upgraded my Pyee version to 11.0, I will try again.

@elacuesta
Copy link
Member

@elacuesta Any luck on a solution?

I don't have plans to work on this, from what I see it's an upstream issue.

@ThinksFast
Copy link

I'm getting a ton of these errors all of a sudden. I was running scrapy + playwright without issue for several months. Was this issue raised with Pyee?

For reference, my environment is using:
python 3.11.6
scrapy 2.11.0
scrapy-playwright 0.0.33
twisted 22.10.0
pyee 11.0.1

Example logs:

2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4511' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4514' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4512' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4515' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4763' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4766' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4764' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4767' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4772' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4773' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4797' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4798' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4809' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4812' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4810' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4813' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4863' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4864' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4869' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4872' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4875' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4870' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4873' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4878' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4881' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4879' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4882' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4884' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4887' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4885' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4888' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4876' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4931' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4932' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>

@mgrist
Copy link

mgrist commented Nov 15, 2023

@ThinksFast I was never able to solve the issue here. I ended up migrating to scrapy-splash.

@ThinksFast
Copy link

@ThinksFast I was never able to solve the issue here. I ended up migrating to scrapy-splash.

Thanks for the follow up! I found out earlier today I was getting this issue due to an exception in my Scrapy pipeline file. The pipeline function to save the data was throwing an exception due to Unique Constraints set up on the table columns. But this exception wasn't getting logged for some reason, it was silent. Anyway, after fixing that, the errors went away for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
upstream issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@elacuesta @auxsvr @stefanondisponibile @ma-pony @mgrist @ThinksFast