WidgetsPool

[meeffe]

Hello and thanks for your effort for creating this repo

For a pretty long time I try to troubleshoot why the WidgetsPool does not work. One thing I can see is this:

future: <Task finished name='Task-2' coro=<WidgetsPool.worker() done, defined at /aioTrends/Trends.py:245> exception=KeyError('keywords')>
Traceback (most recent call last):
  File "/Users/almeco/Downloads/projects/g_trends/aioTrends/Trends.py", line 251, in worker
    await self.widgetInterestOverTime(i, tid, cookies, proxy, userAgent)
  File "/Users/almeco/Downloads/projects/g_trends/aioTrends/Trends.py", line 208, in widgetInterestOverTime
    logger.info(f"DEBUGGING self.qrys[tid]['keywords']: {self.qrys[tid]['keywords']}")
KeyError: 'keywords'

Could you check at your side if everything is running smoothly?

Cookies generating works fine.

edit:
After running cookies generation part my qrys.pkl file after reading it look like this:

from pandas import Timestamp

qrys = {1: {'tickers': ['FTEV'], 'periods': [Timestamp('2021-04-22 00:00:00'), Timestamp('2021-12-21 00:00:00')]},
        2: {'tickers': ['FLM'], 'periods': [Timestamp('2007-01-02 00:00:00'), Timestamp('2007-09-01 00:00:00')]}}
payload = {
    'hl': "en-US",
    'tz': 360,
    'req': {
        'comparisonItem': [
            {'keyword': kw, 'time': qrys[1]['periods'], 'geo': 'US'} for kw in
            qrys[1]['keywords']
        ],
        'category': 0,
        'property': ""
    }
}
print(payload)

and this gave me Keyerror here

[yuz0101]

Hi meefe, thanks for your interest in this project and efforts in troubleshooting. Adding a freq in the queries dataset (qrys.pkl) should fix the issue. Please see below scrn shoot for my tested results. I will update the ReadMe. Sorry for causing any inconvenience to you.

[meeffe]

oh, thanks for your reply. Don't worry. I spend almost an entire day to came with a solution. The problem was that I didn't know what should be put inside qrys.pkl file and how it corresponds with queries.pkl file

And now Im still getting some errors e.g. I had to put full paths to files but stil when e.g. generating widgets I'm getting:

2023-07-24 16:05:31,824Error in filterQrys unpkl pathWgtEptyRes: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
  File "/Users/almeco/Downloads/projects/g_trends/aioTrends/Trends.py", line 575, in filterQrys
    dkeys = await unpkl(self.pathWgtEptyRes, self.lock)
  File "/Users/almeco/Downloads/projects/g_trends/aioTrends/Trends.py", line 149, in unpkl
    async with aiofiles.open(path, 'rb') as f:
  File "/Users/almeco/Downloads/projects/g_trends/venv/lib/python3.10/site-packages/aiofiles/base.py", line 98, in __aenter__
    self._obj = await self._coro
  File "/Users/almeco/Downloads/projects/g_trends/venv/lib/python3.10/site-packages/aiofiles/threadpool/__init__.py", line 97, in _open
    f = yield from loop.run_in_executor(executor, cb)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/.../Downloads/projects/g_trends/aioTrends/data/widgetReceiveEmptyReponse.pkl'

This file is not created. When I try to create it manually (which is not a good idea I believe) it return ran out of input

Like you said it would be a good idea to add some content to Readme if you don't mind.

Also in example.py file an examples regarding actually putting trends to files would be helpful like
at.Aio(1).getMonthlyData(['tickers', 'PHEV'], start='2021-04-22', end='2021-12-21')

Overall its a great piece of code. I finally got it running (but struggle with final setup quite a bit now)

PS. why pkl data format? they are hard to read outside of a python script. Why e.g. not simple json?

[yuz0101]

I updated the project, could you please have a test on your end?

[yuz0101]

got some bugs when using the Aio class, will fix that when I am free

[meeffe]

yes, me too. I will post some of them later or tomorrow. Thanks for your efforts. One thing which I noticed that adding proxies should be more flexible e.g. I use proxies without login and password - 00.00.00.00:8800 (authenticated by IP) and I had to change the reading proxy format in code. Quick fix but perhaps it will be wise to specify which proxy format someone use so that there will be no need to modify the script

[meeffe]

also more examples of actually fetching the data using different methods when using the script will be welcomed mate;) Great work overall

[yuz0101]

thanks mate : ) for all your time and suggestions. I will look at it when I am free. I created this repo months ago for my research project, and I barely have time to maintain it after the project. I am wondering if you are available to be the contributor to this repo. I massively appreciate all your help.