Vote for next features

[gosom]

Please select the feature which you would like to see next.

Voting will be open until 25 Jun 2024

[admbyz]

I vote for "Ability to start and stop the crawler and resume from the previous position" because gui not that much important for me. Other features are nice to have but can be easily implemented / manipulated by scripting languages or excel/google sheets etc except long and lat feature.
Anyway for me crawler able to remember last state is most important one and think technically hardest one to implement. If you make it somehow auto not only dependant to start/resume/start command it will be killer feature. Because you can resume if server down, out of resource, program crashes etc.

[gosom]

I agree, that's a good feature. I am already working on it but needs more work.

[gosom]

I will try to implement lon and lat customization first.
thank you all

[devdovdav]

When scraping a lot of cities next to each other i get a lot of duplicate results.

Scraping all cities for a category that is not available in most cities will cause google to zoom out and collect results from other cities. But these results also appear when scraping the city where the result is actually in.

For a small country i scraped 20x more then needed. Which is huge.

Maybe adding a identifier to one of the columns in gmaps table and doing upsert based on it helps. For example using the google identifier (not sure if you have it at that time). But if first all queries in the txt file are done, followed by the 'scrape jobs' it will reduce the queries significantly.

Imagine doing scraping at scale (200k cities/villages in usa) and then do 20x more then needed. This will drastically decrease the total duration.

[graysuit]

Does it have progressbar ?
It will help to know how much done and remaining