Skip to content

using sbarbett.pihole.manage_lists with large blocklists takes really long #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
muller2002 opened this issue Mar 17, 2025 · 3 comments
Assignees
Labels
bug Something isn't working enhancement New feature or request

Comments

@muller2002
Copy link
Contributor

Hey,

when trying out version 1.0.5 i wanted to use the new sbarbett.pihole.manage_lists.

I have a large collection of ~200 Blocklists. Each blocklist takes around 3 seconds to update, so around 10 minutes per host.

The solution proposed in #2 was faster (around 30seconds per run).

I propose to add a new module, which does the handling of bulk updates and change the role to just call this module for every pihole

@sbarbett sbarbett self-assigned this Mar 17, 2025
@sbarbett sbarbett added bug Something isn't working enhancement New feature or request labels Mar 17, 2025
@sbarbett
Copy link
Owner

Hi Marlena,

There's a reason for the delay, as you can see here:

https://github.com/sbarbett/pihole-ansible/blob/main/roles/manage_lists/tasks/process_list.yml

When I was developing the module and role for managing local DNS records, I ran into some issues with the PiHole API that, at least at a glance, appeared to be rate limiting. To work around this, I added a 1-second delay between each transaction.

When I later added the role for managing lists, I copied that logic under the assumption that similar issues might arise. However, I haven’t actually tested it with a large number of lists.

Can you try running your list updates with the following section in process_list.yml commented out?

    - name: Pause for 1 second
      command: sleep 1
      changed_when: false
      no_log: true 

If omitting the delay doesn’t cause any issues, feel free to submit a PR with it removed, and I’ll merge it. You could also try something shorter, like 100ms.

Thanks,
Shane

@muller2002
Copy link
Contributor Author

muller2002 commented Mar 17, 2025

Hey, already tested that, it speeds up around 30% (2 instead of 3seconds) but still very slow.

I started bulding a mix of your and my solution here: https://github.com/muller2002/pihole-ansible/blob/main/plugins/modules/bulk_list_update.py
But I definetly need to test it more, but first tries it worked.

Please let me know, if you have any thougts about this

muller2002 added a commit to muller2002/pihole-ansible that referenced this issue Mar 17, 2025
This will lead to a lot faster updates, fixes sbarbett#3
@sbarbett
Copy link
Owner

After some consideration, I revised the lower module and implemented your approach in v1.1.0. Eliminating the unnecessary API calls should considerably reduce the time it takes to run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants