Provider and template associations lost for Ansible Tower methods

[svinopterix]

Hello,

Sometimes the associations between Ansible Tower methods and Ansible Tower provider with the respective Job template are lost. Trying to debug this situation. Where those associations are stored in DB?

WBR,
Slava Medvedev

[Fryguy]

Can you verify whether anything about the Job is changing? In particular is the database id changing when this happens? It's possible there's some churn condition in the Ansible Tower refresh. cc @agrare

[svinopterix]

@agrare @Fryguy Hello guys,

Is it possible to temporary disable that automatic refresh? Because this provider detach happens very often and breaks automation workflow though.

PS: We tried to lock the domain with Tower methods but this didn't help.

[agrare]

@svinopterix if you set the refresh_interval to 0 that will disable the scheduled refresh. The setting would be ems_refresh/ansible_tower_automation/refresh_interval

[agrare]

That is an interesting failure, the collection it is failing on is configuration_scripts which isn't the first collection that we collect for: https://github.com/ManageIQ/manageiq-providers-ansible_tower/blob/morphy/app/models/manageiq/providers/ansible_tower/inventory/parser/automation_manager.rb#L7

If there were an issue with the credentials I would have expected it to have failed on the first access...

[svinopterix]

I think it's not the credentials problem. May be it's related to AWX that seldom returns erroneuos result. We notice that the problem happens from time to time when some job template dissapears from the list in MIQ. At that time the method with that template selection loses it's association with provider and template. Then we refresh the AWX provider, and that template returns.

[agrare]

Yeah I suspect you're right about that, either way a failed collection shouldn't cause records to be deleted. I wonder if the issue is actually a successful refresh after this temporary failure deletes them if they aren't returned by the tower API. It would also help to set the ansible_tower log-level to debug so we can see the payloads back-and-forth between the tower API and MIQ.

If you can turn debug logging on we will log the number of records created, updated, and deleted during a refresh. I'd be curious if after you notice one "goes missing" if you can grab the logs for the a few prior refreshes and see if it was actually deleted after the failed refresh or after a subsequent "successful" one.