Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ADD: unix socket support #458

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

bcachet
Copy link

@bcachet bcachet commented Nov 5, 2024

SUMMARY

This PR is adding support to VAULT_ADDR/url for unix socket.
This allow, for example, to connect with vault-agent listening onto Unix socket.

Implementation is following guidance proposed by hvac: https://hvac.readthedocs.io/en/latest/advanced_usage.html#vault-agent-unix-socket-listener

ISSUE TYPE
  • Feature Pull Request
COMPONENT NAME

HashiVaultHelper::get_vault_client

ADDITIONAL INFORMATION

Here is the task configuration

- name: Issue new PKI certificate
  environment:
    VAULT_ADDR: unix:///var/run/vault-agent.socket
  community.hashi_vault.vault_pki_generate_certificate:
    common_name: "{{ ansible_fqdn }}"
    role_name: host
    auth_method: none
  register: cert
  failed_when:
  - "'certificate' not in cert.data.data"
  - "'private_key' not in cert.data.data"

Before the change

TASK [vault : Issue new PKI certificate] ***************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: requests.exceptions.InvalidSchema: No connection adapters were found for 'unix:///var/run/vault-agent.socket/v1/pki/issue/host'
fatal: [<REDACTED>]: FAILED! => {"changed": false, "failed_when_result": "The conditional check ''certificate' not in cert.data.data' failed. The error was: error while evaluating conditional ('certificate' not in cert.data.data): 'dict object' has no attribute 'data'. 'dict object' has no attribute 'data'", "module_stderr": "Shared connection to <REDACTED> closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/home/exoadmin/.ansible/tmp/ansible-tmp-1730810202.0243812-1387826-143459003234056/AnsiballZ_vault_pki_generate_certificate.py\", line 107, in <module>\r\n    _ansiballz_main()\r\n  File \"/home/exoadmin/.ansible/tmp/ansible-tmp-1730810202.0243812-1387826-143459003234056/AnsiballZ_vault_pki_generate_certificate.py\", line 99, in _ansiballz_main\r\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n  File \"/home/exoadmin/.ansible/tmp/ansible-tmp-1730810202.0243812-1387826-143459003234056/AnsiballZ_vault_pki_generate_certificate.py\", line 47, in invoke_module\r\n    runpy.run_module(mod_name='ansible_collections.community.hashi_vault.plugins.modules.vault_pki_generate_certificate', init_globals=dict(_module_fqn='ansible_collections.community.hashi_vault.plugins.modules.vault_pki_generate_certificate', _modlib_path=modlib_path),\r\n  File \"/usr/lib/python3.10/runpy.py\", line 224, in run_module\r\n    return _run_module_code(code, init_globals, run_name, mod_spec)\r\n  File \"/usr/lib/python3.10/runpy.py\", line 96, in _run_module_code\r\n    _run_code(code, mod_globals, init_globals,\r\n  File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\r\n    exec(code, run_globals)\r\n  File \"/tmp/ansible_community.hashi_vault.vault_pki_generate_certificate_payload_da_vd70d/ansible_community.hashi_vault.vault_pki_generate_certificate_payload.zip/ansible_collections/community/hashi_vault/plugins/modules/vault_pki_generate_certificate.py\", line 286, in <module>\r\n  File \"/tmp/ansible_community.hashi_vault.vault_pki_generate_certificate_payload_da_vd70d/ansible_community.hashi_vault.vault_pki_generate_certificate_payload.zip/ansible_collections/community/hashi_vault/plugins/modules/vault_pki_generate_certificate.py\", line 282, in main\r\n  File \"/tmp/ansible_community.hashi_vault.vault_pki_generate_certificate_payload_da_vd70d/ansible_community.hashi_vault.vault_pki_generate_certificate_payload.zip/ansible_collections/community/hashi_vault/plugins/modules/vault_pki_generate_certificate.py\", line 270, in run_module\r\n  File \"/usr/lib/python3/dist-packages/hvac/api/secrets_engines/pki.py\", line 381, in generate_certificate\r\n    return self._adapter.post(\r\n  File \"/usr/lib/python3/dist-packages/hvac/adapters.py\", line 126, in post\r\n    return self.request(\"post\", url, **kwargs)\r\n  File \"/usr/lib/python3/dist-packages/hvac/adapters.py\", line 364, in request\r\n    response = super(JSONAdapter, self).request(*args, **kwargs)\r\n  File \"/usr/lib/python3/dist-packages/hvac/adapters.py\", line 313, in request\r\n    response = self.session.request(\r\n  File \"/usr/lib/python3/dist-packages/requests/sessions.py\", line 544, in request\r\n    resp = self.send(prep, **send_kwargs)\r\n  File \"/usr/lib/python3/dist-packages/requests/sessions.py\", line 651, in send\r\n    adapter = self.get_adapter(url=request.url)\r\n  File \"/usr/lib/python3/dist-packages/requests/sessions.py\", line 744, in get_adapter\r\n    raise InvalidSchema(\"No connection adapters were found for {!r}\".format(url))\r\nrequests.exceptions.InvalidSchema: No connection adapters were found for 'unix:///var/run/vault-agent.socket/v1/pki/issue/host'\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

After the change

TASK [vault : Issue new PKI certificate] ***************************************************************
changed: [<REDACTED>]

@briantist briantist added the enhancement New feature or request label Nov 5, 2024
@briantist
Copy link
Collaborator

Hi @bcachet thanks for opening this!

This error is not from your PR, it's failing in the whole collection and I'm not sure why yet so it will be addressed in a different PR at some point.

The other failures need to be looked into. I'd also like to see some tests added for this new functionality, ideally both units and integration but let's see how implementation goes for those tests.

@bcachet bcachet force-pushed the vault-agent_unix-socket branch 2 times, most recently from 8451c53 to 8ae1636 Compare November 5, 2024 16:42
@bcachet
Copy link
Author

bcachet commented Nov 5, 2024

The other failures need to be looked into. I'd also like to see some tests added for this new functionality, ideally both units and integration but let's see how implementation goes for those tests.

I added some unit tests around requests_unixsocket integration in HashiVaultHelper::get_vault_client.
I need to dive into how integration tests are working to be able to provide an integration test (I guess I will need to update docker-compose environment to add a vault-agent listening on Unix socket)

@briantist
Copy link
Collaborator

The error I mentioned before should be fixed in main now as of #459 , please rebase when you get a chance!

This PR is adding support to `VAULT_ADDR`/`url` for unix socket
This allow to connect with vault-agent listening onto Unix socket

```yaml
      community.hashi_vault.vault_pki_generate_certificate:
        common_name: "{{ ansible_fqdn }}"
        role_name: my-role
        auth_method: none
        engine_mount_point: pki/host/
```
* Validate that client.session is requests_unixsocket.Session when url is unix socket
* Validate that HashiVaultHelper.get_vault_client throws when url is unix socket and requests_unixsocket is not available
@bcachet
Copy link
Author

bcachet commented Nov 6, 2024

I would be interested in discussing about the integration tests strategy.
@briantist What is the best channel/tool to have a discussion about it ?

@briantist
Copy link
Collaborator

I would be interested in discussing about the integration tests strategy. What is the best channel/tool to have a discussion about it ?

This PR is the best place to discuss it I think

@bcachet
Copy link
Author

bcachet commented Nov 6, 2024

Do you have any idea/proposal for an integration test around this change ?
Should I add a vault agent alongside the vault server in the docker-compose environment ?

Copy link

codecov bot commented Nov 6, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.22%. Comparing base (afc77bd) to head (5c4744a).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #458   +/-   ##
=======================================
  Coverage   99.21%   99.22%           
=======================================
  Files         110      110           
  Lines        5767     5797   +30     
  Branches     1088     1094    +6     
=======================================
+ Hits         5722     5752   +30     
  Misses         36       36           
  Partials        9        9           
Flag Coverage Δ
env_docker-default 99.22% <100.00%> (+<0.01%) ⬆️
integration 85.26% <23.07%> (-0.41%) ⬇️
sanity 37.16% <15.38%> (-0.17%) ⬇️
target_auth_approle 89.47% <ø> (ø)
target_auth_aws_iam 50.00% <ø> (ø)
target_auth_azure 53.84% <ø> (ø)
target_auth_cert 86.36% <ø> (ø)
target_auth_jwt 91.30% <ø> (ø)
target_auth_ldap 89.47% <ø> (ø)
target_auth_none 100.00% <ø> (ø)
target_auth_token 71.42% <ø> (ø)
target_auth_userpass 85.71% <ø> (ø)
target_connection_options 74.76% <ø> (ø)
target_controller 83.39% <23.07%> (-0.43%) ⬇️
target_filter_vault_login_token 77.77% <ø> (ø)
target_import 37.16% <15.38%> (-0.17%) ⬇️
target_lookup_hashi_vault 85.29% <ø> (ø)
target_lookup_vault_ansible_settings 54.92% <23.07%> (-0.59%) ⬇️
target_lookup_vault_kv1_get 100.00% <ø> (ø)
target_lookup_vault_kv2_get 100.00% <ø> (ø)
target_lookup_vault_list 100.00% <ø> (ø)
target_lookup_vault_login 100.00% <ø> (ø)
target_lookup_vault_read 100.00% <ø> (ø)
target_lookup_vault_token_create 84.44% <ø> (ø)
target_lookup_vault_write 56.21% <23.07%> (-0.70%) ⬇️
target_module_utils 96.60% <100.00%> (+0.06%) ⬆️
target_module_vault_database_connection_configure 55.44% <23.07%> (-0.79%) ⬇️
target_module_vault_database_connection_delete ?
target_module_vault_database_connection_read 55.18% <23.07%> (-0.79%) ⬇️
target_module_vault_database_connection_reset 55.25% <23.07%> (-0.79%) ⬇️
target_module_vault_database_connections_list 54.68% <23.07%> (-0.78%) ⬇️
target_module_vault_database_role_create 54.68% <23.07%> (-0.78%) ⬇️
target_module_vault_database_role_delete 55.25% <23.07%> (?)
target_module_vault_database_role_read 55.18% <23.07%> (-0.79%) ⬇️
target_module_vault_database_roles_list 54.68% <23.07%> (-0.78%) ⬇️
target_module_vault_database_rotate_root_creds 55.07% <23.07%> (-0.79%) ⬇️
target_module_vault_database_static_role_create 55.44% <23.07%> (-0.79%) ⬇️
target_module_vault_database_static_role_get_creds 55.18% <23.07%> (-0.79%) ⬇️
target_module_vault_database_static_role_read 55.18% <23.07%> (-0.79%) ⬇️
target_module_vault_database_static_role_rotate_creds 55.07% <23.07%> (-0.79%) ⬇️
target_module_vault_database_static_roles_list 54.68% <23.07%> (-0.78%) ⬇️
target_module_vault_kv1_get 97.43% <ø> (ø)
target_module_vault_kv2_delete 55.88% <23.07%> (-0.80%) ⬇️
target_module_vault_kv2_get 97.36% <ø> (ø)
target_module_vault_kv2_write 56.31% <23.07%> (-0.79%) ⬇️
target_module_vault_list 96.96% <ø> (ø)
target_module_vault_login 93.93% <ø> (ø)
target_module_vault_pki_generate_certificate 84.61% <ø> (ø)
target_module_vault_read 96.96% <ø> (ø)
target_module_vault_token_create 91.66% <ø> (ø)
target_module_vault_write 55.08% <23.07%> (-0.78%) ⬇️
target_modules 88.67% <15.38%> (-0.28%) ⬇️
units 96.37% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@briantist briantist left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the things I mentioned in the review should be failing tests, if we were testing the right things, so before trying to address them, directly, I'd like to see the tests modified such that they are catching these conditions (failing) and then we could change the code more.

It's possible that unit tests will be enough, if we are testing the right conditions, so look for tests that test env var precedence, custom session creation, connection retries, etc.

But we may be able to implement minimal integration without adding a Vault agent to the test environment, by catching error messages related to the unix socket, so that we know it's failing for the "right" reason (the socket doesn't exist). For now, I think there's a lot that can be done with units.

@@ -57,6 +61,20 @@ def get_vault_client(
:type hashi_vault_revoke_on_logout: bool
'''

url = kwargs.get('url') or os.environ.get('VAULT_ADDR')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to remove the env var retrieval here. This collection specially processes some env vars and VAULT_ADDR is one of them. I think this method will happen after env late-binding so hopefully we can just remove the or condition here.

See also:

.replace('/', '%2F'))
socket_url = f"http+unix://{socket_path}"
kwargs['url'] = socket_url
kwargs['session'] = requests_unixsocket.Session()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some concerns that creating a blank session here is going to end up erasing certain other connection options if they are specified. We have helper methods for this purpose, see:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants