Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow running detached app commands and follow logs by container ID #906

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Allow running detached app commands and follow logs by container ID #906

wants to merge 6 commits into from
+86 −21

Conversation

aliismayilov
Copy link

@aliismayilov aliismayilov commented Aug 7, 2024 •
edited
Loading

This is useful for long running rake tasks or scripts that can be run without having to keep an open connection to servers.

The command that starts a detached container exits immediately and the output leaves the container ID that can be used for following the logs or investigating them later. To support this, I've also extended the kamal logs command with --container-id option.

Example:

❯ kamal app exec --detach 'bin/rails hello'
Get most recent version available as an image...
Launching command with version latest from new container...
  INFO [21e97fa4] Running docker run --detach --network kamal --env SOLID_QUEUE_IN_PUMA="true" --env-file .kamal/apps/dummy_rails/env/roles/web.env --log-driver "awslogs" --log-opt awslogs-region="eu-central-1" --log-opt awslogs-group="/ec2/dummy-rails" --log-opt awslogs-force-flush-interval-seconds="1" --volume dummy_rails_storage:/rails/storage 233638847975.dkr.ecr.eu-central-1.amazonaws.com/dummy_rails:latest bin/rails hello on 3.77.153.204
  INFO [21e97fa4] Finished in 0.438 seconds with exit status 0 (successful).
App Host: 3.77.153.204
d5bba7b800f054a217623428f0926585b90e075fdd46e368083323e706f1e090

❯ kamal logs --follow --container-id=d5bba7b800f054a217623428f0926585b90e075fdd46e368083323e706f1e090
  INFO Following logs on 3.77.153.204...
  INFO ssh -t root@3.77.153.204 -p 22 'echo d5bba7b800f054a217623428f0926585b90e075fdd46e368083323e706f1e090 | xargs docker logs --timestamps --tail 10 --follow 2>&1'
2024-10-26T13:03:20.334253736Z logger says: 3
2024-10-26T13:03:21.334471050Z logger says: 4
2024-10-26T13:03:22.334622132Z logger says: 5
2024-10-26T13:03:23.334916155Z logger says: 6
2024-10-26T13:03:24.335143905Z logger says: 7
2024-10-26T13:03:25.335388754Z logger says: 8
2024-10-26T13:03:26.335653526Z logger says: 9
2024-10-26T13:03:27.335893494Z logger says: 10
2024-10-26T13:03:28.336115025Z logger says: 11
2024-10-26T13:03:29.336400979Z logger says: 12
2024-10-26T13:03:30.336642630Z logger says: 13
2024-10-26T13:03:31.336861355Z logger says: 14
2024-10-26T13:03:32.337858183Z logger says: 15
2024-10-26T13:03:33.338105717Z logger says: 16
2024-10-26T13:03:34.338330317Z logger says: 17
2024-10-26T13:03:35.338555287Z logger says: 18
2024-10-26T13:03:36.338794180Z logger says: 19
2024-10-26T13:03:37.339028060Z logger says: 20
Connection to 3.77.153.204 closed.
❯ kamal logs --container-id=d5bba7b800f054a217623428f0926585b90e075fdd46e368083323e706f1e090
  INFO Following logs on 3.77.153.204...
  INFO ssh -t root@3.77.153.204 -p 22 'echo d5bba7b800f054a217623428f0926585b90e075fdd46e368083323e706f1e090 | xargs docker logs --timestamps --tail 10 --follow 2>&1'
2024-10-26T13:03:28.336115025Z logger says: 11
2024-10-26T13:03:29.336400979Z logger says: 12
2024-10-26T13:03:30.336642630Z logger says: 13
2024-10-26T13:03:31.336861355Z logger says: 14
2024-10-26T13:03:32.337858183Z logger says: 15
2024-10-26T13:03:33.338105717Z logger says: 16
2024-10-26T13:03:34.338330317Z logger says: 17
2024-10-26T13:03:35.338555287Z logger says: 18
2024-10-26T13:03:36.338794180Z logger says: 19
2024-10-26T13:03:37.339028060Z logger says: 20
Connection to 3.77.153.204 closed.

To support logs inspection, I've also applied kamal logging config to all newly started app exec containers. This resolves #1154.

djmb
djmb requested changes Oct 23, 2024
View reviewed changes
Copy link
Collaborator

@djmb djmb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @aliismayilov,

Sorry for the delay in getting to this! I think we just need some checks for invalid option combinations and we should be good to go!

lib/kamal/cli/app.rb Show resolved Hide resolved
@djmb
Copy link
Collaborator

djmb commented Oct 23, 2024

Also it looks like there are some test failures - CliAppTest#test_exec_detach and CommandsAppTest#test_execute_in_new_detached_container

@aliismayilov aliismayilov force-pushed the detached-run branch 2 times, most recently from 6a21493 to a978ed2 Compare October 23, 2024 14:57
@aliismayilov
Copy link
Author

Hi @aliismayilov,
Sorry for the delay in getting to this! I think we just need some checks for invalid option combinations and we should be good to go!

Thanks for looking into this! I've added an incompatibility check. See if it makes sense.

Also it looks like there are some test failures - CliAppTest#test_exec_detach and CommandsAppTest#test_execute_in_new_detached_container

Not anymore! I must've missed them in the avalanche of log lines 😄

@aliismayilov
Copy link
Author

I just realized that there will be no output from a detached container in the logs. I created a separate issue: #1154

@aliismayilov
Allow running detached app commands
e3085ee 
this is useful for long running rake tasks or scripts
that can be run without having to keep open connection to the server.

Example:
```
kamal app exec 'bin/rails db:backfill_task' --detach
```
@aliismayilov
Make --detach incompatible with reuse or interactive
b4b9b18
@aliismayilov
Enable logging on app exec new containers
690aa98
@aliismayilov
Follow logs of a specific container
9896acc
@aliismayilov aliismayilov changed the title Allow running detached app commands Allow running detached app commands and follow logs by container ID Oct 26, 2024
@aliismayilov aliismayilov marked this pull request as draft October 26, 2024 13:29
@aliismayilov aliismayilov force-pushed the detached-run branch 2 times, most recently from 5a7fdb7 to fb1b4bf Compare October 26, 2024 13:45
@aliismayilov aliismayilov marked this pull request as ready for review October 26, 2024 13:50
@aliismayilov
Copy link
Author

aliismayilov commented Oct 26, 2024 •
edited
Loading

@djmb I've updated the pull request with more changes to support logs inspection for detached containers. In short:

  • all newly started app containers get kamal logging configuration applied
  • detached containers do not have --rm option. This allows inspection of detached containers even after they finished running
  • kamal logs gets a new option --container-id to inspect logs of a specific container

I've also updated the pull request description.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@djmb djmb djmb requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

App exec logs
2 participants
@aliismayilov @djmb