Use cases of Watchtower

[zurferr]

Hi guys,

I am trying to understand if watchtower is suited for production updates and how you are using it?

Right now I am building a self-hosted webapp that constists of 4 docker containers (started with docker-compose) and try to find a good way to update them. The containers will be deployed on customers private VMs. So I have no direct access to them.

The whole idea of instantly pushing new changes to running containers seems a bit dangerous and instable for productive apps.
I would at least like to have some successful integration tests before that. What are your thoughts?

Best Regards
Rick

[zoispag]

Hi Rick,

I am using it for about 2 years now in production. The danger comes from your docker build, so if you are not sure about the quality of your produced image, you might encounter issues, but this is not really related to watchtower itself.

That being said, watchtower does not currently support any rollback mechanism, so you would have to manually push a new image to fix any issues.

What kind of integration tests would you have in mind?

[simskij]

The whole idea of instantly pushing new changes to running containers seems a bit dangerous and instable for productive apps. I would at least like to have some successful integration tests before that. What are your thoughts?

I would too!

While I don't want to either encourage or discourage the use of watchtower in production (I've used it myself in a quite big setup), I'd probably do something along the lines of:

1. Start CI on changes to the main branch
2. Run linting and format checks.
3. Run unit tests and integration tests that don't have any external dependencies
4. Build the docker image
5. Launch the docker image in the CI runners docker daemon
6. Run additional black box integration tests against this container, together with any other containers you might need for it to be valuable
7. Push the image to your registry

Fail and cancel the build at any time if any of the steps are unsuccessful.

[zoispag]

I think lint/format/unit tests go beyond question, when we talk about software quality. These are important and related to running Docker in production. A well-thought CI procedure here is needed of course. What I might add here as a help, is that in my CI, I also run Infra tests fo my Dockerfiles. I am using serverspec using Ruby. A test for example looks like this:

describe "Dockerfile" do
  before(:all) do
    @path = './'
    @image = ::Docker::Image.build_from_dir(@path, 't' => 'platform:rspec')

    set :os, family: :debian
    set :backend, :docker
    set :docker_image, @image.id
  end

  # Verify services
  describe service('apache2') do
    it { should be_enabled }
    it { should be_running }
  end

  describe 'verify packages' do
    %w{zip libssh2-1-dev}.each do |packageName|
      describe package(packageName) do
        it { should be_installed }
      end
    end
  end

  describe 'verify port' do
    describe command("curl --insecure https://localhost:443 2> /dev/null && echo true || echo false") do
      it "confirm port 443" do
        expect(subject.stdout).to match(/true/)
      end
    end
  end
  end
end

The above is only a small portion of serverspec tests of one of my Dockerfiles.

Only when my style, lint, unit, e2e & infra tests are successful, I actually build/push my Dockerfile.

Hope this helps :)

[zurferr]

Thanks guys, that seems like a reasonable workflow to minimize errors.

Initially I was thinking along the lines of:

• push locally tested images to registry
• watchtower pulls new images from remote VMs
• watchtower runs tests on remote VMs
• if successful, stop running images and start new images
• maybe have the potential to reverse back to an earlier version

I guess my problem is that I am looking for a self-contained solution, which I guess does not exist, since it is always specific for the application stack.

[simskij]

Yeah. A couple of people have been asking for some kind of rollback mechanism over the years, but I've always been resisting it. The only error that should be possible™️, given a solid CI process, would be protocol level errors during the pull, which I hope and think docker itself guard against.