Prevent Old bosh from being used while bosh-init is running

[geofffranks]

We have a concourse pipeline to deploy bosh via bosh-init. The first thing it does is check the bosh director for any running tasks. If there are any, it waits for them to finish before trying to redeploy the bosh vm (either via bosh-init or bosh micro deploy). Since bosh-init adds a bunch of intermediary time compiling everything, before tearing down the bosh vm, there's about a 10-15 minute window between checking for bosh tasks, and turning off the bosh director. This makes it very possible to come in and start up a task while bosh-init is about to turn off + delete the old vm. This seems like a bad thing.

Any chance a step could be inserted into the beginning of bosh-init to monit stop all? If bosh-init fails before it deletes the previous bosh vm, the processes would be restarted via monit start all. This would help prevent any background tasks from happening while bosh-init does its compilation, and stemcell/release downloading, and since the VM was about to be destroyed anyway, doesn't seem like it inserts any undesired downtime.

[cppforlife]

im not sure i understand what to monit stop since bosh-init deletes and then creates new vm. new vm is empty until all compilation is done.

Sent from my iPhone

On Nov 19, 2015, at 8:57 AM, Geoff Franks notifications@github.com wrote:

We have a concourse pipeline to deploy bosh via bosh-init. The first thing it does is check the bosh director for any running tasks. If there are any, it waits for them to finish before trying to redeploy the bosh vm (either via bosh-init or bosh micro deploy). Since bosh-init adds a bunch of intermediary time compiling everything, before tearing down the bosh vm, there's about a 10-15 minute window between checking for bosh tasks, and turning off the bosh director. This makes it very possible to come in and start up a task while bosh-init is about to turn off + delete the old vm. This seems like a bad thing.

Any chance a step could be inserted into the beginning of bosh-init to monit stop all? If bosh-init fails before it deletes the previous bosh vm, the processes would be restarted via monit start all. This would help prevent any background tasks from happening while bosh-init does its compilation, and stemcell/release downloading, and since the VM was about to be destroyed anyway, doesn't seem like it inserts any undesired downtime.

—
Reply to this email directly or view it on GitHub.

[geofffranks]

In my experience, bosh-init validates, downloads releases, compiles the cpi for instrumenting the delete/create of the VM, deletes/recreates the vm, and then does compilation of the releases that will be put on the VM.

The validation/download/initial cpi compilation takes a number of minutes, all on its own, before the delete happens.

On Nov 19, 2015, at 11:59 AM, Dmitriy Kalinin notifications@github.com wrote:

im not sure i understand what to monit stop since bosh-init deletes and then creates new vm. new vm is empty until all compilation is done.

Sent from my iPhone

On Nov 19, 2015, at 8:57 AM, Geoff Franks notifications@github.com wrote:

We have a concourse pipeline to deploy bosh via bosh-init. The first thing it does is check the bosh director for any running tasks. If there are any, it waits for them to finish before trying to redeploy the bosh vm (either via bosh-init or bosh micro deploy). Since bosh-init adds a bunch of intermediary time compiling everything, before tearing down the bosh vm, there's about a 10-15 minute window between checking for bosh tasks, and turning off the bosh director. This makes it very possible to come in and start up a task while bosh-init is about to turn off + delete the old vm. This seems like a bad thing.

Any chance a step could be inserted into the beginning of bosh-init to monit stop all? If bosh-init fails before it deletes the previous bosh vm, the processes would be restarted via monit start all. This would help prevent any background tasks from happening while bosh-init does its compilation, and stemcell/release downloading, and since the VM was about to be destroyed anyway, doesn't seem like it inserts any undesired downtime.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub #58 (comment).

[cppforlife]

Ah that's what you mean by time spent. Currently bosh-init does not call drain agent API before stopping and deleting the VM. Drain script for the Director will wait for the tasks to finish, so once that happens it will wait for all tasks to finish.