When using compiled releases, an incorrect stemcell sha leads to a panic

[voelzmo]

When updating stemcells for a bosh-init deployment, I often just update the stemcell version in the manifest and have bosh-init tell me what the sha1 should be, so I don't have to download the same file twice. This doesn't work when using compiled releases with bosh-init.

I use bosh-init version 0.0.81-775439c-2015-12-09T00:36:04Z

Steps to reproduce

• Use at least one compiled release in your bosh-init manifest, e.g.

releases:
   - name: bosh
     url: file://release-bosh-255.3-on-ubuntu-trusty-stemcell-3191.tgz

• Reference a stemcell with an incorrect sha1.

resource_pools:
  - name: vms
    network: private
    stemcell:
      url: https://bosh.io/d/stemcells/bosh-openstack-kvm-ubuntu-trusty-go_agent?v=3191
      sha1: f3b0ed251700048c14c722cd995eb7d27e743fe9

Expected

An error message like this one

  Downloading stemcell... Failed (00:01:17)
Failed validating (00:01:18)

Command 'deploy' failed:
  Failed to download from 'https://bosh.io/d/stemcells/bosh-openstack-kvm-ubuntu-trusty-go_agent?v=3191':
    SHA1 of downloaded file '3c520267dc7671d538d2e94ea9d382196ff1fca5' does not match expected SHA1 'f3b0ed251700048c14c722cd995eb7d27e743fe9'

Actual

bosh-init output is pretty silent

Started validating
  Validating release 'bosh'... Finished (00:01:18)
  Downloading release 'bosh-openstack-cpi'... Skipped [Found in local cache] (00:00:00)
  Validating release 'bosh-openstack-cpi'... Finished (00:00:00)
  Validating cpi release... Finished (00:00:00)
  Validating deployment manifest... Finished (00:00:00)
  Downloading stemcell... Failed (00:00:39)

And the debug log shows a panic happening

File System] 2016/03/09 11:48:20 DEBUG - Checking if file exists /Users/cpi/.bosh_init/downloads/5bdaba97758737e8e606ff09beae1493fa150504-f3b0ed251700048c14c722cd995eb7d27e743fe9
[attemptRetryStrategy] 2016/03/09 11:48:20 DEBUG - Making attempt #0
[File System] 2016/03/09 11:48:20 DEBUG - Creating temp file with prefix tarballProvider
[File System] 2016/03/09 11:48:32 DEBUG - Remove all /Users/cpi/.bosh_init/installations/fa968599-fd1d-416e-64d5-aaf2c18faf9c/tmp/tarballProvider056288482
[attemptRetryStrategy] 2016/03/09 11:48:32 DEBUG - Making attempt #1
[File System] 2016/03/09 11:48:32 DEBUG - Creating temp file with prefix tarballProvider
[File System] 2016/03/09 11:48:45 DEBUG - Remove all /Users/cpi/.bosh_init/installations/fa968599-fd1d-416e-64d5-aaf2c18faf9c/tmp/tarballProvider556844249
[attemptRetryStrategy] 2016/03/09 11:48:45 DEBUG - Making attempt #2
[File System] 2016/03/09 11:48:45 DEBUG - Creating temp file with prefix tarballProvider
[File System] 2016/03/09 11:48:59 DEBUG - Remove all /Users/cpi/.bosh_init/installations/fa968599-fd1d-416e-64d5-aaf2c18faf9c/tmp/tarballProvider683724900
[File System] 2016/03/09 11:48:59 DEBUG - Remove all /Users/cpi/.bosh_init/installations/fa968599-fd1d-416e-64d5-aaf2c18faf9c/tmp/bosh-init-release374149680
[File System] 2016/03/09 11:49:01 DEBUG - Remove all /Users/cpi/.bosh_init/installations/fa968599-fd1d-416e-64d5-aaf2c18faf9c/tmp/bosh-init-release575718863
[Main] 2016/03/09 11:49:01 ERROR - Panic: runtime error: invalid memory address or nil pointer dereference
********************
/tmp/build/72b64e5e-29db-42e4-5635-09c9cadd52b5/gopath/src/github.com/cloudfoundry/bosh-init/internal/github.com/cloudfoundry/bosh-utils/logger/logger.go:146 (0x9a57a)
/usr/local/go/src/runtime/asm_amd64.s:437 (0x5cd1e)
/usr/local/go/src/runtime/panic.go:423 (0x2dd09)
/usr/local/go/src/runtime/panic.go:42 (0x2c3c9)
/usr/local/go/src/runtime/sigpanic_unix.go:24 (0x42dc9)
/tmp/build/72b64e5e-29db-42e4-5635-09c9cadd52b5/gopath/src/github.com/cloudfoundry/bosh-init/cmd/deployment_preparer.go:169 (0x90786)
/tmp/build/72b64e5e-29db-42e4-5635-09c9cadd52b5/gopath/src/github.com/cloudfoundry/bosh-init/ui/stage.go:65 (0xb955f)
/tmp/build/72b64e5e-29db-42e4-5635-09c9cadd52b5/gopath/src/github.com/cloudfoundry/bosh-init/cmd/deployment_preparer.go:182 (0x7a8f2)
/tmp/build/72b64e5e-29db-42e4-5635-09c9cadd52b5/gopath/src/github.com/cloudfoundry/bosh-init/cmd/deploy_cmd.go:74 (0x75ec0)
/tmp/build/72b64e5e-29db-42e4-5635-09c9cadd52b5/gopath/src/github.com/cloudfoundry/bosh-init/cmd/runner.go:27 (0x8d60b)
/tmp/build/72b64e5e-29db-42e4-5635-09c9cadd52b5/gopath/src/github.com/cloudfoundry/bosh-init/main.go:46 (0x284b)
/usr/local/go/src/runtime/proc.go:111 (0x301f0)
/usr/local/go/src/runtime/asm_amd64.s:1696 (0x5f041)

********************

[voelzmo]

@cppforlife any updates on this?

[cppforlife]

@voelzmo that's weird. we do that ourselves in our tests. can you show me stemcell.MF and release.MF?

[voelzmo]

Just to clarify, here is what I do:

• export a release for BOSH on some other director
• on another machine, write a deployment manifest, that
  • uses the same Stemcell version referencing a URL on bosh.io
  • has an incorrect SHA1 in the manifest for the stemcell (because I have no way to obtain it on this machine)
• then I do bosh-init deploy with that manifest

I would expect bosh-init to tell me what the correct SHA1 should be to put in the manifest. That's what you do in your tests? Or do you use an already downloaded stemcell from the filesystem?

Anyways, here is my release.MF

---
compiled_packages:
- name: director
  version: f40bfeb75a1aca2e4d0b89a2ed02d3f78bf2c19a
  fingerprint: f40bfeb75a1aca2e4d0b89a2ed02d3f78bf2c19a
  sha1: 70ddb007e4ee01692f54baf3ee51562421eb8e5c
  stemcell: ubuntu-trusty/3215
  dependencies:
  - libpq
  - mysql
  - ruby
  - s3cli
- name: genisoimage
  version: 008d332ba1471bccf9d9aeb64c258fdd4bf76201
  fingerprint: 008d332ba1471bccf9d9aeb64c258fdd4bf76201
  sha1: 417b8dfcc0e6d8460f89263983f19617a8317a77
  stemcell: ubuntu-trusty/3215
  dependencies: []
- name: health_monitor
  version: c305dc0bd463cc43b69f4365b786b2ba91173108
  fingerprint: c305dc0bd463cc43b69f4365b786b2ba91173108
  sha1: 8cf65bd65a777ac0e4f2c4b9ab65baaf262a6340
  stemcell: ubuntu-trusty/3215
  dependencies:
  - ruby
- name: libpq
  version: 09c8f60b87c9bd41b37b0f62159c9d77163f52b8
  fingerprint: 09c8f60b87c9bd41b37b0f62159c9d77163f52b8
  sha1: a5c433ff18e82840ca6c14fda116c71cded87d03
  stemcell: ubuntu-trusty/3215
  dependencies: []
- name: mysql
  version: b7e73acc0bfe05f1c6cbfd97bf92d39b0d3155d5
  fingerprint: b7e73acc0bfe05f1c6cbfd97bf92d39b0d3155d5
  sha1: 23ad35455bed1a5afa8a431d903801fe60d22088
  stemcell: ubuntu-trusty/3215
  dependencies: []
- name: nats
  version: 6a31c7bb0d5ffa2a9f43c7fd7193193438e20e92
  fingerprint: 6a31c7bb0d5ffa2a9f43c7fd7193193438e20e92
  sha1: fcfd7b3d5f9ec00d1bd1357bfaa541625fe707d5
  stemcell: ubuntu-trusty/3215
  dependencies:
  - ruby
- name: nginx
  version: 1d356bbd17ed8c349fd1053093222d78559687ec
  fingerprint: 1d356bbd17ed8c349fd1053093222d78559687ec
  sha1: 8104f109f912ae7adca1943f2a5f9efd8dce0ca3
  stemcell: ubuntu-trusty/3215
  dependencies: []
- name: postgres
  version: 55dbf1828bbb8e5fcd7dc7550cfb73de89312458
  fingerprint: 55dbf1828bbb8e5fcd7dc7550cfb73de89312458
  sha1: bac0fe32bbda32523981f19d9d3040c208f974fa
  stemcell: ubuntu-trusty/3215
  dependencies: []
- name: postgres-9.4
  version: 83e114e2fc524e9a7dfc27866f5d81e7c588642f
  fingerprint: 83e114e2fc524e9a7dfc27866f5d81e7c588642f
  sha1: 352a0e6fa4cc55926d54014c90592663e98a9cfe
  stemcell: ubuntu-trusty/3215
  dependencies: []
- name: powerdns
  version: 256336d00b1689138490c385c03ad3a8f54b4a9e
  fingerprint: 256336d00b1689138490c385c03ad3a8f54b4a9e
  sha1: e7b9657fb422f516d61ebe6e934d4a0e11bc4f11
  stemcell: ubuntu-trusty/3215
  dependencies: []
- name: registry
  version: e3a7ff24076d57208dc6ced1b64db057d6e36cc0
  fingerprint: e3a7ff24076d57208dc6ced1b64db057d6e36cc0
  sha1: c239e59109c29a9cf0b9c6eceb7029eefee64d90
  stemcell: ubuntu-trusty/3215
  dependencies:
  - libpq
  - mysql
  - ruby
- name: ruby
  version: 030e3d4eafd00b10bbfa879bcf44fba62139205e
  fingerprint: 030e3d4eafd00b10bbfa879bcf44fba62139205e
  sha1: dd57b4df07337c6bdf237f21f90b82cd82791ae6
  stemcell: ubuntu-trusty/3215
  dependencies: []
- name: s3cli
  version: 939372842fa951cdc81440b62a8a325b561751f7
  fingerprint: 939372842fa951cdc81440b62a8a325b561751f7
  sha1: 3fa1c3bc5a7c616a6a2d992f87b3cf47b19ea508
  stemcell: ubuntu-trusty/3215
  dependencies: []
jobs:
- name: nats
  version: 329a9df8c4b7d3655c57e59d8fe2a03d65843cac
  fingerprint: 329a9df8c4b7d3655c57e59d8fe2a03d65843cac
  sha1: 73de8659da15387295b879d8190c2d6d79037ac8
- name: blobstore
  version: 076716f65051569ad5f3d45a80f233c5c8e5bcb0
  fingerprint: 076716f65051569ad5f3d45a80f233c5c8e5bcb0
  sha1: d436a508ceb731265db339b67bb1c4b95fe54791
- name: director
  version: 108174c3d28e668f5150d1ee9ee0ccf38a674441
  fingerprint: 108174c3d28e668f5150d1ee9ee0ccf38a674441
  sha1: 0d01b1f187eb777f0170472e7312b743f1854254
- name: health_monitor
  version: cfe20dfb6bd6daab58c55b38493c45120a557329
  fingerprint: cfe20dfb6bd6daab58c55b38493c45120a557329
  sha1: 7cd37a3fd5e6d80dabbc8a588c7a939845950b92
- name: postgres
  version: 9ad09420dd12f830bfc54bb6891fca33e7ad4ff5
  fingerprint: 9ad09420dd12f830bfc54bb6891fca33e7ad4ff5
  sha1: 81ba89331da6c19e3847eb4baf4bdf2dd813799a
- name: postgres-9.4
  version: 0c57c770101e1ee93da1ba2a32340fac1616e2cb
  fingerprint: 0c57c770101e1ee93da1ba2a32340fac1616e2cb
  sha1: 38d605b1be32860e8cceddfafd4a5bc356e93043
- name: powerdns
  version: df20bcfaa20062ef2e0136f6b6a5dbdc62f7b6a8
  fingerprint: df20bcfaa20062ef2e0136f6b6a5dbdc62f7b6a8
  sha1: 6758d5bd60c6f21a852a42fac9ed0bd52915b748
- name: registry
  version: cb87e2ee9c510212a3c28b106206f3b444980236
  fingerprint: cb87e2ee9c510212a3c28b106206f3b444980236
  sha1: 3401a86554db5e4c3ecc679ed7ef6b5171d30964
commit_hash: dff4913a
uncommitted_changes: true
name: bosh
version: 255.8+dev.1

The stemcell.MF is probably coming directly from the archive on bosh.io, so I guess there shouldn't be a problem with this?