Add collectd to bosh

Add 0.4 release of our paas-collectd-boshrelease to BOSH. Send metrics
to shared deployment graphite server.

The initial reason why we used fork was that we could add custom tags and
pin our deployment to these tags.
Now, because bosh-init can only consume tarballs (you can't upload a release
to bosh-init like you can to BOSH), I had to create a dev-release tarbal
(using standard bosh-cli way bosh create release --with-tarball --force
--version 0.4, more documentation about bosh releases:
https://bosh.io/docs/create-release.html#dev-release-release). I have
created the tarballs from the same code as is tagged. That is, 0.4 release
is created from code that's tagged as 0.4.

It seemed to me that a good place to put the release tarball was in github
release. This practice is also used in some CF repositories (e.g. diego). We
have a spike to analyse our situation with releases completely, but we did
not want to block this story on it (#115142265 - see comments in the story).
In the future, we can easily move the release location to elsewhere and we
can change the way to build the releases.

Because bosh-init currently uses template name as job name[1], it renders
collectd config to say BOSH host is called `collectd`. Add a prefix that
explicitly contains `bosh_` so that we can see which host are metrics
coming from.

[1] cloudfoundry-attic/bosh-init#87

concourse/pipelines/create-bosh-cloudfoundry.yml

@@ -754,6 +754,7 @@ jobs:
               ./bosh-CA-yml/bosh-ca-cert.yml
               ./terraform-outputs/bosh.terraform-outputs.yml
               ./terraform-outputs/vpc.terraform-outputs.yml
+              ./paas-cf/manifests/shared/deployments/collectd.yml
           run:
             path: sh
             args:

manifests/bosh-manifest/deployments/040-collectd.yml

@@ -0,0 +1,16 @@
+---
+releases:
+- name: collectd
+  sha1: 89d9afbb4e682f31e745f07bd40ff569d347343f
+  url: https://github.com/alphagov/paas-collectd-boshrelease/releases/download/0.4/collectd-0.4.tgz
+
+jobs:
+- name: bosh
+  templates:
+  - {name: collectd, release: collectd}
+
+properties:
+  collectd:
+    hostname_prefix: bosh_
+    interval: (( grab meta.collectd.interval ))
+    config: (( grab meta.collectd.config ))

manifests/bosh-manifest/reference-bosh-manifest.yml

@@ -131,6 +131,8 @@ jobs:
     release: bosh
   - name: health_monitor
     release: bosh
+  - name: collectd
+    release: collectd
   - name: registry
     release: bosh
   - name: aws_cpi
@@ -151,6 +153,51 @@ networks:
 - name: public
   type: vip
 properties:
+  collectd:
+    config: |
+      LoadPlugin cpu
+      LoadPlugin disk
+      LoadPlugin entropy
+      LoadPlugin interface
+      LoadPlugin load
+      LoadPlugin memory
+      LoadPlugin swap
+      LoadPlugin uptime
+
+      LoadPlugin df
+      <Plugin df>
+        ReportInodes true
+        ReportReserved true
+      </Plugin>
+
+      LoadPlugin syslog
+      <Plugin syslog>
+        LogLevel warning
+        NotifyLevel WARNING
+      </Plugin>
+
+      LoadPlugin vmem
+      <Plugin vmem>
+        Verbose false
+      </Plugin>
+
+      LoadPlugin "write_graphite"
+      <Plugin "write_graphite">
+       <Node "myNode">
+         #FIXME: hard coded static IP to be removed during #121602315
+         # This address comes from 040-graphite.yml:31 and has been copied
+         # as `grab` cannot interpolate in a multi-line string.
+         Host "10.0.16.20"
+         Port "2003"
+         Prefix "collectd."
+         EscapeCharacter "_"
+         SeparateInstances true
+         StoreRates false
+         AlwaysAppendDS false
+       </Node>
+      </Plugin>
+    hostname_prefix: bosh_
+    interval: 10
   director:
     trusted_certs: |
       -----BEGIN RSA PRIVATE KEY-----
@@ -160,10 +207,14 @@ properties:
       0000000000000000000000000000000000000000000000000000000000000000
       0000000000000000000000000000000000000000000000000000000000000000
       -----END RSA PRIVATE KEY-----
+
 releases:
 - name: bosh
   sha1: 6b12652650b87810dcef1be1f6a6d23f1c0c13a7
   url: https://bosh.io/d/github.com/cloudfoundry/bosh?v=255.8
+- name: collectd
+  sha1: 89d9afbb4e682f31e745f07bd40ff569d347343f
+  url: https://github.com/alphagov/paas-collectd-boshrelease/releases/download/0.4/collectd-0.4.tgz
 - name: bosh-aws-cpi
   sha1: dc4a0cca3b33dce291e4fbeb9e9948b6a7be3324
   url: https://bosh.io/d/github.com/cloudfoundry-incubator/bosh-aws-cpi-release?v=52

manifests/bosh-manifest/spec/shared_config_spec.rb

@@ -0,0 +1,8 @@
+
+RSpec.describe "Bosh collectd properties" do
+  let(:manifest) { manifest_with_defaults }
+
+  it "pulls from a shared config file" do
+    expect(manifest.fetch("properties").fetch("collectd").fetch("interval")).to eq 10
+  end
+end

manifests/bosh-manifest/spec/support/manifest_helpers.rb

@@ -17,7 +17,8 @@ def load_default_manifest
         File.expand_path("../../fixtures/bosh-secrets.yml", __FILE__),
         File.expand_path("../../fixtures/bosh-ssl-certificates.yml", __FILE__),
         File.expand_path("../../fixtures/bosh-terraform-outputs.yml", __FILE__),
-        File.expand_path("../../fixtures/vpc-terraform-outputs.yml", __FILE__)
+        File.expand_path("../../fixtures/vpc-terraform-outputs.yml", __FILE__),
+        File.expand_path("../../../../shared/deployments/collectd.yml", __FILE__)
       ].join(' ')
     )
     expect(status).to be_success, "build_manifest.sh exited #{status.exitstatus}, stderr:\n#{error}"