The Diego team uses GitHub and accepts contributions via pull request.
The diego-release repository is a BOSH release for Diego. The root of this repository doubles as a Golang GOPATH. For more information about configuring your Golang environment and automatically setting your GOPATH to the release directory, see the instructions below.
All Diego components are submodules in diego-release and can be found in the src/github.com/cloudfoundry-incubator and src/github.com/cloudfoundry directories of this repository.
If you wish to make a change to an individual Diego component, submit a pull request to the master branches of its repository. Once accepted, those changes should make their way into diego-release.
If you wish to make a change to diego-release directly, please base your pull request on the develop branch, and not the master branch. The master branch of diego-release is reserved for the latest final BOSH release of Diego, and the only updates to that branch should be through our automated release-creation process.
To verify your changes before submitting a pull request, run unit tests, the inigo test suite, and the CF Acceptance Tests (CATs). See the testing section for more detail.
Follow these steps to make a contribution to any of our open source repositories:
-
Ensure that you have completed our CLA Agreement for individuals or corporations.
-
Set your name and email (these should match the information on your submitted CLA)
git config --global user.name "Firstname Lastname"
git config --global user.email "your_email@example.com"
- All contributions must be sent using GitHub pull requests as they create a nice audit trail and structured approach.
The originating github user has to either have a github id on-file with the list of approved users that have signed the CLA or they can be a public "member" of a GitHub organization for a group that has signed the corporate CLA. This enables the corporations to manage their users themselves instead of having to tell us when someone joins/leaves an organization. By removing a user from an organization's GitHub account, their new contributions are no longer approved because they are no longer covered under a CLA.
If a contribution is deemed to be covered by an existing CLA, then it is analyzed for engineering quality and product fit before merging it.
If a contribution is not covered by the CLA, then the automated CLA system notifies the submitter politely that we cannot identify their CLA and ask them to sign either an individual or corporate CLA. This happens automatially as a comment on pull requests.
When the project receives a new CLA, it is recorded in the project records, the CLA is added to the database for the automated system uses, then we manually make the Pull Request as having a CLA on-file.
##Initial Setup
This BOSH release doubles as a $GOPATH. It will automatically be set up for you if you have direnv installed.
NOTE: diego-release and its components assume you're running go 1.6. The project may not compile or work as expected with other versions of go.
# create parent directory of cf-release and diego-release
mkdir -p ~/workspace
cd ~/workspace
# clone cf-release
git clone https://github.com/cloudfoundry/cf-release.git
pushd cf-release/
# if you're making changes to diego-release,
git checkout release-candidate
## or, if you're making changes to cf-release,
# git checkout develop
./scripts/update
popd
# clone garden-linux-release
git clone https://github.com/cloudfoundry-incubator/garden-linux-release.git
pushd garden-linux-release
git checkout master && git submodule update --init --recursive
popd
# clone diego-release
git clone https://github.com/cloudfoundry-incubator/diego-release.git
pushd diego-release/
# automate $GOPATH and $PATH setup
direnv allow
# switch to develop branch to make changes to diego-release,
git checkout develop
## or, if you're only making changes to cf-release,
# git checkout master
# initialize and sync submodules
./scripts/update
popd
If you do not wish to use direnv, you can simply source the .envrc file in the root of the release repo. You may manually need to update your $GOPATH and $PATH variables as you switch in and out of the directory.
To be able to run unit tests, you'll also need to install the following binaries:
# Install ginkgo
go install github.com/onsi/ginkgo/ginkgo
# Install gnatsd
go install github.com/apcera/gnatsd
# Install etcd
go install github.com/coreos/etcd
# Install consul
if uname -a | grep Darwin; then os=darwin; else os=linux; fi
curl -L -o $TMPDIR/consul-0.5.2.zip "https://releases.hashicorp.com/consul/0.5.2/consul_0.5.2_${os}_amd64.zip"
unzip $TMPDIR/consul-0.5.2.zip -d $GOPATH/bin
rm $TMPDIR/consul-0.5.2.zip
To be able to run the integration test suite ("inigo"), you'll need to have a local Concourse VM. Follow the instructions on the Concourse README to set it up locally using vagrant. Download the fly CLI as instructed and move it somewhere visible to your $PATH.
To run the experimental SQL unit tests, you'll need mysql running locally with the correct sql_mode and user configuration.
On OS X, you can follow the these steps to install and configure mysql:
-
brew install mysql -
mysql.server start -
Run
mysql_secure_installationand set a root password- Follow the on-screen prompts to complete the installation. The answers won't affect whether the unit tests can run.
-
Create /etc/my.cnf with the following contents:
[mysqld]
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
```
5. mysql.server restart
6. Log in to the mysql console as root, using the password you specified in step 3
```
mysql -uroot -p ``` 7. Run the following SQL commands to create a diego user with the correct permissions:
```
CREATE USER 'diego'@'localhost' IDENTIFIED BY 'diego_password';
GRANT ALL PRIVILEGES ON diego\_%.* TO 'diego'@'localhost';
```
8. You should now be able to run the SQL unit tests. To run the tests, run the following command from the bbs submodule:
```
ginkgo -p db/sqldb ```
-
Download the latest Warden Trusty Go-Agent stemcell and upload it to BOSH-lite:
bosh public stemcells bosh download public stemcell (name) bosh upload stemcell (downloaded filename) -
Check out cf-release (runtime-passed branch) from git:
cd ~/workspace git clone https://github.com/cloudfoundry/cf-release.git cd ~/workspace/cf-release git checkout runtime-passed ./scripts/update -
Check out diego-release (develop branch) from git:
cd ~/workspace git clone https://github.com/cloudfoundry-incubator/diego-release.git cd ~/workspace/diego-release git checkout develop ./scripts/update -
Install
spiffaccording to its README.spiffis a tool for generating BOSH manifests that is required in some of the scripts used below. -
Generate the CF manifest:
cd ~/workspace/cf-release ./scripts/generate-bosh-lite-dev-manifestOr if you are running Windows cells along side this deployment, instead generate the CF manifest as follows:
cd ~/workspace/cf-release ./scripts/generate-bosh-lite-dev-manifest \ ~/workspace/diego-release/manifest-generation/stubs-for-cf-release/enable_diego_windows_in_cc.yml -
Generate the Diego manifests:
cd ~/workspace/diego-release ./scripts/generate-bosh-lite-manifests -
Create, upload, and deploy the CF release:
cd ~/workspace/cf-release bosh deployment bosh-lite/deployments/cf.yml bosh create release --force && bosh -n upload release && bosh -n deploy -
Upload the latest garden-linux-release:
bosh upload release https://bosh.io/d/github.com/cloudfoundry-incubator/garden-linux-release -
Upload the latest etcd-release:
bosh upload release https://bosh.io/d/github.com/cloudfoundry-incubator/etcd-release -
Create, upload, and deploy the Diego release:
cd ~/workspace/diego-release bosh deployment bosh-lite/deployments/diego.yml bosh create release --force && bosh -n upload release && bosh -n deploy -
Login to CF and enable Docker support:
cf login -a api.bosh-lite.com -u admin -p admin --skip-ssl-validation && cf enable-feature-flag diego_docker
Now you are configured to push an app to the BOSH-Lite deployment, or to run the CF Smoke Tests or the CF Acceptance Tests.
When working on individual components of Diego, work out of the submodules under src/.
Run the individual component unit tests as you work on them using ginkgo. To see if everything still works, run ./scripts/run-unit-tests in the root of the release.
When you're ready to commit, run:
./scripts/prepare-to-diego <story-id> <another-story-id>...
This will synchronize submodules, update the BOSH package specs, run all unit tests, all integration tests, and make a commit, bringing up a commit edit dialogue. The story IDs correspond to stories in our Pivotal Tracker backlog. You should simultaneously also build the release and deploy it to a local BOSH-Lite environment, and run the acceptance tests. See Running Smoke Tests & CATs.
If you're introducing a new component (e.g. a new job/errand) or changing the main path for an existing component, make sure to update ./scripts/sync-package-specs and ./scripts/sync-submodule-config.
Please follow logging conventions as outlined here.
Once you've followed the steps above to install ginkgo and the other binaries needed for testing, execute the following script to run all unit tests in diego-release.
./scripts/run-unit-tests
If your local concourse VM is up and running, you have the fly CLI visible on your $PATH, and you've cloned garden-linux-release (see Initial Setup for details), you can run
./scripts/run-inigo
from the root of diego-release to run the integration tests.
### Running Smoke Tests, and CATs
You can test that your diego-release deployment is working and integrating with cf-release by running the lightweight cf-smoke-tests or the more thorough cf-acceptance-tests. These test suites assume you have a BOSH environment to deploy cf and diego to. For local development, bosh-lite is an easy way to have a single-VM deployment. To deploy diego to bosh-lite, follow the instructions on deploying diego to bosh-lite.
The instructions below assume you're using bosh-lite and have generated the
manifests with the scripts/generate-bosh-lite-manifests script. This script
will also generate manifests for the errands that run these test suites. If you
did not run that script or are running tests in a different environment,
substitute the relevant manifest files in the bosh deploy commands below.
To run the cf-acceptance-tests against a Diego deployed to bosh-lite, run:
./scripts/run-cats-bosh-lite
To run cf-smoke-tests you can similarly deploy and run an errand to run the tests:
# target the errand for smoke tests when running them
bosh -n -d bosh-lite/deployments/cf.yml run errand smoke_tests
Running the benchmark tests isn't usually needed for most changes. However, for changes to the BBS or the protobuf models, it may be helpful to run these tests to understand the performance impact.
WARNING: Benchmark tests drop the database.
-
Deploy diego-release to an environment (use instance-count-overrides to turn off all components except the database for a cleaner test)
-
Depending on whether you're deploying to AWS or bosh-lite, copy either
manifest-generation/benchmark-errand-stubs/default_aws_benchmark_properties.ymlormanifest-generation/benchmark-errand-stubs/default_bosh_lite_benchmark_properties.ymlto your local deployments or stubs folder and fill it in. -
Generate a benchmark deployment manifest using:
./scripts/generate-benchmarks-manifest \ /path/to/diego.yml \ /path/to/benchmark-properties.yml \ > benchmark.yml -
Deploy and run the tests using:
bosh -d benchmark.yml -n deploy && bosh -d benchmark.yml -n run errand benchmark-bbs