Merge pull request #151 from Clever/INFRANG-6709

[INFRANG-6709] Finish the README and docs

Author: andruwm

README.md

@@ -1,13 +1,20 @@
 # ci-scripts
 
-Re-usable continuous integration (CI) scripts
-
-Inspired by: https://circleci.com/blog/continuous-integration-at-segment/
+Re-usable continuous integration (CI) scripts and tools.
 
 Owned by `#eng-infra`.
 
+## goci
+
+goci is a go application tool intended for use in CI. It helps with building, publishing, and deploying applications. It is the next evolution of many of the scripts in this repository, and offers more automation, configurability and optimization.
+
+See the [README](./cmd/goci/README.md) for more details
+
 ## Scripts
 
+### Note!!!
+Before adding any new scripts, or changing major functionality, you should strongly consider doing that work in the new [private Clever orbs repo](https://github.com/Clever/circleci-orbs/tree/master) instead. Orbs are more composable, more powerful and more maintainable.
+
 ### General-purpose
 
 The following scripts don't rely on any Clever-specific tooling.
@@ -72,10 +79,3 @@ Publishes a workflow to [workflow-manager](https://github.com/clever/workflow-ma
 ```
 $ ./circleci/workflow-publish [WF_URL] [WF_USER] [WF_PASS] [WF_JSON]
 ```
-
-#### Report-card
-
-Disabled.
-
-We still are keeping the script here so that repos that invoke it can do so safely.
-But it is a no-op.

VERSION

@@ -1,7 +1,8 @@
-v1.0.1
-- Instead of emitting an error on failed catalog syncs, simply warn
+v1.0.2
+variable MASTER_COMPARE is now required for local dev
 
 Previously:
+- Instead of emitting an error on failed catalog syncs, simply warn
 - Begin publishing goci binaries
 - Call catalogue config with goci
 - build with goci

cmd/goci/README.md

@@ -1,40 +1,44 @@
 # goci
 
-`goci` is a small command line application which performs packaging and
-publishing of Clever build artifacts.
+`goci` is a small command line application which performs building, publishing, and deploying applications. It is the next evolution of many of the scripts in this repository, and offers more automation, configurability and optimization.
 
-# Configuration
+## Configuration
 
-goci does not accept any arguments and is instead configured entirely
-through environment variables and launch config settings. See the
-[environment](../../internal/environment/environment.go) package for
-detailed documentation of environment variables for configuration. goci
-reads it's configuration from the `build` section of the launch config
-of each application.
+goci accepts very limited arguments which merely change the mode it runs in. The rest of the configuration is entirely through environment variables and launch config settings. See the[environment](../../internal/environment/environment.go) package for detailed documentation of environment variables for configuration. goci reads it's configuration from the `build` section of the launch config of each application. See the [build section](https://github.com/Clever/catapult/blob/master/swagger.yml#L1773) of the launch yaml to learn about the various parameters which configure goci.
 
-# Multi-app Support
+## Modes
+
+1. `goci detect` detects any changed applications according to their launch configuration. This can be used to pass a name of apps to another script.
+2. `goci artifact-build-publish-deploy` builds, publishes and deploys any application artifacts.
+
+
+## Multi-app Support
 
 goci will automatically detect all launch configs in the `launch`
 directory, then perform the following actions as needed.
 
 1. detect the run type of the application
-2. build any docker images
-3. publish all built docker images to ECFR
-4. publish all pre-built lambdas to s3
+2. Run any configured build commands
+3. build any docker images
+4. publish all built docker images to all ECR regions
+5. publish all lambdas to s3 in all regions.
+6. Sync all changed apps with catalog config
+7. Publish new application versions to catapult
+8. Deploy any changed applications.
+
+## Development
 
-Currently, goci does not build lambdas.
+Since goci is just a go app, it can be run locally after building the binary. goci expects to be run from within the root of an applications repo.
 
-## Running and testing locally
+There is a [test app](./testApp) in this repository which you can use to run goci on locally and test various build configurations. Additionally, you can leverage the built in integrations tests in https://github.com/Clever/circleci-orbs/tree/master, which call goci to test new builds of goci in CI.
 
-Since goci is just a go app, it can be run locally after building the
-binary. goci expects to be run from within the root of an applications
-repo. You may use the `testApp` within this repo for testing various
-launch configurations of goci.
+### Running locally
+Running goci locally is not easy because it leverages a lot of variables set in our CI environments. Generally it would be easier to use the private orbs repo, or make a branch on another app like catapult and pull in the development branch of goci there for testing in CI. If you need to run locally, you can view each of the variables required for local runs in the environment file and provide them all to the command.
 
 ```bash
-$ go build -o ./bin/goci ./cmd/goci
+go build -o ./bin/goci ./cmd/goci
 
-$ cp ./bin/goci ./testApp
+cp ./bin/goci ./testApp
 
-$ cd testApp && goci
+cd testApp && goci
 ```

internal/environment/environment.go

@@ -29,7 +29,7 @@ var (
 	PreviousPipelineCompare = envMustString("PREVIOUS_PIPELINE_COMPARE", false)
 	// PrimaryCompare is the git commit range to run change detection
 	// commands against when running for a non-primary branch.
-	PrimaryCompare = envMustString("MASTER_COMPARE", false)
+	PrimaryCompare = envMustString("MASTER_COMPARE", true)
 	// CatapultURL is the dns of the circle-ci-integrations ALB
 	// including the protocol.
 	CatapultURL = envMustString("CATAPULT_URL", true)