Setting up the Development Environment

Purpose

The purpose of this exercise is to enable a developer new to the project to get a local environment fully set up on their machine and ready to start contributing to the repository.

1. Repository and Version Control

Ensure you have push access (most likely by being added to the @bcgov/cas-developers GitHub team) to bcgov/cas-ciip-portal.
- If you will be making pull requests to the repository, you'll also need to be added to the @bcgov/cas GitHub team for the CircleCI security context needed to run some jobs which need access to our OpenShift cluster.
Ensure you have GPG commit signing set up on your local environment.
- First, ensure your git config user.email is set to the email address you want to use for signing.
- You can verify it's working when you commit to a branch and the signature is indicated by git log --show-signature. Once pushed, a "Verified" badge appears next to your commits on GitHub.
Clone a local copy of bcgov/cas-ciip-portal.
This project uses Git submodules to incorporate other CAS-maintained repositories. Configuring a post-checkout Git hook in your local repository can help smooth the experience of using submodules.
From the project root, run git submodule update --init to initialize the submodules and checkout the appropriate state based on the specifications in .gitmodules.

2. Package Management

asdf is a universal package manager used in this project to install and manage versions of the tools listed in .tool-versions. To do this, several asdf plugins will be installed as well.

Install asdf using your system's package manager & follow instructions to add it to your shell.
From the project root, run make install_asdf_tools, which installs the additional asdf plugins needed to manage the various tools in .tool-versions.
- Later versions of gpg might run into this issue
```
gpg: keyserver receive failed: Network is unreachable
gpg: keyserver receive failed: No keyserver available
```
- this script also imports the Node.js release team's OpenPGP keys for checksum verification.
- the script also attempts to install Postgres --with-libxml via asdf to include a database plugin for XML. Your system will need libxml2 installed for this.
- lastly, it installs pip dependencies that relate to pre-commit, which helps us run linters to keep things tidy.
- troubleshooting: make install_asdf_tools --dry-run can be helpful to see what is being run.
You may need to troubleshoot individual steps in make install_asdf_tools on your system; in that case, find this target in the root Makefile and run the steps individually.
asdf reshim should be run after asdf installations to update symlinks for the installed packages. Open a new terminal to use the shims.
Set the version of Postgres installed by asdf as the global version, necessary to prevent later problems installing Sqitch, our database migration tool.
psql --version should verify the installed version of Postgres.
you can verify pre-commit works by running: pre-commit run --all-files from the project root.
Yarn is used as the Javascript package manager, and was installed in this step by asdf from the .tool-versions. All yarn commands must be run from the same directory as the package.json (within app/).

If you're on Linux, you can now skip to Step 4.

2.5 MacOS-Specific Troubleshooting

a. SDK root

If your MacOS command line tools were installed before upgrading to Catalina and haven't already been fixed afterward, you will need to do so now. Outstanding problems installing Postgres due to errors in compiling C libraries in the previous step could indicate this. Read more about the problem here.

The full version of XCode (> 20GB) downloaded from the App Store reportedly works as intended, but if you prefer to use the smaller subset of command line tools installed via xcode-select --install (or manually downloaded and installed from the Apple Developer support page), then you can set the SDKROOT environment variable in your ~/.bash_profile:

export SDKROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/"

...making sure to replace the above path with the actual location and version of your command line tools. Note you will need to manually update this path / version number whenever the command line tools are updated.

source ~/.bash_profile and re-attempt the Postgres installation above.

b. sysroot path

Postgres often needs PG_SYSROOT to be specified.
See "macOS" section: https://www.postgresql.org/docs/current/installation-platform-notes.html#INSTALLATION-NOTES-MACOS

Something like:

export PG_SYSROOT="/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk"

Then retry installing postgres:

POSTGRES_EXTRA_CONFIGURE_OPTIONS='--with-libxml' asdf install postgres 11.4

c. System Integrity Protection on macOS

posgresql.org also advises to turn off SIP on macOS if this is a possibility, since it prevents passing library linking targets around.

d. postgres make order

If all else fails, running make install before make check in the postgres compilation should fix the SIP incompatibility. To make this work with asdf:

Find out where postgres is being downloaded by asdf (something like /var/folders/6w/8rt8tgzd5p9bk8kb98hpmwqmvj3393/T/postgresql-11.4.tar.gz)
Alternatively, change the download path in the asdf plugin install script ~/.asdf/plugins/postgres/bin/install
Untar the file gunzip -c postgresql-11.4.tar.gz | tar xopf -
Edit the makefile to execute install befoore check
- open postgresql-11.4/Makefile
- edit the following line:
  - all check install installdirs installcheck installcheck-parallel uninstall ...
    becomes
  - all install check installdirs installcheck installcheck-parallel uninstall ...
Rezip the archive tar -zcvf postgresql-11.4.tar.gz postgresql-11.4
Finally, compile postgres with asdf:

POSTGRES_EXTRA_CONFIGURE_OPTIONS='--with-libxml' asdf install postgres 11.4

3. Configure Postgres on MacOS

This step is only necessary for MacOS users to set up their local Postgres database (installed by asdf in the last step) to reflect the system defaults on Linux (and by extension, on the server).

Start / stop a Postgres service with pg_ctl start and pg_ctl stop
Following the Postgres docs:

A convenient choice is to create a database with the same name as your current user name*. Many tools assume that database name as the default [database], so it can save you some typing. To create that database, simply type: createdb
* On BCGov computers, this username is likely your IDIR in lowercase (verify with echo $(whoami)).
Verify the database was created using psql --list.
A Postgres user (aka. role) named postgres must be created manually on MacOS, but is created automatically for new installations on Linux:
- Enter the psql shell with: psql to connect to the default database, then list all existing roles using \du. There should be a default role named after your sytem username.
- Enter a SQL statement creating a new role named postgres with the same permissions as the above role:
```
create role postgres with login superuser createrole createdb replication bypassrls;
```
- List all roles again to ensure postgres was created with the appropriate permissions.

4. Set up a Working Perl 5 Environment

Although Perl is not specifically used in this project, our database migration tool Sqitch requires a working Perl 5 environment. Unfortunately, there is no asdf plugin for Perl, so we must manage this ourselves.

NOTE: As the Perl ecosystem hasn't kept pace with the conveniences of modern times, for most people this tends to be the most problematic step.

Use your system package manager to install Perl, ensure the version is >= 5, and add this perl to your $PATH.
- Alternatively, you can try using Perlbrew to install, which helps to manage multiple local Perl versions.
- If using MacOS: Unfortunately, the pre-installed system Perl cannot be relied upon; certain critical headers are missing. Use Homebrew to re-install Perl 5 and ensure that is the perl in your $PATH.
  - Use Perlbrew if you might switch between multiple Perl versions, or: brew update && brew install perl
  - it must be installed somewhere with regular user permissions such as /usr/local/Cellar/ - not somewhere like /Library/ that requires unsafe root permissions to install further Perl modules.
  - Homebrew should install perl in the appropriate location by default.
From the project root, run make install_perl_tools. This invokes the install target in schema/Makefile.
- You may again need to troubleshoot individual steps in make install_perl_tools on your system; in that case, find the install target in schema/Makefile and run the steps individually.
- This script installs the Perl package manager cpanm (aka. cpanminus) and then invokes it to install Perl dependencies from the schema/cpanfile. Namely, these dependencies are:
  - the Postgres database driver for Perl, DBD::Pg
  - the database migration tool, Sqitch
  - the database unit test runner, pgTAP
- By default, cpanm runs each package's tests after installation, which can be quite time-consuming. For this reason, the make install_perl_tools script uses cpanm --notest to skip tests while installing.
- If successful, the post_install_check target that is run as part of make install_perl_tools should output the installed version of Sqitch.
More manual MacOS Perl option with Perlbrew

Perlbrew
```
perlbrew init

# Find the perl you want
perlbrew available

perlbrew install 5.35.0
perlbrew switch 5.35.0


make sure you're using the perlbrew version you expect

ex: `which perl`
/Users/naomiaro/perl5/perlbrew/perls/perl-5.35.0/bin/perl


# Install CPAN modules
perlbrew install-cpanm

# From the project root
cd schema
cpanm --installdeps .
```
Troubleshooting:
- Ensure the version of Postgres installed by asdf in Step 2 was set as the global version using asdf global before installing Sqitch; otherwise, Sqitch may install a separate instance of Postgres.
- Sqitch can alternatively be installed on MacOS with brew tap sqitchers/sqitch && brew install sqitch.
- If you're on MacOS and having trouble with Perl libraries ignoring your newly installed (and thus, properly functioning) Perl in favour of an older system Perl, you can use the $PERL5LIB environment variable to point it to the desired location. For example, point it to the new Perl you installed with non-root / regular user permissions instead of the root-installed system Perl.
- local::lib may need some help finding its local configuration and paths, these can be set through environment variables. More information in that documentation.
```
 ~/.bashrc:
     export PERL5LIB=~/lib/perl5/lib/perl5/

     PERL_LOCAL_LIB_ROOT="/Users/$(whoami)/lib/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}"; export PERL_LOCAL_LIB_ROOT;
     PERL_MB_OPT="--install_base \"/Users/$(whoami)/lib/perl5\""; export PERL_MB_OPT;
     PERL_MM_OPT="INSTALL_BASE=/Users/$(whoami)/lib/perl5"; export PERL_MM_OPT;
```

5. Install pgTAP

Ensure the postgres service is running with pg_ctl start.

cd schema && make install_pgtap

This clones and builds pgTAP, the database unit testing library from source. The cloned repository will be left behind (visible with git status), which you can safely remove after the installation is complete.

6. Deploy the Development Database

createdb ciip_portal_dev ciip_portal_test

psql --list to verify the databases were created.

Rather than interacting with Sqitch directly, we use a custom script for convenience to deploy the database, including seed data for dev, test and production.

From the project root, run .bin/deploy-data.sh --help to see possible options. In development, this is most commonly run with .bin/deploy-data.sh -d -dev to drop the current database and redeploy after pulling down database-changing work from the remote or tampering with the database.

7. Running the Web Application

App Environment Variables

Create an app/.env file and copy the contents of app/.env.example into it. Ask a teammate for example values.

(optional) SMTP Server

It's not necessary for every development task, but in order to run the full suite of end-to-end tests or to view or develop transactional emails triggered by the application, a local SMTP server is required. This is most quickly done using Docker to run MailHog.

Ensure Docker is installed and running, then:

sudo docker run -d -p 1025:1025 -p 8025:8025 mailhog/mailhog

This uses SMTP port 1025 to start a MailHog server on localhost:8025 where you can view emails. It can be stopped by using docker ps to get the container ID and docker stop <id>.

Install Javascript Dependencies

Install the node_modules including devDependencies from the package.json:

cd app && yarn --dev

Run the Server

Run the development server with yarn dev and navigate in your browser to localhost:3004.

Work that changes Relay GraphQL queries will require you to re-run yarn build:relay (which alternatively can be run using a --watch flag).

GraphQL Query Debugger

Graphiql is an interactive in-browser GraphQL IDE available in development at localhost:3004/graphiql. It is handy for developing queries interactively before inserting them into a Relay fragment.

7.5 Bypass Authentication in Development

There are some shortcuts to bypass authentication in development and run the server while logged in as a preset user:

cd app && yarn dev <AS_USER>

...where <AS_USER> is one of the following:

AS_REPORTER
AS_ANALYST
AS_ADMIN
AS_PENDING

Learn more here about the functionality available to each user type.

8. Run Tests

Unit Tests with Jest

cd app && yarn test

Front-end unit tests are heavily snapshot-based. Work that changes the DOM will result in a diff from the last accepted snapshot and cause related tests to fail. You can update the snapshots and review / accept the diff with yarn test -u.

Database Unit Tests with pgTAP

cd schema && make unit

If pg_prove can't be found and you installed perl via Homebrew, make sure the perl libraries are on your path.

find /usr/local -name pg_prove

export PATH="/usr/local/Cellar/perl/5.32.1_1/bin:$PATH"

End-to-end Tests with Cypress

Cypress Environment

Ask a teammate for the app/cypress.env.json, which specifies some preset usernames and passwords required by the tests.

Test Email Server

A local SMTP server is a precondition for running the email specs (for example, if running the entire suite), for which you can use Docker to run MailHog.

Run Cypress Specs

Cypress needs the test schema deployed for the specs to run:

cd test_helper_schema && SQITCH_TARGET="ciip_portal_dev" sqitch deploy

First, ensure the web app is running (cd app && yarn dev) without bypassing authentication using AS_REPORTER or similar auth flags. Running the app with the following flags will prevent certain asynchronously rendered components from causing spurious failures:

cd app

yarn dev AS_CYPRESS

For test error debugging and to observe tests' behavior in the browser as they run:

cd app && yarn cypress

To run the tests more efficiently in a headless mode:

cd app && yarn test:e2e

Options can be passed to Cypress through this command, for example to run an individual test or subset:

cd app && yarn test:e2e --spec cypress/integration/accessibility/*

9. Tidy Your Work

pre-commit runs a variety of formatting and lint checks configured in .pre-commit-config.yaml which are required for a pull request to pass CircleCI.

pre-commit install will configure a pre-commit hook to run before every commit; alternatively, you can run it manually with:

pre-commit run --all-files

If you are impatient and your work is isolated to Javascript, it may be faster to run only the linter and formatter (eslint and prettier), but it may not catch everything (such as the end-of-file fixer and trailing whitespace):

yarn lint && yarn format

10. Commit Something

Commit Message Conventions

We use gitlint to check commit message formatting. You can enable it by using pre-commit install --hook-type commit-msg.

This project follows the commit message conventions outlined by Convential Commits. Besides the standard commit types (message prefixes) feat and fix, we use some other types described there based on the Angular convention; some common ones among those are test, docs, chore and refactor. You can find the configuration details in the .gitlint file

These facilitate the automated creation of changelogs, using the standard-version utility.

We also extend this prefix convention to the naming of branches, eg: docs/add-readme or feat/some-feature.

Victory Lap

Using the above conventions, make a branch with your name called feat/my-name-onboarding and add a high five somewhere in the README:

🙌

Make a draft pull request against the default branch, and request another developer's review. This will not be merged, but pushing to GitHub and running CircleCI steps will help verify you have all the correct permissions and signed commits.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dev-environment-setup.md

dev-environment-setup.md

Setting up the Development Environment

Purpose

1. Repository and Version Control

2. Package Management

2.5 MacOS-Specific Troubleshooting

a. SDK root

b. sysroot path

c. System Integrity Protection on macOS

d. postgres make order

3. Configure Postgres on MacOS

4. Set up a Working Perl 5 Environment

More manual MacOS Perl option with Perlbrew

5. Install pgTAP

6. Deploy the Development Database

7. Running the Web Application

App Environment Variables

(optional) SMTP Server

Install Javascript Dependencies

Run the Server

GraphQL Query Debugger

7.5 Bypass Authentication in Development

8. Run Tests

Unit Tests with Jest

Database Unit Tests with pgTAP

End-to-end Tests with Cypress

Cypress Environment

Test Email Server

Run Cypress Specs

9. Tidy Your Work

10. Commit Something

Commit Message Conventions

Victory Lap

Files

dev-environment-setup.md

Latest commit

History

dev-environment-setup.md

File metadata and controls

Setting up the Development Environment

Purpose

1. Repository and Version Control

2. Package Management

2.5 MacOS-Specific Troubleshooting

a. SDK root

b. sysroot path

c. System Integrity Protection on macOS

d. postgres make order

3. Configure Postgres on MacOS

4. Set up a Working Perl 5 Environment

More manual MacOS Perl option with Perlbrew

5. Install pgTAP

6. Deploy the Development Database

7. Running the Web Application

App Environment Variables

(optional) SMTP Server

Install Javascript Dependencies

Run the Server

GraphQL Query Debugger

7.5 Bypass Authentication in Development

8. Run Tests

Unit Tests with Jest

Database Unit Tests with pgTAP

End-to-end Tests with Cypress

Cypress Environment

Test Email Server

Run Cypress Specs

9. Tidy Your Work

10. Commit Something

Commit Message Conventions

Victory Lap