GitHub - ceonelson/forwardemail.net: Privacy-focused encrypted email for everyone.

Forward Email is the 100% open-source and privacy-focused email service @ https://forwardemail.net.

How do I get started

For Consumers

Visit https://forwardemail.net to get started!

For Developers

See Requirements and Local Development Guide below.

Requirements

macOS

Install n and Node v18.20.4:

curl -L https://raw.githubusercontent.com/mklement0/n-install/stable/bin/n-install | bash -s -- -y 18.20.4

Ensure that you are running on Node v18.20.4:
```
node --version
v18.20.4
```

Install brew and the following dependencies using brew command:

brew tap mongodb/brew
brew install mongodb-community redis libtool automake autoconf nasm
brew services start mongodb-community
brew services start redis

Install pnpm:

corepack enable
corepack prepare pnpm@latest --activate

Fork the repository from GitHub

Clone your fork locally (replace forwardemail with your username):

git clone git@github.com:forwardemail/forwardemail.net.git
cd forwardemail.net

Install npm dependencies:
```
pnpm install
```
Install wkhtmltopdf (optional: only used for generating PDF receipts)

Ubuntu

Install n and Node v18.20.4:

curl -L https://raw.githubusercontent.com/mklement0/n-install/stable/bin/n-install | bash -s -- -y 18.20.4

Ensure that you are running on Node v18.20.4:
```
node --version
v18.20.4
```

Install pnpm:

corepack enable
corepack prepare pnpm@latest --activate

Fork the repository from GitHub

Clone your fork locally (replace forwardemail with your username):

git clone git@github.com:forwardemail/forwardemail.net.git
cd forwardemail.net

Install npm dependencies:
```
pnpm install
```

Install fonts:

echo ttf-mscorefonts-installer msttcorefonts/accepted-mscorefonts-eula select true | sudo debconf-set-selections

sudo apt-get install xfonts-75dpi fontconfig libxrender1 xfonts-base ttf-mscorefonts-installer libfontconfig fonts-powerline

Install wkhtmltopdf (optional: only used for generating PDF receipts):

wget "https://github.com/wkhtmltopdf/packaging/releases/download/0.12.6.1-2/wkhtmltox_0.12.6.1-2.$(lsb_release -c -s)_$(dpkg --print-architecture).deb"
sudo dpkg -i "wkhtmltox_0.12.6.1-2.$(lsb_release -c -s)_$(dpkg --print-architecture).deb"

Install MongoDB by following the guide at https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-ubuntu/#install-mongodb-community-edition
Install Redis by following the guide at https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-ubuntu-20-04.

Local Development Guide

Once you have followed Requirements, you should now have all the dependencies, repository, and npm packages installed.

You can start any of the services using our pre-built commands to make it easy. Note that all of these pre-built commands are using nps.

Service Name	Command	Default Development Port	Development Preview URL
Web	`npm start web`	`3000`	http://localhost:3000
API	`npm start api`	`4000`	http://localhost:4000
Bree	`npm start bree`	None	None
SMTP	`npm start smtp`	`2432`	`telnet localhost 2432`
MX	`npm start mx`	`2525`	`telnet localhost 2525`
IMAP	`npm start imap`	`2113`	`telnet localhost 2113`
POP3	`npm start pop3`	`2115`	`telnet localhost 2115`
SQLite	`npm start sqlite`	`3456`	`telnet localhost 3456`
CalDAV	`npm start caldav`	`5000`	http://localhost:5000

You can test the local SMTP, IMAP, POP3, and CalDAV servers using Thunderbird, telnet, or openssl. Note that all local development servers do not require TLS and are running with { rejectUnauthorized: true } option passed to TLS server configurations.

Try running the local web server:

npm start webAndWatch

It should open a new tab for you with the local web server running and terminal output using signale. You can now open your browser to http://localhost:3000 for a development playground.

You can also run all apps at once:

npm start all

Note that if you open your browser to http://localhost:3000 and no assets are rendering, then you must have forgotten to run a build beforehand. You can run builds manually by running npm run build.

An easy way to kill all existing Node apps running is by typing killall node.

Server Infrastructure

Naming Convention

Our server alias naming convention consists of the following fields, joined together by a hyphen, and converted to lower case:

App name (e.g. "web", "api", "bree", "smtp", "imap", "pop3", "sqlite", or "caldav")
(Optional) App count (starting with 1) of the application (relative to the same provider and region). Only applicable for apps with potential count > 1.
Provider name (abbreviated to 2 characters, e.g. "do" for "Digital Ocean", but you can optionally use more verbose for providers such as "Vultr" as "vultr")
Region name (this is the region name given by the provider, e.g. "sfo3" for DO's SFO3 region)

For example, one of our web servers is named web-do-sfo3 and another is web-vultr-dallas.

Load Balancing

All server aliases with the same hostname (with a minimum count of at least 2) are set in Cloudflare under a geo-located load balancer.

Unless otherwise noted, all of the servers should have dedicated CPU's and not be running in a shared CPU environment.

Provisioning

See the ansible folder for our Ansible configuration and playbooks, which we use to provision servers with.

We recommend you to install yamllint and configure it in your editor while working with Ansible playbooks.

Also note that ansible-lint is a helpful linting tool you can use if you plan on making changes to playbooks. Note that our current playbooks have several existing lint errors.

First you must provision Ubuntu 18.04 LTS 64-bit server(s) using Digital Ocean, Linode, Vultr, or your host of choice. These newly provisioned server(s) should have your SSH key automatically added.

Follow the Deployment guide below for automatic provisioning and deployment instructions.

Deployment

Ensure that you have pm2 installed locally:
```
npm i -g pm2
```

Ensure that you have ansible and ansible-lint installed locally:

For macOS:

brew install ansible ansible-lint

For Ubuntu:

sudo apt-add-repository ppa:ansible/ansible
sudo apt update
sudo apt install ansible
pip3 install ansible-lint

Install ansible-galaxy requirements (assumes current working directory is the root of this repository):
```
ansible-galaxy install -r ansible/requirements.yml
```
Set up host configuration by copying the hosts.yml file template:
```
cp ansible/playbooks/templates/hosts.yml hosts.yml
```
Edit this configuration and update the file with your newly created server aliases and IP addresses. You can add more than one host to each group if you are setting up load balancing. Refer to the Naming Convention documentation for our recommended approach to server alias naming. Note that this file is automatically ignored by git. If you have a private repository and would like to commit this, then remove hosts.yml from the root .gitignore file.
```
vim hosts.yml
```
Set up environment configuration by copying the env file template:
```
cp ansible/playbooks/templates/env .env.production
```
Edit this configuration and reference the official Lad documentation for a list of all available environment variables (or see .env.defaults). You will need to open this file in your preferred editor and set the values for any fields containing TODO, whereby you replace TODO with the appropriate value. Preserve double quotes where they are already defined.
```
vim .env.production
```
Generate pm2 ecosystem files using our automatic template generator. We created an ansible-playbook.js which loads the .env.production environment variables rendered with @ladjs/env into process.env, which then gets used in the playbooks. This is a superior, simple, and the only known dotenv approach we know of in Ansible. Newly created ecosystem-api.json, ecosystem-bree.json, ecosystem-web.json, ecosystem-smtp.json, ecosystem-imap.json, ecosystem-pop3.json, ecosystem-sqlite.json, and ecosystem-caldav.json files will now be created for you in the root of the repository. If you ever more add or change IP addresses, you can simply re-run this command.
```
node ansible-playbook ansible/playbooks/ecosystem.yml -l 'localhost'
```
Set up the web, API, and CalDAV server(s) (see patterns and ansible-playbook flags docs if you need help). If you completely (or partially) run this playbook (or any others below), then the second time you try to run it may not succeed. This is because we prevent root user access through security hardening. To workaround this, run the same command but without --user root appended as it will default to the devops user created.
```
node ansible-playbook ansible/playbooks/http.yml --user root -l 'http'
```

Set up the Bree server(s):

node ansible-playbook ansible/playbooks/bree.yml --user root -l 'bree'

Set up the SMTP server(s):

node ansible-playbook ansible/playbooks/smtp.yml --user root -l 'smtp'

Set up the IMAP server(s):

node ansible-playbook ansible/playbooks/imap.yml --user root -l 'imap'

Set up the POP3 server(s):

node ansible-playbook ansible/playbooks/pop3.yml --user root -l 'pop3'

Set up the SQLite server(s):

node ansible-playbook ansible/playbooks/sqlite.yml --user root -l 'sqlite'

Set up the MX server(s):

node ansible-playbook ansible/playbooks/mx1.yml --user root -l 'mx1'
node ansible-playbook ansible/playbooks/mx2.yml --user root -l 'mx2'

Set up GitHub deployment keys for all the servers. Note that the deployment-keys directory is ignored from git, so if you have a private repository and wish to commit it, then remove deployment-keys from the .gitignore file.
```
node ansible-playbook ansible/playbooks/deployment-keys.yml -l 'imap:pop3:smtp:http:bree:sqlite' --user deploy
```

Go to your repository "Settings" page on GitHub, click on "Deploy keys", and then add a deployment key for each servers' deployment key copied to the deployment-keys directory. If you're on macOS, you can use the pbcopy command to copy each file's contents to your clipboard. Use tab completion for speed, and replace the server names and paths with yours. You can also use the gh CLI at https://cli.github.com/manual/gh_repo_deploy-key_add as shown below (switch the repo/org/repo paths and deployment key paths below to yours):

gh repo deploy-key add deployment-keys/api-do-sf-ca.pub -R forwardemail/forwardemail.net
gh repo deploy-key add deployment-keys/api-vu-sj-ca.pub -R forwardemail/forwardemail.net
gh repo deploy-key add deployment-keys/web-do-sf-ca.pub -R forwardemail/forwardemail.net
gh repo deploy-key add deployment-keys/web-vu-sj-ca.pub -R forwardemail/forwardemail.net
gh repo deploy-key add deployment-keys/bree-vu-sj-ca.pub -R forwardemail/forwardemail.net
gh repo deploy-key add deployment-keys/smtp-vu-sj-ca.pub -R forwardemail/forwardemail.net
gh repo deploy-key add deployment-keys/smtp-do-sf-ca.pub -R forwardemail/forwardemail.net
gh repo deploy-key add deployment-keys/imap-vu-sj-ca.pub -R forwardemail/forwardemail.net
gh repo deploy-key add deployment-keys/imap-do-am-nl.pub -R forwardemail/forwardemail.net
gh repo deploy-key add deployment-keys/pop3-vu-sj-ca.pub -R forwardemail/forwardemail.net
gh repo deploy-key add deployment-keys/sqlite-do-sf-ca.pub -R forwardemail/forwardemail.net
gh repo deploy-key add deployment-keys/caldav-do-sf-ca.pub -R forwardemail/forwardemail.net

Set up PM2 deployment directories on all the servers:

pm2 deploy ecosystem-web.json production setup

pm2 deploy ecosystem-api.json production setup

pm2 deploy ecosystem-bree.json production setup

pm2 deploy ecosystem-smtp.json production setup

pm2 deploy ecosystem-imap.json production setup

pm2 deploy ecosystem-pop3.json production setup

pm2 deploy ecosystem-sqlite.json production setup

pm2 deploy ecosystem-caldav.json production setup

Create a SSL certificate at Namecheap (we recommend a 5 year wildcard certificate), set up the certificate, and download and extract the ZIP file with the certificate (emailed to you) to your computer. We do not recommend using tools like LetsEncrypt and certbot due to complexity when you have (or scale to) a cluster of servers set up behind load balancers. In other words, we've tried approaches like lsyncd in combination with crontab for certbot renewals and automatic checking. Furthermore, using this exposes the server(s) to downtime as ports 80 and 443 may need to be shut down so that certbot can use them for certificate generation. This is not a reliable approach, and simply renewing certificates once a year is vastly simpler and also makes using load balancers trivial. Instead you can use a provider like Namecheap to get a cheap SSL certificate, then run a few commands as we've documented below. This command will prompt you for an absolute file path to the certificates you downloaded. Renewed your certificate after 1 year? Simply follow this step again. Do not set a password on the certificate files. When using the openssl command (see Namecheap instructions), you need to use *.example.com with an asterisk followed by a period if you are registering a wildcard certificate.
```
node ansible-playbook ansible/playbooks/certificates.yml --user deploy
```
Important: If you renew or change certificates in the future, then after running the previous command, you will subsequently need to reload the processes as such:
```
#
# NOTE: See the "Important" note above BEFORE running this command.
#       This command ONLY APPLIES for certificate renewals/changes.
#
pm2 deploy ecosystem-web.json production exec "pm2 reload all"
pm2 deploy ecosystem-api.json production exec "pm2 reload all"
pm2 deploy ecosystem-smtp.json production exec "pm2 reload all"
pm2 deploy ecosystem-imap.json production exec "pm2 reload all"
pm2 deploy ecosystem-pop3.json production exec "pm2 reload all"
pm2 deploy ecosystem-sqlite.json production exec "pm2 reload all"
pm2 deploy ecosystem-caldav.json production exec "pm2 reload all"
```
Create a DKIM key for your domain name (must match WEB_HOST environment variable) with a default selector of default (must match DKIM_KEY_SELECTOR environment variable). Then upload it to the servers:
```
node ansible-playbook ansible/playbooks/dkim.yml --user deploy
```
(Optional) Create a Google application credentials profile file and store it locally. You only need this if you want to support automatic translation. The following command will prompt you for the absolute file path (e.g. /path/to/client-profile.json). See the mandarin docs for more information.
```
node ansible-playbook ansible/playbooks/gapp-creds.yml -l 'imap:pop3:smtp:http:bree:sqlite' --user deploy
```
(Optional) Copy over custom TTF or OTF fonts to be installed on the server (e.g. used for PDF rendering, rendering with Sharp, open-graph images, etc):
```
node ansible-playbook ansible/playbooks/fonts.yml -l 'imap:pop3:smtp:http:bree:sqlite' --user deploy
```
Note that at the time of this writing we copy these files:
- inconsolata-dz.otf
- VCHoney-Bold.otf
- VCHoney-Regular.otf
- VCHoney-SemiBold.otf
(Optional) Copy over GPG keys to be installed on the server (e.g. used for GPG signing security.txt, see https://forwardemail.net/security.txt).

NOTE: This assumes that you have also set in .env file the keys of GPG_SECURITY_KEY with the full file path to the key and GPG_SECURITY_PASSPHRASE with the GPG passphrase. You can export via gpg --armor --export-secret-key YOURKEYIDHERE > .gpg-security-key. You can get YOURKEYIDHERE via gpg --list-keys. You can generate a key with gpg --full-generate-key (e.g. for support@yourdomain.com or security@yourdomain.com). Note you should also update the path in config/index.js for openPGPKey value.
```
node ansible-playbook ansible/playbooks/gpg-security-key.yml -l 'imap:pop3:smtp:http:bree:sqlite' --user deploy
```

Copy the .env.production to the servers:

node ansible-playbook ansible/playbooks/env.yml -l 'imap:pop3:smtp:http:bree:sqlite' --user deploy

Run an initial deploy to all the servers:

pm2 deploy ecosystem-web.json production

pm2 deploy ecosystem-api.json production

pm2 deploy ecosystem-bree.json production

pm2 deploy ecosystem-smtp.json production

pm2 deploy ecosystem-imap.json production

pm2 deploy ecosystem-pop3.json production

pm2 deploy ecosystem-sqlite.json production

pm2 deploy ecosystem-caldav.json production

Save the process list on the servers so when if the server were to reboot, it will automatically boot back up the processes:

pm2 deploy ecosystem-web.json production exec "pm2 save"

pm2 deploy ecosystem-api.json production exec "pm2 save"

pm2 deploy ecosystem-bree.json production exec "pm2 save"

pm2 deploy ecosystem-smtp.json production exec "pm2 save"

pm2 deploy ecosystem-imap.json production exec "pm2 save"

pm2 deploy ecosystem-pop3.json production exec "pm2 save"

pm2 deploy ecosystem-sqlite.json production exec "pm2 save"

pm2 deploy ecosystem-caldav.json production exec "pm2 save"

Test by visiting your web and API server in your browser (click "proceed to unsafe" site and bypass certificate warning).
Configure your DNS records for the web and API server hostnames and respective IP addresses.
Test by visiting your web and API server in your browser (in an incognito window). There should not be any certificate warnings (similar to the one that occurred in step 15).
(Optional) Remove the local .env.production file for security purposes. If you do this, then make sure you have a backup, or securely back up off the server in the future before destroying the server.
```
rm .env.production
```
(Optional) Remove the local certificate files you downloaded locally and specified in step 11. If you do this, then make sure you have a backup, or securely back up off the server in the future before destroying the server.
Finished. If you need to deploy again, then push your changes to GitHub master branch and then follow step 14 again. We recommend you to read the Ansible getting started guide, as it provides you with insight into commands like ansible all -a "echo hello" which can be run across all or specific servers.

Deployment Advice

If you do not change any assets, then there is no reason to do a full deployment.

For example, if you made changes to a web controller, then you only need to deploy it to the web codebase without a build:

pm2 deploy ecosystem-web.json production exec "git reset --hard HEAD && git pull origin master && pm2 reload all"

Name		Name	Last commit message	Last commit date
Latest commit History 2,781 Commits
.github/workflows		.github/workflows
.husky		.husky
ansible		ansible
app		app
assets		assets
config		config
emails		emails
helpers		helpers
jobs		jobs
locales		locales
media		media
routes		routes
scripts		scripts
test		test
.babelrc		.babelrc
.browserslistrc		.browserslistrc
.commitlintrc.js		.commitlintrc.js
.copywrite.hcl		.copywrite.hcl
.dockerignore		.dockerignore
.editorconfig		.editorconfig
.env.defaults		.env.defaults
.env.schema		.env.schema
.eslintignore		.eslintignore
.gitattributes		.gitattributes
.gitignore		.gitignore
.lintstagedrc.js		.lintstagedrc.js
.npmrc		.npmrc
.postcssrc.js		.postcssrc.js
.prettierrc.js		.prettierrc.js
.pug-lintrc.js		.pug-lintrc.js
.remarkignore		.remarkignore
.remarkrc.js		.remarkrc.js
.stylelintrc.js		.stylelintrc.js
.xo-config.js		.xo-config.js
.yamllint.yml		.yamllint.yml
LICENSE.md		LICENSE.md
README.md		README.md
ansible-playbook.js		ansible-playbook.js
ansible.cfg		ansible.cfg
api.js		api.js
ava.config.js		ava.config.js
bree.js		bree.js
caldav-server.js		caldav-server.js
caldav.js		caldav.js
docker-compose-dev.yml		docker-compose-dev.yml
docker-compose.yml		docker-compose.yml
ecosystem-api.json		ecosystem-api.json
ecosystem-bree.json		ecosystem-bree.json
ecosystem-caldav.json		ecosystem-caldav.json
ecosystem-imap.json		ecosystem-imap.json
ecosystem-mx.json		ecosystem-mx.json
ecosystem-pop3.json		ecosystem-pop3.json
ecosystem-smtp.json		ecosystem-smtp.json
ecosystem-sqlite.json		ecosystem-sqlite.json
ecosystem-web.json		ecosystem-web.json
gulpfile.js		gulpfile.js
imap-server.js		imap-server.js
imap.js		imap.js
mx-server.js		mx-server.js
mx.js		mx.js
nodemon.json		nodemon.json
package-scripts.js		package-scripts.js
package.json		package.json
pop3-server.js		pop3-server.js
pop3.js		pop3.js
proxy.js		proxy.js
smtp-bree.js		smtp-bree.js
smtp-server.js		smtp-server.js
smtp.js		smtp.js
sqlite-bree.js		sqlite-bree.js
sqlite-server.js		sqlite-server.js
sqlite.js		sqlite.js
web.js		web.js

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Table of Contents

How do I get started

For Consumers

For Developers

Requirements

macOS

Ubuntu

Local Development Guide

Server Infrastructure

Naming Convention

Load Balancing

Provisioning

Deployment

Deployment Advice

License

About

Releases

Packages

Languages

License

ceonelson/forwardemail.net

Folders and files

Latest commit

History

Repository files navigation

Table of Contents

How do I get started

For Consumers

For Developers

Requirements

macOS

Ubuntu

Local Development Guide

Server Infrastructure

Naming Convention

Load Balancing

Provisioning

Deployment

Deployment Advice

License

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages