-
Notifications
You must be signed in to change notification settings - Fork 0
dev01 computer setup
Tom Purucker edited this page Mar 12, 2019
·
27 revisions
This page is intended to describe the basic setup process for a development machine, either an EPA Windows or Mac.
- create a github account if you do not have one
- get yourself added to the quanted group
- windows command line git commands install git-scm
- git-scm
- git comes with xcode command line tools additional install
git config --global credential.helper wincred
git config credential.helper 'cache --timeout=9000'
- get os 64-bit dependent installation files
- install python 3.X via anaconda at anaconda
- default install location on windows @ C:\Users\lan___id\AppData\Local\Continuum\Anaconda3 (or2) (hidden on epa machines)
- default install location on mac @ ~/anaconda/bin/python
Select Start, select Control Panel. double click System, and select the Advanced tab. Click Environment Variables. ... In the Edit System Variable (or New System Variable) window, specify the value of the PATH environment variable for pythonpath to C:\Users\lan___id\AppData\Local\Continuum\Anaconda3
clone from git from parent of where you want to have source
you may want to use your elevated privileges to create a git directory on
the root drive of your machine, for example c:\git
git clone https://github.com/quanted/qed.git
git checkout current_dev_branch # (usually dev)
Add dropbox-based secret txt files with passwords to a subdirectory called /secrets.
SAM data available from dropbox https://www.dropbox.com/home/Preprocessed
Also from aws console: https://s3.console.aws.amazon.com/s3/buckets/sampreprocessed/?region=us-east-1&tab=overview
Data can be browsed here: http://sampreprocessed.s3-website-us-east-1.amazonaws.com/
pip install awscli
aws configure
# download sam inputs from aws s3
# to the directory /qed/flask_qed/pram_flask/ubertool/ubertool/sam/bin/Preprocessed
# the data in this directory is .gitignored
aws s3 sync s3://sam2preprocessed .
# where . assumes you are running from /qed/flask_qed/pram_flask/ubertool/ubertool/sam/bin/Preprocessed
aws s3 sync s3://qed-basins .
# where . assumes you are running from /qed/flask_qed/hms_flask/need to create a .gitignored dir for localdev
- Open preferred console/bash terminal
- Go to program directory, on windows it may be found here C:\Program Files\MongoDB\Server\3.4\bin
- Run './mongod.exe --dbpath PATH/TO/FILE', where the --dbpath specifies a directory for data storage, or retrieval if data files exist in directory.
- The mongoDB database is now accessible through localhost:27017
- prerequisites: download and compile redis for mac/linux
- windows: download redis
- celery conda/pip install described in pycharm integration
- Open preferred console/bash terminal
- Go to program directory, on windows it may be found here C:\Program Files\Redis
- Inside the conf directory, open redis.conf and verify that the port specified is 6379
- If it is not, create a new conf file and change the port number to 6379
- Run './redis-server.exe ./conf/redis.conf', change the config file if necessary
- The redis database is now accessible through localhost:6379
- Issue: Running the command starts but then stops.
- Open up the task manager and check if redis-server.exe is already running, if it is kill the process (assuming you aren't actively using it somewhere else). Retry starting redis.
- From the command line, activate your python virtual envir you wish to run celery in.
- On windows, run 'pip install celery==3.1.24' this is the last version of celery to support windows.
- On mac/linux, run 'pip install celery'
- Go to the root directory of the application you wish to run, for the qed project if you are wanting to run flask this would be ./qed/flask_qed/
- Run 'celery worker -A PROJ -Q QUEUE --loglevel=DEBUG -c CONCURRENCY -n NAME'
- -A is the celery application file, for flask this is celery_cgi
- -Q is the list of celery queues to enable for this worker
- --loglevel specifies the type of logs to receive in stdout
- -c sets the max concurrency, default if not used is all available threads on the machine
- -n is the Name of the celery worker
- Celery worker documentation can be found here for celery 4.1.0
- To run the celery worker for flask_qed, execute 'celery worker -A celery_cgi -Q qed --loglevel=DEBUG' -n qed_worker', concurrency can be specified if you wish to restrict the threads available to celery.
export PS1="\A \w \\$ \[$(tput sgr0)\]"
- Create new 'Run/Debug Configuration' using 'Python'
- Set script to 'C:\PATHTO celery.exe in env'
- example: 'C:\Program Files\Anaconda3\env\py36_qed\Scripts\Celery.exe'
- Set Script parameters to: '-A tasks worker -Q sam -loglevel=debug -concurrency=1 -n sam_worker'
- Set Working Directory to 'C:\PATHTO celery.exe in env'
- example: 'C:\Program Files\Anaconda3\env\py36_qed\Scripts\Celery
- Save/Apply and Run celery (trying to run in debug mode fails)
- sample code can be found in quanted\qed_ubertool sam branch
- In pycharm with run flask_cgi.py and django in debug mode, celery can be run and watched
- QED celery process diagram
- gdal
- google earth engine
- install r
- install rstudio
- jetbrains webstorm
- sublime text
- db browser for sqlite
- postman
- google app engine
- dropbox
- kindle
- microsoft office
- docker
- google drive
- gitter
- slack
- chrome
- skype
- spotify
- anaconda navigator
- antisleep (app store)
- memory clean (app store)
- avast mac security
- omnidisksweeper
- xquartz - tcl/tck for R
- finder -> preferences -> show all filename extensions
- finder -> configure favorites (macintosh hd, desktop, applications, downloads, user, dropbox, git)
- finder -> view -> show path bar
- show hidden files
- iterm2 for mac
- git clone https://github.com/mbadolato/iTerm2-Color-Schemes.git
- create new: preferences -> profile -> ubuntu
- add github
- linked to iterm2 color scheme
- link colors from github pull
- preferences -> general -> native full screen
- preferences -> appearance -> show tab even when there is only one
- preferences -> profiles -> general -> working directory and send text at start ls -al