-
Notifications
You must be signed in to change notification settings - Fork 2
Home
The command-line tools to the SIM-CITY framework consists of SIM-CITY client. The JSON-REST interface is called SIM-CITY webservice. If you have any questions, please submit an issue.
All services will be served over HTTP. To allow multiple services on the same host and port, we use nginx as a proxy server.
Install it with
sudo apt-get install nginx
and add proxies to /etc/nginx/sites-enabled/0-default, e.g.
location /couchdb {
proxy_pass http://127.0.0.1:5984;
}
location ~ /\.ht {
deny all;
}CouchDB is used to store the input and output of simulations, and keep track of where they are running. It has its own authentication system.
Unless you already have access to a CouchDB installation, install CouchDB, either as a system package:
sudo apt-get install couchdb
or with docker:
docker install klaemo/couchdb
It can be administered at http://localhost:5984/_utils/index.html. Set up an admin user here. Once it is configured, add a proxy to it with nginx. The database will be initialised with the SIM-CITY client.
Geographic data will be stored in a PostGIS database and served with a Geoserver. There is a howto for installing these: https://nlesc.gitbooks.io/geospatial-handbook/content/geoserver-howto.html.
The SIM-CITY client manages starting jobs, and assigning tasks. See the README therein to see its usage.
It needs to be installed at two locations: your laptop and any cluster you want to run on. For example, assume you're using the <lisa.surfsara.nl> cluster. First, add lisa to your ssh path, by adding the following lines to ~/.ssh/config:
Host lisa
HostName lisa.surfsara.nl
User myuser
and copy your public SSH key to Lisa:
$ scp ~/.ssh/id_rsa.pub lisa:.
$ ssh lisa
lisa$ mkdir -p .ssh
lisa$ cat id_rsa.pub >> .ssh/authorized_keys
lisa$ rm id_rsa.pub
On your laptop and on Lisa, make the following ~/.simcity_client file:
[task-db]
# CouchDB URL
url = https://myhost/couchdb/
# CouchDB username
username = myuser
# CouchDB password
password = mypassword
# CouchDB database
database = simcity
[Execution]
# Base execution directory for local working files
tmp_dir = $TMPDIR
# Base directory for output files that need to be stored
output_dir = $HOME/input
# Base directory for input files
input_dir = $HOME/output
[lisa-host]
# where the run script is located
path = sim-city-client/scripts
script = lisaRun.sh
host = torque://lisa
method = xenonSee sim-city-client/config.ini.dist for additional configuration examples.
On lisa, run:
mkdir $HOME/input $HOME/output $HOME/tmp
git clone https://github.com/indodutch/sim-city-client.git
module load python/2.7
pip install --user virtualenv
~/.local/bin/virtualenv simcity
And add the following to ~/.bashrc:
export TMPDIR=${TMPDIR:-$HOME/tmp}
module load python/2.7
. simcity/bin/activate
Then run
source ~/.profile
make install
On your own machine, run the same commands except module load python/2.7.
Initialise the CouchDB database for the SIM-CITY client by running:
python sim-city-client/scripts/createDatabase.py myadmin
Now you can install any command or model that you want on lisa, and run it with
sim-city-client/scripts/submitJob.py ~/command.sh lisa
To run it locally as well, install exactly the same command ~/command.sh on your own machine. You can then process jobs locally with
sim-city-client/scripts/run.py -eP `hostname`-`date +%F%T`
If a command had an error and you have fixed the reason for the error, you can run
sim-city-client/scripts/scrub.py error
If you don't want to process the commands with errors anymore, run
sim-city-client/scripts/deleteFromView.py error
If jobs were deleted before they started, or ended before they could register this to the database, run
sim-city-client/scripts/deleteFromView.py pending_jobs
and
sim-city-client/scripts/scrub.py active_jobs
Then running
sim-city-client/scripts/startJob.py lisa
will succeed for sure.
To access these functionalities from a web service, install sim-city-webservice:
git clone https://github.com/indodutch/sim-city-webservice.git
cd sim-city-webservice
pip install -U .
To the ~/.simcity_client file, add a few new lines:
[Simulations]
# the maximum number of concurrent jobs that the service has
max_jobs = 4
# default host to submit to
default_host = lisaAnd then serve the service with
make serve
To start the web service automatically, create a new daemon that does these steps using your local daemon provider (e.g., upstart or systemd).
SIM-CITY client can also be used to make a scenario exploration. For example, in a fire response simulation, a fire is started in every ward, and for each the fire response time is measured. First the job is submitted in python:
import csv
import simcity
import sys
import os
import geojson
if len(sys.argv) != 2:
sys.exit("Usage: {} ensemble_name".format(sys.argv[0]))
host = "lisa"
command = "~/bangalore-matsim-0.4/matsim.sh"
version = "0.4"
simulation = 'matsim'
ensemble = sys.argv[1]
max_jobs = 16
# starting a fire in the center of each ward
with open('input/blr/gis/wards.csv') as f:
for row in csv.DictReader(f):
fire = {'id': row['id'], 'x': row['lat'], 'y': row['lon']}
simcity.add_task({
'command': command,
'name': simulation,
'version': version,
'ensemble': ensemble,
'input': {
'numberOfThreads': 16,
'fires': [fire],
'populationSampleFactor': 0.01,
},
})
for i in range(max_jobs):
simcity.submit_if_needed(hostname=host, max_jobs=max_jobs)If some jobs have gotten an error, evaluate why, and scrub them with
python sim-city-client/scripts/scrub.py error
python sim-city-client/scripts/submitJob.py lisa
After the simulations have finished, analyse the output in python:
def add_responsetime(wards):
"""
Add response times to a dict with geojson wards.
In the ward properties, a first_responder and second_responder
property are added, with response time in seconds. This data is
retrieved from the ensemble, which is assumed to start a single
fire in each ward.
"""
db = simcity.get_task_database()
design_doc = simcity.ensemble_view(
db, simulation, version,
'/couchdb/' + simcity.get_config().section('task-db')['database'], ensemble)
tasks = db.get_from_view('all_docs', design_doc=design_doc)
for task in tasks:
if task.done == 0:
print("skipping ward {0}, its simulation is not done"
.format(task.input['fires'][0]['id']))
elif task.done == -1:
print("skipping ward {0}, its simulation has errors: {1}"
.format(task.input['fires'][0]['id'], task.error))
else:
try:
# Get the fire engine paths data from the database
firepaths_str = simcity.download_attachment(task, task.id, 'GeoFirePaths.json')
# Parse it
with open(os.path.join(task.id, 'GeoFirePaths.json')) as f:
firepaths = geojson.load(f)
collection = geojson.FeatureCollection.to_instance(firepaths)
# Collect response times and put it in the relevant wards.
responsetime = sorted([path.properties['responsetime'] for path in collection.features])
ward = wards[task['input']['fires'][0]['id']]
ward['properties']['first_responder'] = responsetime[0]
ward['properties']['second_responder'] = responsetime[1]
except (KeyError, IndexError) as ex:
print(ex)