Skip to content

Using the servers

Jump to bottom
Gábor Recski edited this page Mar 23, 2021 · 15 revisions

Please read this document carefully before using the institute servers for the first time, at least the part on [Monitoring].(https://github.com/recski/tuw-nlp/wiki/Using-the-servers#monitoring-and-nice)

Linux

The institute servers are running Ubuntu Linux. Using them requires at least a basic knowledge of the Linux command-line environment. There are many introductory materials online, this one covers the absolute basics.

Monitoring and nice

All computing resources (CPU, RAM, I/O, disk space, etc.) are shared with other users and it is your responsibility to make sure that you do not disrupt other users' work. The htop tool makes it easy to track CPU and memory use, the du tool can help in analyzing disk usage. If you need to use many CPUs at once, always use the nice command to launch your processes. If you're doing something I/O intensive, you should also use ionice. Always keep an eye on the overall CPU and memory use of the server so that you know immediately if there is trouble.

Setting up your environment

Python

You should avoid using the default python environment (you cannot install packages into it anyway). Virtual environment managers will let you create a Python environment for each project where you can install packages. We recommend conda, you can get an easy-to-use minimal version here.

Other

If you require other packages, usually you should find a way to install it locally, i.e. without sudo

Useful tools

Tmux

By default your processes are tied to the SSH connection you are using to access the servers, so e.g. an internet fallout can cause them to stop. You can avoid this by using a terminal multiplexer, we recommend tmux.

Editors

We recommend using terminal-based editors and developing your code on the servers. Some of us use vim, here's an online interactive tutorial. You can also launch a terminal-based tutorial by running vimtutor.

Jupyter

If you want to use jupyter on the server, you can use port forwarding to access it from the browser on your computer. First, start jupyter on the server with the following command (if the port is in use, choose another one):

jupyter notebook --port=8088 --ip=0.0.0.0 --no-browser

Then open a terminal on your own machine and start port-forwarding like this:

ssh -L 8088:localhost:8088 <server address> -N

Windows users may have to specify the ssh port (if not 22) and the username:

ssh -L 8088:localhost:8088 <username>@<server address> -p <ssh port> -N

Now you can acccess your jupyter server on localhost:8088. If you're using a newer version of jupyter, you will need to authenticate by pasting into the browser the link that jupyter prints on startup, which looks something like this:

http://0.0.0.0:8088/?token=<some long token>

Take care to stop jupyter when you're done using it, to save resources.

See also:

Clone this wiki locally