Munch Lab

Jupyter on a remote compute node (without tears)

You can run Jupyter notebook in your browser from a compute node on the cluster. This way your analysis runs on the file system where your data is, and you can keep data, code and documentation in one place. It only works using Chrome browser.

One time setup

To make this work you need to make a hashed password on the cluster. If you use Anaconda python on the cluster (you should!) you can follow the steps below. Start by logging into the cluster. Then paste and execute the following,  one block at a time:

conda install jupyter
cd /home/$USER
mkdir -p .jupyter
cd .jupyter
openssl req -x509 -nodes -days 10000 -newkey rsa:2048 -keyout mykey.key -out mycert.pem

If the dir you are in does not contain a file called jupyter_notebook_config.py, you need to make one by executing this:

jupyter notebook --generate-config

Now edit jupyter_notebook_config.py to set these parameters:

hashed_pass=`python -c "from __future__ import print_function; from notebook.auth import passwd; res = passwd(); print(res)"`
c.NotebookApp.certfile = '/home/$USER/.jupyter/mycert.pem'
c.NotebookApp.keyfile = '/home/$USER/.jupyter/mykey.key'
c.NotebookApp.open_browser = False
c.NotebookApp.ip = '*'
c.NotebookApp.port = $UID
c.NotebookApp.password = '$hashed_pass'
EOF

You also need to set up a your ssh connection to the cluster so you can connect securely without typing the password every time. If you have not done this already this is how you do it:

First test if you have these two files on your local machine:

~/.ssh/id_rsa
~/.ssh/id_rsa.pub

if not you generate them like this you generate a pair of authentication keys like this – do not enter a passphrase when prompted:

ssh-keygen -t rsa

Now use ssh to create a directory ~/.ssh on the cluster (assuming your username there is X):

ssh X@login.genome.au.dk mkdir -p .ssh

Finally append the public key on your local machine to the file .ssh/authorized_keys on the cluster and enter the password one last time:

cat ~/.ssh/id_rsa.pub | ssh X@login.genome.au.dk 'cat >> .ssh/authorized_keys'

From now on you can log into B as b from A as a without being prompted for a password.

Start a notebook server

This is a little more involved but I have written a script that automates these steps. You can get it here or clone it:

git clone https://github.com/kaspermunch/slurm_jupyter

It has lots of options, but to start a jupyter server under some project (say monkey) you just need to execute the script like this and it will open a window in your Chrome browser:

slurm_jupyter -A monkey

If your user name on the cluster (eg. donald) is different from that on your local machine you also need to supply the that:

slurm_jupyter -u donald -A monkey

The first time Chrome opens the connection to the cluster it will give you an error page saying “Your connection is not private”. You then need to click “Advanced” and then “Proceed to localhost (unsafe)”.  Then your file tree on the cluster should appear.

To stop the server just press Ctrl-C and the script will do all the canceling, closing and cleanup for you before it exits.

If you are into manual labor here are the steps to do it the hard way:

On the cluster, start an interactive session on a compute node like this (here under the ‘monkey’ project’, for 7 days and with 100g of ram).

srun -A monkey --time 7-0 --mem 100g --pty bash

Next you need to take note of the name of the compute node you are on.  You can see the name of the node like this:

hostname -s

You also need to know your user id on the cluster.

id -u $USER

Now start the jupyter (iPython) server  with

jupyter notebook

Now you jupyter server is running on the compute node. Keep the terminal window open for as long as you want to use it.

Now open a new terminal window on your local machine. you need to open a ssh connection from your machine to the jupyter server. Say the host name was s02n64, that your user id on the cluster is 6358, and that your user name is kmt. In that case you need to open an ssh connection from your own machine to the cluster like this:

ssh -L6358:s02n64:6358 kmt@login.genome.au.dk

Keep this terminal window open for as long as you want to access the jupyter server.

Now if your user name was 6358, then you can now point your open the Chrome browser (not Safari) to https://localhost:6358. It will complain that the connection is not secure. Click ‘Advanced’ and allow Chrome to use the connection anyway. Now you are prompted for your cluster password – and voila!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: