Use VS Code to run notebooks on remote servers in environments built on top of LCG views

README.md

Use VS Code to run notebooks on remote servers in environments built on top of LCG views

(Note that this whole thing is a bit cursed as LCG views are a terrible way to distribute Python environments)

Setup

Get a working VS Code server on the remote

Requires:

• Remote - SSH installed on the local machine.
  • c.f. VS Code Remote Development using SSH docs.
• Python extension installed on the remote server.
• Jupyter extension installed on the remote server.

Get a Python virtual environment that sits on top of your LCG view with cvmfs-venv

[01:53] login02.af.uchicago.edu:~ $ export PATH=~/.local/bin:"${PATH}"
[01:53] login02.af.uchicago.edu:~ $ curl -sL https://raw.githubusercontent.com/matthewfeickert/cvmfs-venv/main/cvmfs-venv.sh -o ~/.local/bin/cvmfs-venv
[01:53] login02.af.uchicago.edu:~ $ chmod +x ~/.local/bin/cvmfs-venv
[01:53] login02.af.uchicago.edu:~ $ setupATLAS -3 --quiet
[01:53] login02.af.uchicago.edu:~ $ lsetup 'views LCG_103 x86_64-centos7-gcc11-opt'
************************************************************************
Requested:  views ...
 Setting up views LCG_103:x86_64-centos7-gcc11-opt ...
>>>>>>>>>>>>>>>>>>>>>>>>> Information for user <<<<<<<<<<<<<<<<<<<<<<<<<
************************************************************************
[01:54] login02.af.uchicago.edu:~ $ mkdir -p ~/.venvs
[01:54] login02.af.uchicago.edu:~ $ cd ~/.venvs
[01:54] login02.af.uchicago.edu:~/.venvs $ cvmfs-venv lcg-venv
# Creating new Python virtual environment 'lcg-venv'
[01:55] login02.af.uchicago.edu:~/.venvs $ . lcg-venv/bin/activate
(lcg-venv) [01:55] login02.af.uchicago.edu:~/.venvs $ cd
(lcg-venv) [01:56] login02.af.uchicago.edu:~ $ python -m pip list --local
Package    Version
---------- -------
pip        23.2
setuptools 68.0.0
wheel      0.40.0
(lcg-venv) [01:56] login02.af.uchicago.edu:~ $

Install ipykernel into your local virtual environment

(lcg-venv) [02:00] login02.af.uchicago.edu:~ $ python -m pip install --upgrade ipykernel
(lcg-venv) [02:00] login02.af.uchicago.edu:~ $ python -m pip list --local
Package      Version
------------ -------
comm         0.1.3
debugpy      1.6.7
ipykernel    6.24.0
jupyter_core 5.3.1
pip          23.2
setuptools   68.0.0
traitlets    5.9.0
wheel        0.40.0
(lcg-venv) [02:01] login02.af.uchicago.edu:~ $

Create an IPython kernel for your VS Code server to use while interacting with the notebook

(lcg-venv) [02:01] login02.af.uchicago.edu:~ $ python -m ipykernel install --user --name="lcg-venv" --display-name="LCG view 103 + venv"
Installed kernelspec lcg-venv in /home/feickert/.local/share/jupyter/kernels/lcg-venv
(lcg-venv) [02:02] login02.af.uchicago.edu:~ $

Create an environment startup script for this kernel and mutate the kernelspec to use it

(You'll need to write both these yourselves, or at least edit their existing versions)

• /home/feickert/.local/share/jupyter/kernels/lcg-venv/startup.sh:

#!/usr/bin/env bash

export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
# Allows for working with wrappers as well
source "${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh" --quiet || echo "~~~ERROR: setupATLAS failed!~~~"

lsetup 'views LCG_103 x86_64-centos7-gcc11-opt'

. ~/.venvs/lcg-venv/bin/activate

exec python -m ipykernel_launcher $@

• /home/feickert/.local/share/jupyter/kernels/lcg-venv/kernel.json:

{
 "argv": [
  "/bin/bash",
  "/home/feickert/.local/share/jupyter/kernels/lcg-venv/startup.sh",
  "-f",
  "{connection_file}"
 ],
 "display_name": "LCG view 103 + venv",
 "language": "python-custom",
 "metadata": {
  "debugger": true
 }
}

N.B.: The choice of "python-custom" as the "language" in the IPython kernelspec is an arbitrary choice. The only important thing is that the "language" is not set to "python".

Usage

• Open a notebook in VS Code's remote explorer and select the kernel you made.
• Run any notebook cell. This will cause the kernel to execute the startup.sh and give you a working environment.
• Use as normal.

References

• Running Jupyter notebooks within VSCode (ATLAS Internal)
• microsoft/vscode-jupyter GitHub Issue: Jupyter plugin hides some custom kernelspecs when the executable defined in the argv cannot be found

vscode-remote-explorer-lcg-view-venv.png