Setup a PyPi Mirror

setup-pypi-mirror.md

Setup PyPi Mirror

This guide describes how to create a local PyPI mirror for use in isolated networks without internet access. We'll use bandersnatch to dump packages, a web server to serve them, and devpi to enable search functionality.

This is possible using devpi, web server, bandersnatch and volume to store the packages.

• Dump all pypi packages with bandersnatch,
• Serve them for install via web server
• Enable the search feature by mirroring the web server using devpi.

Dump PyPi using bandersnatch

Take a computer with internet access (or at least access to pypi.python.org) and install bandersnatch:

You can do it in a venv:

python3 -m venv venv
source venv/bin/activate
pip install bandersnatch==5.3.0

OR via virtualenv

virtualenv venv
source venv/bin/activate
pip install bandersnatch

Configure bandersnatch to dump the files to the specified path by editing /etc/bandersnatch.conf:

directory = /path/to/external/storage

Start the mirroring process:

bandersnatch mirror

There will be a web directory with at least two directories in it i.e. packages and simple.

This operation may take few days to be done, depending on your internet access speed. As of writing this, pypi storage requires about 250GB of storage space.

Setup the webserver

Copy the web folder onto the server and configure the web server to make the web folder the root folder of a site (can be in subdirectory).

Configure pip on the server for the user that will run devpi to point to the webserver:

~/.pip/pip.conf

[global]
index-url = http://localhost/path/to/web/simple

Note: It is important that the index-url points to simple.

Now you should be able to use pip to install any python package you have dumped from pypi. You can ensure it by making a virtualenv and a pip install:

python3 -m venv test
source test/bin/activate
python3 -m pip install argparse
deactivate
rm -rf test

Install devpi

Install devpi running the following (virtualenv is still usable for that):

python3 -m pip install devpi-server==6.5.0 devpi-web==4.0.7 devpi-client==6.0.1

The devpi-client is not necessary on the server, it is just a CLI from which we will configure the devpi server. You can install it on one of the local machine if you want to. I will assume it is installed on the server though.

Run devpi-server:

devpi-server --start --no-root-pypi

If you want to stop the server, use --stop. --no-root-pypi is used to tell the server not to create a mirror to the standard pypi server at first launch. It is not necessary to add it again for further launches.

Now you should have devpi-server running on the port 3141. You can try to connect to it using a web browser.

Configure devpi-server to mirror the web server

Using the CLI, connect to the devpi-server:

devpi use http://localhost:3141/
devpi login root --password ''
devpi index -c pypi_mirror type=mirror mirror_url=https://your-server.com/path/to/web/simple

Note: Once again, it is important the url points to simple.

Now the full pypi mirror should be usable, test the mirror:

devpi use http://localhost:3141/root/pypi_mirror/+simple/

python3 -m venv test
source test/bin/activate
python3 -m pip search flask
python3 -m pip install argparse
deactivate
rm -rf test

The pip search should return a lot of results.

Automation and Maintenance

Set up a cron job to update the mirror regularly:

0 2 * * * /path/to/venv/bin/bandersnatch mirror

Security Considerations

• Use HTTPS for both the web server and devpi-server.
• Implement authentication for accessing the PyPI mirror.
• Regularly update all components to patch security vulnerabilities.