Local development of packages#
We recommend you create isolated virtual environments using mamba (conda) to develop existing or new projects, then pip install development requirements over that mamba environment. We offer guidance of two variants for installing dependencies: one exclusively using Python packages, and a second which installs most dependencies as conda packages (being able to better handle non-Python dependencies such as Nvidia CUDA-compiled packages). In both cases, the top-level package (or packages) is (are) always installed on the development environment through pip (with the –editable option).
Note
Pip may be configured with command-line options as shown below, but equally through environment variables, or configuration files. We leave to the developer’s discretion the decision on how to best manage their own use of pip.
Note
You may develop software against different development (c.f.
Setting up Development Profiles). In the context of these
instructions, we assume your development profile is located at
../profile.
In this variant, the latest (beta) versions of internally developed dependencies are fetched from our local package registry if applicable. Furthermore, external dependencies are fetched from PyPI and respect versions used on the continuous integration (CI) server. It is useful to reproduce bugs reported on the CI, during test builds:
$ git clone <PACKAGE-URL> # e.g. git clone git@gitlab.idiap.ch/software/clapp
$ cd <PACKAGE> # e.g. cd clapp
$ conda activate base # only required if conda/mamba is not on your $PATH
(base) $ mamba create -n dev python=3.10 pip
(base) $ conda activate dev
(dev) $ pip install --pre --index-url https://token:<YOUR-GITLAB-TOKEN>@gitlab.idiap.ch/api/v4/groups/software/-/packages/pypi/simple --extra-index-url https://pypi.org/simple --constraint ../profile/python/pip-constraints.txt --editable '.[qa,doc,test]'
Note
If you need to install private packages developed through GitLab, you
must generate a personal token with at least access to
the package registry (currently implemented through the read_api
privilege).
Otherwise, you may suppress token:<YOUR-GITLAB-TOKEN>@ from the
index-url option above.
Tip
Optionally, you may create a standard Python virtual environment (instead of a conda virtual environment) with either venv or virtualenv, and then apply the same instructions above to locally install dependencies and the package itself.
The base Python version in this case will be that used to create the virtual environment.
In this variant, the latest (beta) versions of internally developed dependencies are fetched from our local conda (beta) package registry, if applicable. Furthermore, external dependencies are fetched from conda-forge and respect versions used on the continuous integration (CI) server. It is useful to reproduce bugs reported on the CI, during conda-package test builds, or to install further non-Python dependencies required for package development (e.g. Nvidia CUDA-enabled packages):
$ git clone <PACKAGE>
$ cd <PACKAGE>
$ conda activate base # only required if conda/mamba is not on your $PATH
(base) $ devtool env -vv .
(base) $ mamba env create -n dev -f environment.yaml
(base) $ conda activate dev
(dev) $ pip install --no-build-isolation --no-dependencies --editable .
Note
The application devtool env uses the conda API to parse your
package’s recipe (typically at conda/meta.yaml) and
pyproject.toml, and then to search dependencies (including those for
quality-assurance, documentation and tests, which may be not listed on
conda/meta.yaml). The installation respects CI constraints
established on your chosen profile.
After that step, your package will be installed and ready for use inside the
dev environment. You must activate the virtual environment everytime you
want to further develop the package, or simply deactivate it when you branch
off to another activity.
With the development environment active, you can optionally test the package installation by either building its Sphinx documentation, doctests, running its test suite, or the quality assurance (pre-commit) checks:
(dev) $ pre-commit run --all-files # quality assurance
(dev) $ pytest -sv tests/ # test units
(dev) $ sphinx-build doc sphinx # documentation
(dev) $ sphinx-build doctest doc sphinx # doctests
Developing multiple existing packages simultaneously#
It may happen that you may want to develop several packages against each other
for your project. This is the case if you are changing a high-level package
package-a, that in turn depends on functionality on package-b you may
also need to adapt. While you change package-b, you want to verify how
these changes work on package-a. The procedure to accomodate this is
similar to the above, except you will git-clone and pip-install more packages:
In this variant, the latest (beta) versions of internally developed dependencies are fetched from our local package registry if applicable. Furthermore, external dependencies are fetched from PyPI and respect versions used on the continuous integration (CI) server.
To setup your environment, you must install packages in reverse dependence
order, with the top-level package (package-a in this example), being
installed by last.
$ git clone <PACKAGE-A>
$ cd <PACKAGE-A>
$ git clone <PACKAGE-B> src/<PACKAGE-B>
$ conda activate base # only required if conda/mamba is not on your $PATH
(base) $ mamba create -n dev python=3.10 pip
# get the constraints for the "target" development environment.
# this is just an example:
(base) $ curl -O constraints.txt https://gitlab.idiap.ch/software/dev-profile/-/raw/main/python/pip-constraints.txt
(base) $ conda activate dev
(dev) $ for pkg in "src/package-b" "."; do pip install --pre --index-url https://token:<YOUR-GITLAB-TOKEN>@gitlab.idiap.ch/api/v4/groups/software/-/packages/pypi/simple --extra-index-url https://pypi.org/simple --constraint constraints.txt --editable "${pkg}[qa,doc,test]"; done
In this variant, the latest (beta) versions of internally developed dependencies are fetched from our local conda (beta) package registry, if applicable. Furthermore, external dependencies are fetched from conda-forge and respect versions used on the continuous integration (CI) server. It is useful to reproduce bugs reported on the CI, during conda-package test builds:
$ git clone <PACKAGE-A>
$ cd <PACKAGE-A>
$ git clone <PACKAGE-B> src/<PACKAGE-B>
$ conda activate base # only required if conda/mamba is not on your $PATH
(base) $ devtool env -vv src/package-b .
(base) $ mamba env create -n dev -f environment.yaml
(base) $ conda activate dev
(dev) $ for pkg in "src/package-b" "."; do pip install --no-build-isolation --no-dependencies --editable "${pkg}"
Installing all constrained packages#
If you plan to develop many packages together, it may be faster to first pre-install all (constrained) packages, to then pip-install all the individual packages. Because the mamba (or conda) ecosystem is a superset of Python packages, we only provide this option:
$ conda activate base # only required if conda/mamba is not on your $PATH
(base) $ devtool fullenv -vv
(base) $ mamba env create -n dev -f environment.yaml
(base) $ conda activate dev
(dev) $ for pkg in "src/*"; do pip install --no-build-isolation --no-dependencies --editable "${pkg}"
Creating new packages#
To create a new package, use our cookiecutter template and associated instructions. Do not copy another package source code, or no Xmas gifts for you…