Command-line Interface#

This section contains an overview of command-line applications shipped with this package.

devtool#

Idiap development tools - see available commands below

devtool [OPTIONS] COMMAND [ARGS]...

env#

Creates a development environment for one or more projects.

The environment is created by scanning conda’s meta.yaml and Python pyproject.toml files for all input projects. All input that is not an existing file path, is considered a supplemental conda package to be installed. The environment is dumped to disk in the form of a conda-installable YAML environment. The user may edit this file to add Python packages that may be of interest.

To interpret meta.yaml files found on the input directories, this command uses the conda render API to discover all profile- constrained and unconstrained packages to add to the new environment.

devtool env [OPTIONS] META...

Options

-P, --profile <profile>#

Directory containing the development profile (and a file named profile.toml), or the name of a configuration key pointing to the development profile to use

Default:

default

-p, --python <python>#

Version of python to build the environment for

Default:

3.11

-u, --public, -U, --no-public#

Set this to include private channels/indexes on your plan. For conda packages in this case, you must execute this within the Idiap intranet.

-s, --stable, -S, --no-stable#

Set this to exclude beta channels from your build

-o, --output <output>#

The name of the environment plan file

Default:

environment.yaml

-v, --verbose#

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messags), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Arguments

META#

Required argument(s)

Examples:

  1. Creates an environment installation plan for developing the currently checked-out package, and the development profile in ../profile:

    $ git clone <package>
$ cd <package>
$ conda activate base
(base) devtool env -vv .
(base) $ mamba env create -n dev -f environment.yaml
(base) $ conda activate dev
(dev) $ pip install --no-build-isolation --no-dependencies --editable .

    You may, of course, hand-edit the output file environment.yaml to adjust for details, add conda or Python packages you’d like to complement your work environment. An example would be adding debuggers such as ipdb to the installation plan before calling mamba env create.

  2. By default, we use the native Python version of your conda installation as the Python version to use for the newly created environment. You may select a different one with –python=X.Y’. You may also set the output filename with `–output=name.yaml``, if the default does not please you:

    $ conda activate base
(base) devtool env -vv --python=3.9 --output=whatever-i-like.yaml .

  3. To develop multiple packages you checked out, just add the meta package files of all packages you wish to consider, then pip-install the packages on teh top of the created environment, in reverse dependence order (e.g. package A depends on B):

    $ mkdir dev-dir
$ cd dev-dir
$ git clone <repo-of-B> src/B
$ git clone <repo-of-A> src/A
$ conda activate base
(base) $ devtool env -vv src/*
(base) $ mamba env create -n dev -f environment.yaml
(base) $ conda activate dev
(dev) $ pip install --no-build-isolation --no-dependencies --editable "src/B"
(dev) $ pip install --no-build-isolation --no-dependencies --editable "src/A"

fullenv#

Creates a development environment with all constrained packages.

devtool fullenv [OPTIONS]

Options

-P, --profile <profile>#

Directory containing the development profile (and a file named profile.toml), or the name of a configuration key pointing to the development profile to use

Default:

default

-p, --python <python>#

Version of python to build the environment for

Default:

3.11

-Y, --only-python, --no-only-python#

Only installs Python packages, and not conda packages (except for Python itself, and pip)

Default:

False

-o, --output <output>#

The name of the environment plan file

Default:

environment.yaml

-v, --verbose#

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messags), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Examples:

  1. Creates a development environment with all constrained packages (assumes the default profile is configured):

$ conda activate base
(base) $ devtool fullenv -vv
(base) $ mamba env create -n dev -f environment.yaml
(base) $ conda activate dev

  1. Creates a development environment with a specific profile:

$ conda activate base
(base) $ devtool fullenv -vv -P ../profile
(base) $ mamba env create -n dev -f environment.yaml
(base) $ conda activate dev

Tip

You may hand-edit the output file environment.yaml to adjust for details, add conda or Python packages you’d like to complement your work environment. An example would be adding debuggers such as ipdb to the installation plan before calling mamba env create.

gitlab#

Commands that interact directly with GitLab.

Commands defined here are supposed to interact with gitlab, and add/modify/remove resources on it directly. To avoid repetitive asking, create a configuration file as indicated in the Automated GitLab interaction section of the user guide.

devtool gitlab [OPTIONS] COMMAND [ARGS]...

badges#

Creates stock badges for a project repository.

devtool gitlab badges [OPTIONS] PACKAGE

Options

--update-readme, --no-update-readme#

Whether to update badges in the readme or not.

-d, --dry-run, --no-dry-run#

Only goes through the actions, but does not execute them (combine with the verbosity flags - e.g. -vvv) to enable printing to help you understand what will be done

-s, --server <server>#

The documentation server. Default value is https://www.idiap.ch/software/{group}

-v, --verbose#

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messags), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Arguments

PACKAGE#

Required argument

Examples:

  1. Creates (by replacing) all existing badges in a gitlab project (software/idiap-devtools):

    devtool gitlab badges software/idiap-devtools

Note

This command also affects the README.md file.

changelog#

Generates changelog file for package(s) from the Gitlab server.

This script generates changelogs for either a single package or multiple packages, depending on the value of TARGET. The changelog (in markdown format) is written to the output file CHANGELOG.

There are two modes of operation: you may provide the package name in the format <gitlab-group>/<package-name>. Or, optionally, provide an existing file containing a list of packages that will be iterated on.

For each package, we will contact the Gitlab server and create a changelog using merge-requests (default), tags or commits since a given date. If a starting date is not passed, we’ll use the date of the last tagged value or the date of the first commit, if no tags are available in the package.

devtool gitlab changelog [OPTIONS] TARGET...

Options

-o, --output <output>#
-m, --mode <mode>#

Changes the way we produce the changelog. By default, uses the text in every merge request (mode “mrs”). To use tag annotations, use mode “tags”. If you use “commits” as mode, we use the text in commits to produce the changelog

Default:

mrs

Options:

mrs | tags | commits

-s, --since <since>#

A starting date in any format accepted by dateutil.parser.parse() (see https://dateutil.readthedocs.io/en/stable/parser.html) from which you want to generate the changelog. If not set, the package’slast release date will be used

-v, --verbose#

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messags), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Arguments

TARGET#

Required argument(s)

Examples:

  1. Generates the changelog for a single package using merge requests, outputs results to a file named changelog.md, in markdown format:

    devtool gitlab changelog -vv group/package.xyz -o changelog.md

  2. The same as above, but dumps the changelog to stdout instead of a file:

    devtool gitlab changelog -vv group/package.xyz

  3. Generates the changelog for a single package looking at commits (not merge requests):

    devtool gitlab changelog -vv --mode=commits group/package.xyz

  4. Generates the changelog for a single package looking at merge requests starting from a given date of January 1, 2016:

    devtool gitlab changelog -vv --mode=mrs --since=2016-01-01 group/package.xyz

  5. Generates the changelog for a set of packages, listed one per line in a text file:

    devtool gitlab changelog -vv --mode=mrs --since=2016-01-01 group/package.xyz

getpath#

Downloads files and directories from gitlab.

Files are downloaded and stored. Directories are recursed and fully downloaded to the client.

devtool gitlab getpath [OPTIONS] PACKAGE PATH [OUTPUT]

Options

-r, --ref <ref>#

Download path from the provided git reference (may be a branch, tag or commit hash). If not set, then use the default package branch as reference to download.

-v, --verbose#

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messags), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Arguments

PACKAGE#

Required argument

PATH#

Required argument

OUTPUT#

Optional argument

Examples:

  1. Get the file profile.toml from software/dev-profile’s default branch:

    devtool gitlab getpath software/dev-profile profile.toml

  2. Get the directory gitlab (and eventual sub-directories) from software/dev-profile:

    devtool gitlab getpath software/dev-profile gitlab

jobs#

Lists jobs on a given runner identified by description.

devtool gitlab jobs [OPTIONS] [TAGS]...

Options

-s, --status <status>#

The status of jobs we are searching for - one of “running”, “success”, “failed” or “canceled”

Default:

running

Options:

running | success | failed | canceled

-v, --verbose#

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messags), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Arguments

TAGS#

Optional argument(s)

Examples:

  1. List running jobs on any runners with tag “bob” (default):

    devtool gitlab jobs -vv

  2. List running jobs on a runner with tag “macos”:

    devtool gitlab jobs -vv macos

  3. List running jobs on a runner with tag “macos” and “foo”:

    devtool gitlab jobs -vv macos foo

lasttag#

Returns the last tag information on a given PACKAGE.

devtool gitlab lasttag [OPTIONS] PACKAGE

Options

-v, --verbose#

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messags), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Arguments

PACKAGE#

Required argument

Examples:

  1. Get the last tag information of the software/clapp package:

    devtool gitlab lasttag software/clapp

  2. Get the last tag information of the beat/beat.core package:

    devtool gitlab lasttag beat/beat.core

release#

Tags packages on GitLab from an input CHANGELOG in markdown format.

By using a CHANGELOG file as an input (e.g. that can be generated with the changelog command), this script goes through all packages listed (and in order):

  • Modifies pyproject.toml with the new release number and pins the dependencies according to the specified profile’s constraints (if one was specified)

  • Sets-up the README links to point to the correct pipeline and documentation for the package

  • Commits, tags and pushes the git project adding the changelog description for the GitLab release page

  • Waits for the pipeline launched by the previous step to end

  • Bumps the package version again, to the next beta patch

  • Re-modifies the README to point to the “latest” documentation and pipeline versions

  • Re-commits and pushes the whole with the option [ci skip].

N.B.: When the option pin-dependencies is set, the versions of the dependencies in pyproject.toml will be pinned to those of the Python constraints.txt file available in the select development profile (choose using option --profile).

The changelog is expected to have the following structure:

# group-name/package-name: ``major``|``minor``|``patch``

  Description of changes in group-name/package-name

# group-name/package-name-2: ``major``|``minor``|``patch``

  Description of changes in group-name/package-name-2

The headings, each, correspond to package names, followed by a colon (:), and then one of the following keywords: major, minor, or patch, indicating which part of the version number will be bumped during the release procedure (N.B.: following semantic version numbering).

An indented piece of text marks the release notes for the package to be tagged, in any amount of detail. The description of a single package is suffixed by another package heading, or the end of the file.

You may use GitLab-flavoured markdown (GLFM) to refer to closed issues or merge requests. Alternatively, use the command changelog to auto-generate the description for your release.

devtool gitlab release [OPTIONS] CHANGELOG

Options

-P, --profile <profile>#

Directory containing the development profile (and a file named profile.toml), or the name of a configuration key pointing to the development profile to use

Default:

default

-p, --pin-dependencies, --no-pin-dependencies#

If set, then pin dependencies from the dev-profile on the package to be released. By default your default dev-profile is used. You may override this using the –profile option

-d, --dry-run, --no-dry-run#

Only goes through the actions, but does not execute them (combine with the verbosity flags - e.g. -vvv) to enable printing to help you understand what will be done

-v, --verbose#

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messags), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Arguments

CHANGELOG#

Required argument

Examples:

  1. Runs the release procedure for all packages listed in changelog.md:

    devtool gitlab release -vv changelog.md

    Tip

    In case of errors, just edit the changelog file to remove packages already released before relaunching the application.

  2. The option --dry-run can be used to let the script print what it would do instead of actually doing it:

    devtool gitlab release -vv --dry-run changelog.md

  3. You may also pin package dependencies upon the release, so that the shipped package respects a particular development profile set of pins:

    devtool gitlab release -vv --pin-dependencies changelog.md

    The default profile is used, if set on your configuration file. Otherwise, you may specify it explicitly like:

    devtool gitlab release -vv --profile=specific --pin-dependencies changelog.md

runners#

Commands for handling runners.

devtool gitlab runners [OPTIONS] COMMAND [ARGS]...
disable#

Disables runners on whole gitlab groups or single projects.

You may provide project names (like “group/project”), whole groups, files containing list of projects to load or omit the last argument, in which case all projects using this runner will be affected.

devtool gitlab runners disable [OPTIONS] NAME [TARGETS]...

Options

-d, --dry-run, --no-dry-run#

Only goes through the actions, but does not execute them (combine with the verbosity flags - e.g. -vvv) to enable printing to help you understand what will be done

-v, --verbose#

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messags), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Arguments

NAME#

Required argument

TARGETS#

Optional argument(s)

Examples:

  1. Disables the runner with description “macmini” in project software/clapp and software/auto-intersphinx:

    devtool gitlab runners disable -vv macmini software/clapp software/auto-intersphinx

  1. Disables the runner with description “macmini” for all projects in group software:

    devtool gitlab runners disable -vv macmini software

  2. Disables the runner with description “macpro” on all projects it is associated to. Notice this command effectively deletes the runner from the gitlab instance:

    devtool gitlab runners disable -vv pro
enable#

Enables runners on whole gitlab groups or single projects.

You may provide project names (like “group/project”), whole groups, and files containing list of projects to enable at certain runner at.

devtool gitlab runners enable [OPTIONS] NAME TARGETS...

Options

-g, --group, --no-group#

If set, consider the the provided name as a group name

-d, --dry-run, --no-dry-run#

Only goes through the actions, but does not execute them (combine with the verbosity flags - e.g. -vvv) to enable printing to help you understand what will be done

-v, --verbose#

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messags), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Arguments

NAME#

Required argument

TARGETS#

Required argument(s)

Examples:

  1. Enables the runner with description “linux-srv01” on all projects inside groups “group1” and “group2”:

    devtool gitlab runners enable --group -vv linux-srv01 group1 group2

  2. Enables the runner with description “linux-srv02” on a specific project:

    devtool gitlab runners enable -vv linux-srv02 group1/my-project
list#

Lists projects a runner is associated to.

devtool gitlab runners list [OPTIONS] NAME

Options

-v, --verbose#

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messags), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Arguments

NAME#

Required argument

Examples:

  1. Lists all projects a runner is associated to:

    devtool gitlab runners list -vv macmini

settings#

Updates project settings.

devtool gitlab settings [OPTIONS] PROJECTS...

Options

-a, --avatar <avatar>#

Set this to update the project icon (avatar)

-D, --description <description>#

Set this to update the project description

-g, --group, --no-group#

If set, consider the the provided name as a group name

-A, --archive, --unarchive#

Set this to archive or unarchive a project

-d, --dry-run, --no-dry-run#

Only goes through the actions, but does not execute them (combine with the verbosity flags - e.g. -vvv) to enable printing to help you understand what will be done

-v, --verbose#

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messags), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Arguments

PROJECTS#

Required argument(s)

Examples:

  1. List settings in a gitlab project (software/idiap-devtools):

  2. Simulates an update to the project description:

update-pins#

Updates pip/mamba/conda package constraints (requires conda)

The update is done by checking-up conda-forge and trying to create an environment with all packages listed on the current conda constraints. Once an environment has been resolved, it is “copied” to the Python (pip) constraints file, by excluding non-Python packages.

devtool update-pins [OPTIONS] [MANUAL_PINS]...

Options

-P, --profile <profile>#

Directory containing the development profile (and a file named profile.toml), or the name of a configuration key pointing to the development profile to use

Default:

default

-p, --python <python>#

Version of python to build the environment for

Default:

3.11

-p, --only-pip, -P, --no-only-pip#

Set this to update only pip constraints. The pins will then be copied from the conda-constraints to the pip one.

-v, --verbose#

Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messags), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).

Default:

0

Arguments

MANUAL_PINS#

Optional argument(s)

Examples:

  1. Updates the default constraints taking as base a Python 3.10 environment:

    devtool update-pins --python=3.10

  2. Updates the default constraints using the default Python version (installed on the base environment), and respecting a few provided constraints (manual pins):

    devtool update-pins --python=3.10 opencv=4.5.1 pytorch=1.9

  3. Updates the constraints on a checked-out profile directory:

    devtool update-pins --python=3.10 --profile=../another-profile