1.6. Miniconda: Python(s) in a convenient setup

1.6.1. Overview

This page is neither a recommendation to use Conda, nor is it a full tutorial on using it. It is just meant to be a helpful starting point to Conda, if it sounds appealing to you. Examples here are just for managing a few Python needs for AFNI.

Python is increasingly used in the imaging world. Everyone and their sheepadoodle seems to be writing in it and coming across it in some part of their daily work. Part of its appeal is that there exist many useful modules for carrying out tasks (plotting, machine learning, statistical modeling, writing in XKCD font, etc.).

Along with the flexibility and broad appeal of Python comes a need for managing and maintaining dependencies: making sure the necessary dependencies are installed, adding/updating modules, and using the requisite version of Python. For example, there were major changes between Python 2 and 3 (just try evaluating 1/2 in each case), and occasionally across sub-versions code can break; we can only wait and see what will happen with Python 4.

Anyway, managing potentially multiple versions of Python, as well as diferent sets of dependencies can be tricky. There are different ways of doing this, but through personal experience I have found using “Conda” to be (by far) the easiest and most straightforward. We can basically set up one or more environments (or envs) with specific Python versions, modules and dependencies, and we can swap back and forth between them fairly easily—they are each separate and distinct, and don’t get in each other’s way. We can set one to be “on” by default, not really think about it often, and then just change to another environment if we need to. Nice.

We will use the specific installation form of Conda called “Miniconda”, because it starts of being light-weight, and you can then add whatever you need. Some official reference pages for this are:
https://docs.conda.io/projects/conda/en/latest/user-guide/index.html
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

These notes apply to either Linux or Mac OS. Miniconda can also be installed on Windows directly, but since AFNI doesn’t run there, who cares?

These notes are only for setting up Conda for a single user. Instructions for installing Conda for multiple users is described on the conda website here.

Very important note: you should not need your sudo password to install Conda on your system, nor to run later “conda” commands. Using sudo with conda will just cause headaches later, so don’t do it.

1.6.2. Set up Conda (verbose)

Download+install miniconda

  1. To download+install, copy+paste:


    • ... for Linux:

      curl -O https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b

    • ... for Mac:

      curl -O https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
bash Miniconda3-latest-MacOSX-x86_64.sh -b

    If you don’t have curl -O on your computer, you can instead replace it with curl -O.

    NB: The -b option runs the installer in “batch mode”, saying yes to all prompts: install into home directory, accept licenses. To go through those options manually, don’t include that option.


  2. To initialize conda in shells, copy+paste:

    conda init bash tcsh zsh

    This puts conda setup text into each of the above shell’s ~/.*rc files.

    NB: Conda/miniconda gets setup with path names hardwired into its files, so you cannot just move your “miniconda3” directory and update these path locations later and still have it work.


  3. Make updates known to terminal

    Do one of the following:

    1. Open a new terminal. You should now see a text string like “(base)” to the left of your terminal prompt. (Below, we show you can optionally disable that text.)


    2. Source your shell’s ~/.*rc file (to know your current shell, type echo $0):


      • ... for bash:

        source ~/.bashrc

      • ... for tcsh or csh:

        source ~/.cshrc

      • ... for zsh:

        source ~/.zshrc

    You should see a string “(base)” string stuck before your terminal prompt now. (Below, you can optionally disable that text.)

    Type conda -V to see the version number. NB: It should be at least 4.6.

Load/exit basic conda environments

To see the list of currently available conda environments, type:

conda env list

The name of each available environment appears in the first column (its file location appears in the second). The currently active environment has an asterisk * after its name (one might not be loaded).

To load or “activate” an environment in that list called ENV_NAME, type:

conda activate ENV_NAME

To exit or “deactivate” the current environment, type:

conda deactivate

Make+check conda environments (e.g., Python for AFNI)

There are many aspects to creating a new environment. We only provide the most basic here. For example, conda can manage much more complicated environments, beyond loading just Python+modules.

Here is an example of creating a new environment, one that packages Python version 3.9 and a few useful modules (whose unspecified version numbers will be whatever conda decides):

conda create -y                 \
      -n py39_afni_tiny         \
      python=3.9                \
      matplotlib numpy scipy

This new environment’s name is “py39_afni_tiny”; I called it this because that is basically the minimal set of modules used within AFNI (at present).

To make a similar setup for Python 2.7 (no earlier versions of Python should be used), one could run:

conda create -y                 \
      -n py27_afni_tiny         \
      python=2.7                \
      matplotlib numpy scipy    \
      pillow

Now, if I type conda list env, I will see a list of all my available environments (where ${USER} would actually be replaced by my username):

# conda environments:
#
base                  *  /home/${USER}/miniconda3
py27_afni_tiny           /home/${USER}/miniconda3/envs/py27_afni_tiny
py39_afni_tiny           /home/${USER}/miniconda3/envs/py39_afni_tiny

As noted above, to switch to py39_afni_tiny, I would type:

conda activate py39_afni_tiny

To see what modules are installed in your active environment (and their version numbers) you can run:

conda list

... which, in the current “py39_afni_tiny” would be as follows (and you might have slightly different things):

- show list output y/n -

So, in this environment, I could run a program that imports matplotlib, whereas in the “base” environment, I couldn’t.

Add to+update conda environments

To add a new package or module NEW_PACK to an existing environment ENV_NAME, one can use the following syntax:

conda install -n ENV_NAME NEW_PACK

... so, for example example, you could add the scipy module to one of the above environments with:

conda install -n py27_afni_tiny ipython

To update a module or package CURR_PACK in a currently active environment, you can use:

conda update CURR_PACK

... for example,:

conda update matplotlib

So, let’s say you want one primary environment on your OS to have all your packages of interest loaded, so you don’t have to hop between environments when using different programs. You could make one that has everything you know you need loaded now, and then in the future you could simply keep adding to it. This might be useful with AFNI, in particular, because there are so few requirements here (modern Python with a very small number of modules).

Specify default environment for the terminal

By default, conda will load the “base” environment in any new terminal. To instead have a different environment ENV_NAME loaded in each new terminal/shell, we can add the line conda activate ENV_NAME in the shell’s ~/.*rc file somewhere after the # >>> conda initialize >>> lines.

Since I am running “bash” shell, I have added the following line in my ~/.bashrc file (by opening that file with a text editor):

conda activate py39_afni_tiny

After sourcing that file or opening a new terminal, conda env list should show that environment loaded, in this and in any new terminals. If that did not work, please check that that the conda version is at least 4.6 (via conda -V).

If you do choose to automatically activate your own env like this, then you might also want to run this in a terminal:

conda config --set auto_activate_base false

so that conda doesn’t pre-load the “base” environment unnecessarily (taking a bit of time).

Disable conda prompt string (opt)

Personally I don’t like having the name of the conda environment always appearing before my prompt, like “(base)” or whatever. To not display that text, you can run:

conda config --set changeps1 False

To make your existing terminal recognize this change, source your shell’s ~/.*rc file, e.g. source ~/.bashrc or source ~/.cshrc. Or open a new terminal.

If in the future you want to re-enable this behavior, then you can always run:

conda config --set changeps1 True

These commands edit a text file called ~/.condarc. You can open it and see what defaults/settings you have made, if you wish.

1.6.3. Set up Conda (quick)

  1. Download and install


    • ... for Linux:

      curl -O https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b

    • ... for Mac:

      curl -O https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
bash Miniconda3-latest-MacOSX-x86_64.sh -b

  2. Initialize conda in shells

    conda init bash tcsh zsh

  3. Make updates known to terminal


    Open a new terminal, or source your shell’s ~/.*rc file.

  4. Remove annoying prompt string (opt)

    conda config --set changeps1 False

  5. Make some new environments: AFNI minimal Python

    conda create -y                 \
      -n py39_afni_tiny         \
      python=3.9                \
      matplotlib numpy scipy

conda create -y                 \
      -n py27_afni_tiny         \
      python=2.7                \
      matplotlib numpy scipy    \
      pillow

  6. Load an existing environment

    Copy+paste:

    conda activate ENV_NAME

    For example, from above to setup for AFNI:

    conda activate py39_afni_tiny

  7. Activate an env by default


    To activate some env ENV_NAME by default, put conda activate ENV_NAME in your shell’s ~/.*rc file, after the # <<< conda initialize <<< line.
    For example, to set up for AFNI, put conda activate py39_afni_tiny there.

    NB1: This assumes your conda version (conda -V) is at least 4.6.

    NB2: If you do automatically activate your own env, then also copy+paste the following to not pre-load the “base” env (adding unnecessary time):

    conda config --set auto_activate_base false
    NB3: In general, you don’t want to keep appending different conda activate ... commands in a ~/.*rc file, as each one takes a bit of time.

  8. Add to an existing environment

    Once you have built an environment, if you decide you another package that you might have forgotten, you can do so with:

    conda install -n ENV_NAME PACK_NAME

    For example,

    conda install -n py27_afni_tiny pandas

1.6.4. Short list of conda commands

List available modules (starred/asterisked one is active):

conda env list

Deactivate current module:

conda deactivate

Activate/switch to a specific environment/module:

conda activate ENV_NAME

See module+version list in current env:

conda list

Update a package in the current environment:

conda update PACKAGE

Add a package to some environment:

conda install -n ENV_NAME PACK_NAME

Add a channel to an active environment (with top priority among channels):

conda config --add channels NEW_CHANNEL

Add a channel to an active environment (with bottom priority among channels):

conda config --append channels NEW_CHANNEL

Remove an existing environment (ENV_NAME cannot be active when this command is run):

conda remove --name ENV_NAME --all

Update conda program version:

conda update -n base -c defaults conda

1.6.5. A note on making envs, re. AFNI and more

It is entirely up to you, Dear User, what modules you install and how you organize your environments (and if you even choose to use Conda). At the moment AFNI has very minimal Python requirements. In fact, the AFNI set of recommended modules might simply fit inside those requirements that you have for other software/uses, and you might not need to do anything new.

We certainly don’t anticipate or desire a person to set up one specific environment for running AFNI, then another for running some other software, and then another for another project... While that is possible, it seems annoying and inefficient, and often unnecessary. So, hopefully, you can set up one environment (or a small number of them) and not have to switch too much.

1.6.6. Fancier things with Conda

There are a lot of fancy things that can be done with Conda that we will not describe here. A good starting point is the Managing Environments documentation.

Cloning

One concept with Conda is cloning environments: if I can setup a Conda environment on my laptop with a certain set of modules, each with a certain version number, then I can “clone” it and use that exact recipe to setup a duplicate environment on a different computer. This is a nice concept for reproducibility (as sometimes using different version numbers of modules can affect outputs/results).

More on cloning and building identical conda envs can be read.

Note that in practice, truly duplicating environments exactly is actually pretty tough. Getting very close might be good enough for most purposes, though, in practice.

Make conda environments, more generally

The environment builder works a bit like a package manager, where it can get a lot of common modules from a default, central repository, but if you want more specialized ones, you might have to add from another place. To add new repositories to pull from, you add a channel to your Conda setup.

Let’s say you want to add the Sphinx module with cloud-theme support (I doubt you will, but just as an example). If you try:

conda create -y                 \
    -n py37_afni_with_sph       \
    python=3.7                  \
    matplotlib numpy scipy      \
    sphinx cloud_sptheme

You will likely get the following message:

- show text output y/n -
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - cloud_sptheme

Current channels:

  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

This message: 1) tells us our current channels don’t contain this module; 2) shows us our current channels; and 3) helpfully directs us to a webpage to search for a new channel that might have it.

So, searching for “cloud_sptheme” at https://anaconda.org/, one of the top package owners appears to be “conda-forge” (and this is a fairly large platform). So, to add it to my repository list for getting modules, I would run:

conda config --add channels conda-forge

Then, I can retry my conda create .. command above, which should result in success this time.

Thus, if you try to build an environment and get told that some desired module can’t be found, you can search for it amongst available channels, add that channel to your Conda setup, and try again.