1.7. Miniconda: Python(s) in a convenient setup¶
1.7.1. Overview¶
Python is increasingly used in the imaging world. Everyone and their sheepadoodle seems to be writing in it and coming across it in some part of their daily work. Part of its appeal is that there exist many useful modules for carrying out tasks (plotting, machine learning, statistical modeling, writing in XKCD font, etc.).
Along with the flexibility and broad appeal of Python comes a need for managing and maintaining dependencies: making sure the necessary dependencies are installed, adding/updating modules, and using the requisite version of Python. For example, there were major changes between Python 2 and 3 (just try evaluating 1/2 in each case), and occasionally across sub-versions code can break; we can only wait and see what will happen with Python 4.
Anyway, managing potentially multiple versions of Python, as well as diferent sets of dependencies can be tricky. There are different ways of doing this, but through personal experience I have found using “Conda” to be (by far) the easiest and most straightforward. We can basically set up one or more environments with specific Python versions, modules and dependencies, and we can swap back and forth between them fairly easily—they are each separate and distinct, and don’t get in each other’s way. We can set one to be “on” by default, not really think about it often, and then just change to another environment if we need to. Nice.
This page is neither a recommendation to use Conda, nor is it a full tutorial on using it. It is just meant to be a helpful starting point to Conda, if it sounds appealing to you, to manage your Python needs.
We will use the specific installation form of Conda called “Miniconda”, because it starts of being light-weight, and you can then add whatever you need.
1.7.2. Set up Conda (verbose)¶
Whether you are using Linux—either directly or through the Windows Subsystem Linux—or Mac, the setup will basically follow the same steps. (Miniconda can also be installed on Windows directly, but since AFNI doesn’t run there, who cares?)
This example is for setting up Conda for a single user system. Instructions for installing Conda for multiple users is described on the conda website here. We don’t discuss it further.
Note that you should not need your sudo
password to
install Conda on your system, nor to run later “conda” commands.
Download+install miniconda¶
Click here and select installer to download (e.g., to home directory):
Note on selection: Most computers now are 64-bit, and selecting the “Python 3.7” version makes sense (you can still use it to set up Python 2.* environments on your system). For Macs, I used the bash script version (rather than pkg), just because the former is more similar to what is done on Linux.
Run the downloaded installer script:
Type
bash SCRIPT_NAME
, such as:For Linux:
bash Miniconda3-latest-Linux-x86_64.sh
For Mac:
bash Miniconda3-latest-MacOSX-x86_64.sh
You will be prompted for various things:
hit
Enter
to read agreement; hit spacebar to navigate through it quickly; typeyes
to agree to it.when prompted about installation directory, I just left it in the default “home”.
did I wish to initialize Miniconda3? I typed: yes
I then got a message that I was successfully set up; and also a “conda: Command not found message”. So I did…
Make updates known to terminal:
Open a new terminal. You should see a text string like “(base)” to the left of your terminal prompt, and it will be to the left of every prompt (though you can optionally turn that off, which I do).
You can also source the
~/.*rc
file for your shell, which should update your current terminal. (To know what type of shell you are using, you can typeecho $0
.)For bash:
source ~/.bashrc
For tcsh:
source ~/.cshrc
You should see a string “(base)” string stuck before your terminal prompt now. Typing
conda -V
should also show you your version number.Note: your conda version should be at least 4.6.
Sidenote¶
What has Conda done to initialize things in the terminal? It has
stuck some commands into your shell’s startup file; in my
~/.bashrc
file (because I use bash
shell), I can now see the
following text:
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/USERNAME/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/home/USERNAME/miniconda3/etc/profile.d/conda.sh" ]; then
. "/home/USERNAME/miniconda3/etc/profile.d/conda.sh"
else
export PATH="/home/USERNAME/miniconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
... where USERNAME
is replaced with my actual username. If you
chose to install miniconda in a different location than your home
directory, then the paths shown would be different.
Note that conda/miniconda gets setup with path names hardwired into its files, so you will not just be able to move your “miniconda3” directory and update these path locations later and still have it work.
Disable conda prompt string (opt)¶
I do want to have a conda environment up and running in each new terminal by default (so that Python is immediately available), but I don’t like having “(base)” appearing before my prompt all the time. To turn it off, you can just run:
conda config --set changeps1 False
and then in each new terminal, you won’t have that appear anymore.
(To make your existing terminal recognize this change, source your
shell’s ~/.*rc
file, e.g. source ~/.bashrc
or source
~/.cshrc
.)
If in the future you want to re-enable this behavior, then you can always run:
conda config --set changeps1 True
Make basic conda environments¶
To see what conda environments are currently available on your computer, you can type:
conda env list
The name of all available environments will appear in the first
column, as well as their locations in the second. The environment
which is currently loaded will have an asterisk *
after its name.
To create a new environment, at its simplest you can specify a Python version and a set of modules to install, such as:
conda create -y \
-n py37_afni_tiny \
python=3.7 \
matplotlib numpy
where the new environment’s name will be “py37_afni_tiny”; I called it this because that is basically the minimal set of modules used within AFNI (and even those aren’t used very often).
To make a similar setup for Python 2.7 (no earlier versions of Python should be used), one could run:
conda create -y \
-n py27_afni_tiny \
python=2.7 \
matplotlib numpy \
pillow ipython
Now, if I type conda list env
, I will see a list of all my
available environments:
# conda environments:
#
base * /home/USERNAME/miniconda3
py27_afni_tiny /home/USERNAME/miniconda3/envs/py27_afni_tiny
py37_afni_tiny /home/USERNAME/miniconda3/envs/py37_afni_tiny
To turn off a Conda environment, you can run:
conda deactivate
To activate a particular environment, you can run conda activate
NAME
, such as:
conda activate py27_afni_tiny
To see what modules are installed in your active environment (and their version numbers) you can run:
conda list
... which, in the current “py27_afni_tiny” would be as follows (and you might have slightly different things):
- show list output y/n -So, in this environment, I could run a program that imports matplotlib, whereas in the “base” environment, I couldn’t.
To add a module to an existing environment, you can use the following syntax:
conda install -n <env_name> <package>
... so, for example example, you could add the scipy module to one of the above environments with:
conda install -n py27_afni_tiny scipy
To update a module or package in a currently active environment, you can use:
conda update <package>
... for example,:
conda update matplotlib
Specify default environment for the terminal¶
From the Conda initialization, the “base” environment is the default
one running in any new terminal. We might prefer to make one of our
newly made environments the default. To do so, I will include a line
conda activate NAME
in my shell’s ~/.*rc
file somewhere
after the “conda initialize” lines.
Thus, since I am running “bash” shell, I have the following line in my
~/.bashrc
file:
conda activate py37_afni_tiny
Once you source that ~/.*rc
file again or open a new terminal, you
should see that specific environment loaded (look for the asterisked
env in the output of conda env list
). If it didn’t work,
please check that your conda version is at least 4.6 (via conda
-V
).
If you do choose to automatically activate your own env like this, then you might also want to run this in a terminal:
conda config --set auto_activate_base false
so that conda doesn’t pre-load the “base” environment unnecessarily (taking a bit of time).
Make conda environments, more generally¶
The environment builder works a bit like a package manager, where it can get a lot of common modules from a default, central repository, but if you want more specialized ones, you might have to add from another place. To add new repositories to pull from, you add a channel to your Conda setup.
Let’s say you want to add the Sphinx module with cloud-theme support (I doubt you will, but just as an example). If you try:
conda create -y \
-n py37_afni_with_sph \
python=3.7 \
matplotlib numpy \
sphinx cloud_sptheme
You will likely get the following message:
- show text output y/n -This message: 1) tells us our current channels don’t contain this module; 2) shows us our current channels; and 3) helpfully directs us to a webpage to search for a new channel that might have it.
So, searching for “cloud_sptheme” at https://anaconda.org/, one of the top package owners appears to be “conda-forge” (and this is a fairly large platform). So, to add it to my repository list for getting modules, I would run:
conda config --add channels conda-forge
Then, I can retry my conda create ..
command above, which should
result in success this time.
Thus, if you try to build an environment and get told that some desired module can’t be found, you can search for it amongst available channels, add that channel to your Conda setup, and try again.
1.7.3. Set up Conda (quick)¶
Download and install
At ensuing prompts, I mostly type “yes” and/or accept default options; see verbose description above.
For Linux and tcsh:
set script_file = Miniconda3-latest-Linux-x86_64.sh curl -O https://repo.anaconda.com/miniconda/${script_file} bash ${script_file}
when done:
source ~/.cshrc
For Linux and bash:
script_file=Miniconda3-latest-Linux-x86_64.sh curl -O https://repo.anaconda.com/miniconda/${script_file} bash ${script_file}
when done:
source ~/.bashrc
For Mac and tcsh:
set script_file = Miniconda3-latest-MacOSX-x86_64.sh curl -O https://repo.anaconda.com/miniconda/${script_file} bash ${script_file}
when done:
source ~/.cshrc
For Mac and bash:
script_file=Miniconda3-latest-MacOSX-x86_64.sh curl -O https://repo.anaconda.com/miniconda/${script_file} bash ${script_file}
when done:
source ~/.bashrc
Remove annoying prompt string (opt)
conda config --set changeps1 False
Add any channels, if needed (opt)
Many standard modules are default channels, but one can add more as necessary, e.g.:
conda config --add channels conda-forge --add channels anaconda
To find out what channel has your module of interest, use the searchbar at the top of this page: https://anaconda.org
Make some new environments
conda create -y \ -n py37_afni_tiny \ python=3.7 \ matplotlib numpy conda create -y \ -n py27_afni_tiny \ python=2.7 \ matplotlib numpy \ pillow ipython
Add to an existing environment
conda install -n py27_afni_tiny scipy
Activate an env by default
Specify the env to activate in your
~/.*rc
file.Open up the
~/.bashrc
or~/.cshrc
text file and putconda activate NAME
after the conda-initialization lines in that file, or, e.g. copy+paste:For tcsh:
echo "" >> ~/.cshrc echo "conda activate py37_afni_tiny" >> ~/.cshrc echo "" >> ~/.cshrc
For bash:
echo "" >> ~/.bashrc echo "conda activate py37_afni_tiny" >> ~/.bashrc echo "" >> ~/.bashrc
Note: This assumes your conda version (
conda -V
) is at least 4.6.If you do automatically activate your own env, then you might also want to do this:
conda config --set auto_activate_base false
so that conda doesn’t pre-load the “base” environment unnecessarily (taking a bit of time).
Quicktasks with Conda
List modules (starred/asterisked one is active):
conda env list
Deactivate current module:
conda deactivate
Activate/switch to a specific module:
conda activate NAME
See module+version list in current env:
conda list
Update a package in the current environment:
conda update PACKAGE
1.7.4. Fancier things with Conda¶
There are a lot of fancy things that can be done with Conda that we will not describe here. A good starting point is the Managing Environments documentation.
Cloning¶
One concept with Conda is cloning environments: if I can setup a Conda environment on my laptop with a certain set of modules, each with a certain version number, then I can “clone” it and use that exact recipe to setup a duplicate environment on a different computer. This is a nice concept for reproducibility (as sometimes using different version numbers of modules can affect outputs/results).
More on cloning and building identical conda envs can be read.
Note that in practice, truly duplicating environments exactly is actually pretty tough. Getting very close might be good enough for most purposes, though, in practice.
View Conda Cheatsheet¶
It’s here: the conda cheatsheet.
1.7.5. A note on making envs and choosing modules¶
It is entirely up to you, Dear User, what modules you install and how you organize your environments (and if you even choose to use Conda). At the moment AFNI-land has pretty minimal Python requirements. In fact, the AFNI set of recommended modules might simply fit inside those requirements that you have for other software/uses, and you might not need to do anything new.
We certainly don’t anticipate or desire a person to set up one specific environment for running AFNI, then another for running some other software, and then another for another project… While that is possible, it seems annoying and inefficient. And unnecessary. So, hopefully, you can set up one environments (or maybe two, because of the Python-2.7 and Python-3.* split), and not have to switch too much.