Revision as of 11:46, 18 September 2020

The default Python installed is 2.7.5 and 3.4.10, compiled by GCC 4.8.5.

We also have Anaconda3 and Miniconda3 and installed so that you can create your own Python development environments. The difference between Miniconda and Anaconda is that Anaconda has a lot more numeric and scientific libraries installed by default.

Loading the anaconda module

Once logged into HPC, you can use the module command to load the anaconda module.

module load anaconda

For miniconda

module load miniconda

Please note that you can only use one of the "conda" modules at a time. If you have anaonda loaded, but need to switch to miniconda, you will first need to unload the anaconda module:

module unload anaconda
module load miniconda

Creating Anaconda or Miniconda Python environment

You can use the conda create command to create a new Python environment. For example, to create a Python environment named mypy38 with Python 3.8:

conda create --name mypy38 python=3.8

You'll then be provided with the location of the environment being created and the packages will be installed. Then type y to confirm the installation. By default, this will create the environment in your home directory at ~/.conda/env/. If you would like to save it to another location, you can use the -p option:

conda create -p /PATH/mypy38 python=3.8

Please note, after the creation, you may be told to use "conda activate mypy38" to activate the environment. If you use the command, you'll then be told to do "conda init". You probably don't want to do that, as it'll alter your basrc file and load one of the conda version by default.

Using a Python Environment in miniconda or anaconda

To get a list of available conda environment, you can use the command: conda env list. You will get a list of the available Python environment.

# conda environments:
#
mypy38 /home/mtech/bdeng/.conda/envs/mypy38
base * /opt/ohpc/pub/apps/anaconda3

Activating a Python environment

To use the mypy38 environment, use the command:

source activate mypy38

You'll then notice the environment name prepended to the command prompt, for example:

(mypy38) [username@oredigger ~]$

Now you can check your python version:

(mypy38) [username@oredigger ~]$ python -V
Python 3.8.5

Installing new Python packages

Once your Python environment is activated, you can install additional packages for your project. For example, to install the scipy package, use the command: (mypy38) [username@oredigger ~]$ conda install scipy

For some packages, you may need to specify the channel using the -c option. Refer to the documentations of the package you'll need. To get a list of conda packages installed, use:

(mypy38) [username@oredigger ~]$ conda list

Deactivating a Python environment

When you finished with your environment or you need to switch to a different environment. You can deactivate the current environment with:

(mypy38) [username@oredigger ~]$ conda deactivate

You'll see the environment name removed from the command prompt:

[username@oredigger ~]$

Additional conda commands

To remove a conda environment: conda env remove --name environment_name

(if environment not in default path, use:conda env remove -p /PATH/environment_name)

To export a conda environment, first activate the environment, then use: conda env export > environment.yml

This will export a list of your environment's packages to the file environment.yml

For more conda commands:

Conda Cheat Sheet

Conda User Guide

Use Python environment in a job submission script

To use a Python environment in a Slurm job submission script, below is a sample using 1 core with the mypy38 Python environment: #!/bin/sh
#SBATCH -J pythontest #Name of the computation
#SBATCH -N 1 # Total number of nodes requested
#SBATCH -n 1 # Total number of tasks per node requested
#SBATCH -t 01:00:00 # Total run time requested - 1 hour
#SBATCH -p normal # compute nodes partition requested

module load anaconda
source activate mypy38
python mypython.py