Getting Started with DES at Fermilab

Accounts and Access

To access Fermilab's computing resources and the DES Collaboration tools follow the instructions found here:

By following these instructions you should be granted: (1) a Fermilab user name, (2) a Fermilab Kerberos account, (3) an entry in the DES membership database, (4) a request for access to the DES data through DESDM, and (5) access to the DES Collaboration wiki on Redmine. Once you have access to Redmine, more complete documentation for Collaboration tools and services can be found here:

After you have been granted a Fermilab username, you can follow the instructions here to get an account on the DES cluster machines. This may involve sending an email to Liz Buckley-Geer and Brian Yanny, so be sure to include some context regarding who you are and what you will be working on. Once you have been granted access to the DES cluster, you can login to one of the DES machines using kerberos and ssh:

> kinit	<username>@FNAL.GOV
> ssh -XY <username>

There are some quirks for getting ssh to forward the right credentials, so check out the instructions on the "Getting Started" page if you have trouble.

While at Fermilab, there are several wireless networks available: fgz, guest, eduroam. The guest network is accessible to everyone, but has limited bandwith and may not have all ports open. The eduroam network should be accessible if you are member of an eduroam institution, but also has limited bandwith. The fgz network is the primary Fermilab wireless network. Devices can be registered for the fgz network by following these instructions.

Using Linux

Most scientific computing these days is performed on some flavor of the Linux operating system. Over the past few years, the entry barrier to Linux has be significantly lowered by the popularization of user-oriented Linux operating systems such as Ubuntu. Fermilab computers run Scientific Linux which is a rebuild of Red Hat Enterprise Linux. There are various Linux and command line tutorials spread accross the web. Here are a few that caught my eye:

While these tutorials may be a good place to start, Google is the single most important resource when learning Linux (and programming in general). A close second is the Stack Overflow user forum. However, it can be difficult to find the answer to a question that you don't know how to ask. If that is the case, find a person to ask!

Scientific Python with Anaconda

Python has become the standard for scientific computing in astrophysics and cosmology. One straightforward and robust way to install python and its suite of scientific computing packages is through Anaconda. Anaconda provides an installation of python, numpy, scipy, matplotlib, etc. as well as a package manager, conda, for installing other commonly used packages like astropy.

If you are working on the DES cluster machines at Fermilab, Anaconda and many of the common packages used for DES science are already installed, and you can skip to the "Setup an Environment" section. If you find that a package you need is not already installed, Anaconda is flexible enough to allow you to easily install and integrate that package into a new python environment (see "Create an Environment"). Finally, if you want to install Anaconda on your own machine, some useful links are included in the "Installation" section.

Setup an Environment

The DES cluster has a centralized Anaconda installation. The centralized installation contains many common packages and should allow you to get started quickly. Below we provide a minimal set of instructions to get you started; more detailed instructions can be found on the conda documentation

To activate the current default DES environment (des17a) from the centralized Anaconda installation on the DES cluster using the bash shell:

> export CONDA_DIR=/cvmfs/
> export PATH=$CONDA_DIR/bin:$PATH
> source activate des17a
Unfortunately, the activate script is bash-specific. For cshell (and its derivatives) setting the path to an environment and any subsidiary setup must be done explicitly:
> setenv CONDA_DIR /cvmfs/
> setenv PATH ${CONDA_DIR}/envs/des17a/bin:$PATH
> source ${CONDA_DIR}/envs/des17a/etc/conda/activate.d/iraf.csh

By default, conda will search for environments under the root installation (i.e., $CONDA_DIR/envs) and in your home directory (i.e., $HOME/.conda/envs). You can modify your conda search path either by setting the CONDA_ENVS_PATH environment variable or appending to the envs_dir entry in your .condarc file. A more detailed discusion of conda configuration can be found here with a detailed discussion of conda directory paths here.

Create an Environment

At some point it may be necessary to update packages or install new packages that are not included in the centralized installation. Additionally, you may want to create your own custom software environment associated to a specific analysis. To achieve these goals, you will want to create a new conda environment. By default, new environments that you create will live in your home directory under $HOME/.conda. Since conda environments can grow quite large, the best practice is to symlink $HOME/.conda to one of the large DES disks before you start creating environments:

> mkdir -p /data/des40.a/data/$USER/conda
> ln -s /data/des40.a/data/$USER/conda $HOME/.conda
To create a new environment, start by performing the general conda setup described. Use the conda create command to create a new environment containing numpy, scipy, and matplotlib:
> conda create --no-update-deps --name myenv numpy scipy matplotlib
This command will create an environment named myenv in $HOME/.conda/envs. The --no-update-deps tells conda to avoid updating dependencies if possible. This will save time and space by allowing conda to link against existing versions of python, ipython, etc. when it creates the environment. Your new environment can be activated similarly to the centralized environments (see above for directions on modifying your environment path):
> source activate myenv
If you find yourself making a bunch of environments, make sure to clean up your conda directory from time to time:
> conda clean --packages
> conda clean --tarballs


Anaconda is already installed on the Fermilab DES cluster and a personal installation should not be necessary (skip to the next section for setup instructions). However, if you want to install Ananconda on your own machine, follow the installation instructions here:

Once Anaconda is installed, you can use conda to install additional packages into working environments. For more details on package installation see here:

For the more advanced user, it is also possible to perform a light weight install of conda itself using the Miniconda:


Here is the official python tutorial:
As a more interactive introduction, I've heard some good things about Code Academy:

The main packages that are used within DES for scientific computing are numpy, scipy, matplotlib, and astropy.

Product Management with EUPS

For some projects it is important to strictly control the versions and packages of software that you are using. Anaconda (described above) provides a light-weight, python-oriented way of doing this. However, many projects at Fermilab rely on the older Extended Unix Product System (EUPS) or Unix Product System (UPS) package managers.

DES hosts a variety of EUPS-controlled packages and depending on the project you are working on you will probably need to customize your setup. However, the following instructions should suffice to get you started with 90% of the (python) packages that you need.

> CVMFS=/cvmfs/
> source $CVMFS/users/kadrlica/
> setup healpy 1.8.1+4
> setup pyfits 3.3+3
> setup easyaccess 1.3.1+1

This should setup most of the basic scientific python packages such as: numpy, scipy, matplotlib, pandas, and fitsio (as well as pyfits and healpy which are explicitly setup). Due to the way that EUPS builds the PYTHONPATH, the EUPS python start up and imports can be significantly slower than Anaconda.

I strongly recommend *against* mixing Anaconda with EUPS. The two package managers set the PYTHONPATH in different ways and do not play nicely with each other. This is not saying that the two systems cannot work with one another, just that it's probably a headache that you would like to avoid. If you are starting from scratch, I would suggest using Anaconda; however, many times working with EUPS is unavoidable.

 Contact Me!