Compilers

Intel Compilers

The Intel compilers (icc, icpc, ifort) are available through the "intel" module on El Gato.

$module load intel

The Intel MPI compilers (mpiicc, mpiifort) are available through the "intel-mpi" module.

$module load intel-mpi

These modules must be loaded within LSF scripts when using binaries compiled with the Intel compilers.

The Infiniband network and core placement can be tuned with Intel MPI to improve performance. We suggest that when using Intel MPI, you define the following environmental variables in your LSF script:


export I_MPI_PIN_DOMAIN=auto
export I_MPI_PIN_PROCESSOR_LIST=0-15
export I_MPI_FABRICS=shm:dapl
export I_MPI_DAPL_DIRECT_COPY_THRESHOLD=65536
export I_MPI_DAPL_UD=enable
export I_MPI_DAPL_RDMA_MIXED=enable
export I_MPI_DYNAMIC_CONNECTION=0
export I_MPI_HYDRA_IFACE=ib0
export I_MPI_HYDRA_BOOTSTRAP=lsf

The above assumes you are using 16 cores per node; otherwise you would adjust the I_MPI_PIN_PROCESSOR_LIST variable appropriately.


NVIDIA Compilers (NVCC + CUDA)

To gain access to the NVIDIA CUDA compiler nvcc, load the cuda module:


$module load cuda

To use CUDA within an LSF job, you will need to include a module load cuda line in your LSF script.


OpenMPI Compilers

The OpenMPI compilers (mpicc, mpicxx) are available through the "openmpi" module.

$module load openmpi

This module must be loaded within LSF scripts when using binaries compiled with the OpenMPI compilers.

We suggest that when using OpenMPI, you define the following environment variables in your LSF script:

export I_MPI_PIN_DOMAIN=auto
export I_MPI_PIN_PROCESSOR_LIST=0-15
export I_MPI_DAPL_DIRECT_COPY_THRESHOLD=65536
export I_MPI_DAPL_UD=enable
export I_MPI_DAPL_RDMA_MIXED=enable
export I_MPI_DYNAMIC_CONNECTION=0
export I_MPI_HYDRA_BOOTSTRAP=lsf
export OMPI_MCA_btl=self,openib

The last line is roughly equivalent to this as an alternative:
mpirun -np 256 --mca btl self,openib ./mpi_hello_world > output.txt