BioDynaMo Singularity file

Hello everyone,

As some of you may already know, I have been working on creating a singularity file for installing BioDynaMo on high performance computers (HPCs). The reason singularity has been chosen over alternatives such as docker is due to the high usage of singularity on research based HPC systems.

Singularity was created by sylabs and is considered to have fewer security issues than docker. Hence the heavy usage of HPC systems. Extensive documentation can be found on Sylabs own webiste but a good place to start is the introduction Introduction to Singularity — Singularity container 3.5 documentation.

I have created a singularity file which will allow users to created an image of BioDynaMo. This image can easily be secure copied from your local machine to any HPC of choice for which you have suitable clearance.

I am currentlly working on adding a new branch to BioDynaMo on GitHub which will use some additional tests alongside existing automated tests to ensure that the Singularity file does not begin to fail in future.

Please find the singularity document below. Feedback is much appreciated. I am also happy to answer usage questions for those interested in using singularity for exporting BioDynaMo to HPCs.

If anyone would like me to put a brief example of how to run the script to create the file please let me know.

~ J

Bootstrap:docker
From:ubuntu:18.04

%environment


%post -c /bin/bash

  apt-get -y update &&
  apt-get -y install python &&
  apt-get -y install python3 &&
  apt-get -y install git &&
  apt-get -y install curl &&
  apt-get -y install make &&
  apt-get -y install gcc &&
  apt-get -y install g++ &&
  apt-get -y install wget &&
  apt-get -y install wamerican && 
  apt-get -y install libffi-dev &&
  apt-get -y install libncurses5-dev &&
  apt-get -y install zlib1g &&
  apt-get -y install zlib1g-dev &&
  apt-get -y install bzip2 &&
  apt-get -y install zlib1g-dev && 
  apt-get -y install aptitude &&
  aptitude -y install libreadline-dev &&
  apt-get -y install libssl-dev &&
  apt-get install libsqlite3-dev &&
  apt-get install apt-utils &&
  apt-get -y install openmpi-bin &&
  apt-get -y install libopenmpi-dev &&
  apt-get -y install libxkbcommon-x11-dev &&
  apt-get -y install bsdmainutils &&
  apt-get -y install clang &&
  apt-get -y install clang-format &&
  apt-get -y install clang-tidy &&
  apt-get -y install doxygen &&
  apt-get -y install graphviz &&
  apt-get -y install libxml2-dev &&
  apt-get -y install llvm-7 &&
  apt-get -y install llvm-7-dev &&
  apt-get -y install llvm-7-runtime &&
  apt-get -y install valgrind &&
  apt-get -y install libgsl-dev &&
  apt-get -y install freeglut3-dev &&
  apt-get -y install libbz2-dev &&
  apt-get -y install libnuma-dev &&
  apt-get -y install libomp5 &&
  apt-get -y install libomp-dev &&
  apt-get -y install libopenmpi-dev &&
  apt-get -y install libpthread-stubs0-dev &&
  apt-get -y install zlib1g-dev &&
  apt-get -y install libbz2-dev &&
  apt-get -y install libffi-dev &&
  apt-get -y install liblzma-dev &&
  apt-get -y install libreadline-dev &&
  apt-get -y install libsqlite3-dev &&
  apt-get -y install libssl-dev &&
  apt-get -y install python-openssl &&
  apt-get -y install tk-dev &&
  apt-get -y install xz-utils &&
  apt-get -y install zlib1g-dev &&
  apt-get -y install sudo &&
  apt-get -y install software-properties-common
  

  wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
  bash Miniconda3-latest-Linux-x86_64.sh -b -f -p /miniconda3/
  rm Miniconda3-latest-Linux-x86_64.sh

  export PATH="/miniconda3/bin:$PATH"
  conda install -y -c conda-forge pip numpy 
  conda update -y --all
  apt -y install git
  apt -y install curl
  rm -rf ~/.pyenv
  curl https://pyenv.run | bash
  export PATH="$HOME/.pyenv/bin:$PATH"
  eval "$(pyenv init -)"

  export PYENV_ROOT=/opt/pyenv
  export PATH="/opt/pyenv/bin:$PATH" 
  curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
  PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv -y install 3.9.1
  export PATH=/opt/pyenv/versions/3.9.1/bin:$PATH
  export PATH=/opt/pyenv/versions/3.9.1/bin/:$PATH

  wget https://bootstrap.pypa.io/get-pip.py
  python3 get-pip.py
  python3 -m pip install -U pip
  python3 -m pip install --upgrade pip
  python3 -m pip install numpy
  apt-get install -y python3-pip
  apt-get install -y freeglut3-dev valgrind

  pip install cmake --upgrade
  pyenv global 3.9.1
  
  apt-get -y dist-upgrade   

  git clone https://github.com/BioDynaMo/biodynamo.git
  cd biodynamo 
  export SILENT_INSTALL=1
  ./prerequisites.sh all 
  mkdir build 
  cd build 
  cmake ..  
  make -j8
  cd build


%runscript


1 Like

Hi everyone,

please, find attached an updated version of the Singularity configuration file which I used to install the BioDynaMo container on an HPC running CentOS 8. Moreover, I included below a small bash script that I put in the .bashrc file on my HPC home folder. It can be used to run specific commands, set/unset environment variables, etc…, when a Singularity shell is running.

Singularity configuration file:

Bootstrap:docker
From:ubuntu:18.04

%environment

%post -c /bin/bash

  export DEBIAN_FRONTEND=noninteractive 
  export TZ=Europe/Berlin

  apt-get -y update &&
  apt-get -y install python &&
  apt-get -y install python3 &&
  apt-get -y install git &&
  apt-get -y install curl &&
  apt-get -y install make &&
  apt-get -y install gcc &&
  apt-get -y install g++ &&
  apt-get -y install wget &&
  apt-get -y install wamerican && 
  apt-get -y install libffi-dev &&
  apt-get -y install libncurses5-dev &&
  apt-get -y install zlib1g &&
  apt-get -y install zlib1g-dev &&
  apt-get -y install bzip2 &&
  apt-get -y install zlib1g-dev && 
  apt-get -y install aptitude &&
  aptitude -y install libreadline-dev &&
  apt-get -y install libssl-dev &&
  apt-get -y install libsqlite3-dev &&
  apt-get -y install apt-utils &&
  apt-get -y install openmpi-bin &&
  apt-get -y install libopenmpi-dev &&
  apt-get -y install libxkbcommon-x11-dev &&
  apt-get -y install bsdmainutils &&
  apt-get -y install clang &&
  apt-get -y install clang-format &&
  apt-get -y install clang-tidy &&
  apt-get -y install doxygen &&
  apt-get -y install graphviz &&
  apt-get -y install libxml2-dev &&
  apt-get -y install llvm-7 &&
  apt-get -y install llvm-7-dev &&
  apt-get -y install llvm-7-runtime &&
  apt-get -y install valgrind &&
  apt-get -y install libgsl-dev &&
  apt-get -y install freeglut3-dev &&
  apt-get -y install libbz2-dev &&
  apt-get -y install libnuma-dev &&
  apt-get -y install libomp5 &&
  apt-get -y install libomp-dev &&
  apt-get -y install libopenmpi-dev &&
  apt-get -y install libpthread-stubs0-dev &&
  apt-get -y install zlib1g-dev &&
  apt-get -y install libbz2-dev &&
  apt-get -y install libffi-dev &&
  apt-get -y install liblzma-dev &&
  apt-get -y install libreadline-dev &&
  apt-get -y install libsqlite3-dev &&
  apt-get -y install libssl-dev &&
  apt-get -y install python-openssl &&
  apt-get -y install tk-dev &&
  apt-get -y install xz-utils &&
  apt-get -y install zlib1g-dev &&
  apt-get -y install sudo &&
  apt-get -y install software-properties-common

  apt-get -y install locales locales-all
  dpkg-reconfigure locales
  locale-gen
  
  wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
  bash Miniconda3-latest-Linux-x86_64.sh -b -f -p /miniconda3/
  rm Miniconda3-latest-Linux-x86_64.sh

  export PATH="/miniconda3/bin:$PATH"
  conda install -y -c conda-forge pip numpy 
  conda update -y --all
  apt -y install git
  apt -y install curl
  sudo  rm -rf ~/.pyenv
  curl https://pyenv.run | bash
  export PATH="$HOME/.pyenv/bin:$PATH"
  eval "$(pyenv init --path)"
  eval "$(pyenv virtualenv-init -)"
  eval "$(pyenv init -)"

  export PYENV_ROOT="/opt/pyenv"
  export PATH="/opt/pyenv/bin:$PATH" 
  curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
  PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.9.1
  export PATH="/opt/pyenv/versions/3.9.1/bin:$PATH"
  export PATH="/opt/pyenv/versions/3.9.1/bin/:$PATH"

  wget https://bootstrap.pypa.io/get-pip.py
  python3 get-pip.py
  python3 -m pip install -U pip
  python3 -m pip install --upgrade pip
  python3 -m pip install numpy
  apt-get install -y python3-pip
  apt-get install -y freeglut3-dev valgrind

  pip install cmake --upgrade
  pyenv global 3.9.1
  
  apt-get -y dist-upgrade   

  git clone https://github.com/BioDynaMo/biodynamo.git
  cd biodynamo 
  export SILENT_INSTALL=1
  ./prerequisites.sh all
  mkdir build 
  cd build 
  cmake ..  
  make -j8

%runscript

Bash script (example):

# Check if we're running a Singularity shell
if [ -n "$SINGULARITY_CONTAINER" ]; then
    echo -e "You are running a Singularity container"

    source /biodynamo/build/third_party/root/bin/thisroot.sh
    export Qt5_DIR="/biodynamo/build/third_party/qt/lib/cmake/Qt5"
    export ParaView_DIR="/biodynamo/build/third_party/paraview/lib/cmake/paraview-5.9"


    source /biodynamo/build/bin/thisbdm.sh
    return
fi