serghei notes

Table of Contents

> index / notes : serghei

About

SERGHEI is a multi-dimensional, multi-domain, and multi-physics model framework for environmental and landscape simulation, designed with an outlook towards Earth System Modelling. It aims to provide a modelling environment for hydrodynamics, ecohydrology, morphodynamics, and, most importantly, interactions and feedbacks among such processes at different levels of complexity and across spatiotemporal scales. At the core of SERGHEI's technical innovation is its high-performance implementation, built from scratch on the Kokkos portability layer. Consequently, SERGHEI achieves performance-portability from personal computers to top supercomputing systems, including GPU-based and heterogeneous systems.

Building SERGHEI-GW on HLRN

Terminology

SERGHEI-GW is the subsurface flow module of SERGHEI; HLRN is the supercomputing cluster of Norddeutscher Verbund für Hoch- und Höchstleistungsrechnen.

Getting the code

First, we download SERGHEI and its dependencies. SERGHEI relies on kokkos and ParallelNetCDF libraries.

git clone -b subsurface_solver https://gitlab.com/serghei-model/serghei.git
git clone https://github.com/kokkos/kokkos.git
wget https://parallel-netcdf.github.io/Release/pnetcdf-1.13.0.tar.gz

SERGHEI-GW additionally relies the kokkos-kernels library.

git clone https://github.com/kokkos/kokkos-kernels.git

Setting up the environment

module load gcc/9.3.0
module load openmpi/gcc.9/4.1.4
module load cmake/3.25.3
module load autoconf/2.72

Building kokkos

We need to build kokkos first. Navigate into the kokkos folder and create a folder build, then navigate into it. From within the build folder, run

cmake ../ -DCMAKE_CXX_COMPILER=g++ -DCMAKE_INSTALL_PREFIX=./ -DKokkos_ENABLE_OPENMP=On
cmake --build .

which should build kokkos.

Building PnetCDF

We need to build PnetCDF. Navigate into the PnetCDF folder and create a a folder build, then navigate into it.

../configure --prefix=$HOME/sw-local/pnetcdf-1.13.0/build
make
make install

which should build PnetCDF into the build folder.

Building kokkos-kernels

Zhi Li, Tongji University, writes on 4th of April 2024 (personal communication):

You could follow these steps:

  1. Create 2 folders named (for example) kokkoskernels-build and kokkoskernels-install
  2. cd into kokkoskernels-build, then building kokkoskernels with cmake, make sure that -DCMAKE_INSTALL_PREFIX=kokkoskernels-install
  3. Still inside kokkoskernels-build, type make install to install kokkoskernels.
  4. Finally, when building serghei, use -DKokkosKernels_ROOT=kokkoskernels-install

Navigate into the kokkos-kernels folder and create a folder build and install, then navigate into the build folder. From within the build folder, run

export Kokkos_DIR=$HOME/sw-local/kokkos/build
cmake ../ -DCMAKE_CXX_COMPILER=g++ -DCMAKE_INSTALL_PREFIX=./ \
      -DKokkos_ENABLE_OPENMP=On -DCMAKE_INSTALL_PREFIX=../install
cmake --build .

which should build kokkos-kernels. Now, install it into the install folder through

make install

Building SERGHEI-GW

Zhi Li, Tongji University, writes on 3rd of April 2024 (personal communication):

Currently there are two versions of the CMake input files (which could make people confused):

  1. The old CMake file available in the subsurface_solver branch and in the Wiki.
  2. The new CMake file I wrote last week, which is available in the cmake branch.

The new CMake file is still under experiment. I believe Daniel is still working on it. For now I recommend to use the old CMake file, in which the pnetcdf path is set in src/CMakeLists.txt as:

target_include_directories(serghei PUBLIC/Users/zli/Codes/PnetCDF/include)
target_link_libraries(serghei PUBLIC/Users/zli/Codes/PnetCDF/lib/libpnetcdf.a)

Here, you can explicitly set the path to the include directory and to the libpnetcdf.a library.

Navigate into the serghei folder. Navigate into src and create a build folder here. Open the file CMakeLists.txt in the folder src. Set the PnetCDF paths as described in the mail by Zhi Li, but in our case:

target_include_directories(serghei PUBLIC /home/usrname/sw-local/pnetcdf-1.13.0/include)
target_link_libraries(serghei PUBLIC /home/usrname/sw-local/pnetcdf-1.13.0/lib/libpnetcdf.a)

At the beginning of src/CMakeLists.txt, add the following lines to suppress annoying warnings:

cmake_minimum_required(VERSION 3.16 FATAL_ERROR)
project(SERGHEI-GW)

Execute

cmake ../ -DCMAKE_CXX_COMPILER=g++ -DEnable_GPU=OFF \
      -DKokkosKernels_ROOT=path-to-kokkoskernels/install
make

which should build SERGHEI-GW into a the folder src/bin.


Impressum

Author: ilhan özgen xian

Created: 2024-04-05 Fri 17:44

Validate