serghei notes

SERGHEI is developed at following institutions:

• Forschungszentrum Jülich
• Universidad de Zaragoza
• Technische Universität Braunschweig
• Tongji University (同济大学)

SERGHEI is being used in following research projects:

• ADAPTEX: Dynamic adaptive grids (SCALEXA, BMBF, Germany)
• Eulerian/Lagrangian modeling of debris transport for flash flood prediction (UNIZAR, Spain, and UPPA, France)
• High Performance Computing Tools for Smart Agriculture and Forestry: soil loss (UNIZAR, Spain)
• PREDICTHIA: Predictive computational tools and optimization through Artificial Intelligence for hydro-morphodynamic risks and Climate Change scenarios (Spanish Ministry of Science, Spain)

The shallow water equations-based overland flow module is published by Caviedes-Voullième et al. (2023)°.

SERGHEI-GW is the subsurface flow module of SERGHEI; HLRN is the supercomputing cluster of Norddeutscher Verbund für Hoch- und Höchstleistungsrechnen.

First, we download SERGHEI and its dependencies. SERGHEI relies on kokkos and ParallelNetCDF libraries.

git clone -b subsurface_solver https://gitlab.com/serghei-model/serghei.git
git clone https://github.com/kokkos/kokkos.git
wget https://parallel-netcdf.github.io/Release/pnetcdf-1.13.0.tar.gz

SERGHEI-GW additionally relies the kokkos-kernels library.

git clone https://github.com/kokkos/kokkos-kernels.git

We need to build kokkos first. Navigate into the kokkos folder and create a folder build, then navigate into it. From within the build folder, run

cmake ../ -DCMAKE_CXX_COMPILER=g++ -DCMAKE_INSTALL_PREFIX=./ -DKokkos_ENABLE_OPENMP=On
cmake --build .

which should build kokkos.

We need to build PnetCDF. Navigate into the PnetCDF folder and create a a folder build, then navigate into it.

../configure --prefix=$HOME/sw-local/pnetcdf-1.13.0/build
make
make install

which should build PnetCDF into the build folder.

Zhi Li, Tongji University, writes on 4th of April 2024 (personal communication):

You could follow these steps:

1. Create 2 folders named (for example) kokkoskernels-build and kokkoskernels-install
2. cd into kokkoskernels-build, then building kokkoskernels with cmake, make sure that -DCMAKE_INSTALL_PREFIX=kokkoskernels-install
3. Still inside kokkoskernels-build, type make install to install kokkoskernels.
4. Finally, when building serghei, use -DKokkosKernels_ROOT=kokkoskernels-install

Navigate into the kokkos-kernels folder and create a folder build and install, then navigate into the build folder. From within the build folder, run

export Kokkos_DIR=$HOME/sw-local/kokkos/build
cmake ../ -DCMAKE_CXX_COMPILER=g++ -DCMAKE_INSTALL_PREFIX=./ \
      -DKokkos_ENABLE_OPENMP=On -DCMAKE_INSTALL_PREFIX=../install
cmake --build .

which should build kokkos-kernels. Now, install it into the install folder through

make install

Zhi Li, Tongji University, writes on 3rd of April 2024 (personal communication):

Currently there are two versions of the CMake input files (which could make people confused):

1. The old CMake file available in the subsurface_solver branch and in the Wiki.
2. The new CMake file I wrote last week, which is available in the cmake branch.

The new CMake file is still under experiment. I believe Daniel is still working on it. For now I recommend to use the old CMake file, in which the pnetcdf path is set in src/CMakeLists.txt as:

target_include_directories(serghei PUBLIC/Users/zli/Codes/PnetCDF/include)
target_link_libraries(serghei PUBLIC/Users/zli/Codes/PnetCDF/lib/libpnetcdf.a)

Here, you can explicitly set the path to the include directory and to the libpnetcdf.a library.

Navigate into the serghei folder. Navigate into src and create a build folder here. Open the file CMakeLists.txt in the folder src. Set the PnetCDF paths as described in the mail by Zhi Li, but in our case:

target_include_directories(serghei PUBLIC /home/usrname/sw-local/pnetcdf-1.13.0/include)
target_link_libraries(serghei PUBLIC /home/usrname/sw-local/pnetcdf-1.13.0/lib/libpnetcdf.a)

At the beginning of src/CMakeLists.txt, add the following lines to suppress annoying warnings:

cmake_minimum_required(VERSION 3.16 FATAL_ERROR)
project(SERGHEI-GW)

Execute

cmake ../ -DCMAKE_CXX_COMPILER=g++ -DEnable_GPU=OFF \
      -DKokkosKernels_ROOT=path-to-kokkoskernels/install
make

which should build SERGHEI-GW into a the folder src/bin.