DLBFoam: Dynamic load balancing for fast reactive simulations
DLBFoam v1.1 - What's new?
DLBFoam v1.1 introduces a fully analytical chemistry Jacobian via pyJac, and optimized ODE solution routines via LAPACK. Combined with the load balancing features, v1.1 provides up to x250 speed-up compared to standard OpenFOAM chemistry model. If you are interested with using only dynamic load balancing without any third party dependency, please use DLBFoam v1.0.
What is DLBFoam?
DLBFoam is an open-source library for OpenFOAM. It introduces dynamic load balancing and a zonal reference mapping model for fast chemistry calculation in parallel simulations. In addition, it also introduces a fully analytical Jacobian formulation and optimized ODE solution routines for further speed-up.
Why do I need this?
Load imbalance in parallel reactive simulations is an issue that causes very long simulation times in OpenFOAM simulations utilizing finite-rate chemistry.
DLBFoam introduces runtime load balancing through MPI routines to minimize the load imbalance between ranks and gain speed-up. The implementation details can be found in our paper [1]. In addition, the cell-wise chemistry problem is vastly improved by the analytical Jacobian formulation and optimized matrix operations in the ODE solver class. The details for those implementations can be found in our follow-up paper [2].
Prerequisites
- OpenFOAM installation (with correct version)
- LAPACK (Intel-MKL, OpenBLAS or standalone)
- Cmake
- ct2foam (Optional)
Compilation
DLBFoam can be compiled by typing the following command after sourcing appropriate OpenFOAM version and making sure a valid LAPACK installation exists:
./Allwmake --clean --platform
DLBFoam requires LAPACK packages for improved ODE routines (LAPACKE C interface for OPENBLAS and standalone installation). There are three different ways to provide LAPACK for DLBFoam:
-
Intel-MKL: Install/load Intel-MKL on your workstation/cluster. Installation instructions can be found online. If you are on cluster, you can most probably load it by:
module load intel-mklmake sure that the
MKLROOTbash variable is defined by typingecho $MKLROOTto your terminal. This option is the fastest option for Intel processors due to optimization of MKL with Intel hardware. -
OpenBLAS: Install/load OpenBLAS. It may be available on your cluster as a module, similar to Intel-MKL. Make sure that the
OPENBLAS_INSTALL_ROOTbash variable is defined by typingecho $OPENBLAS_INSTALL_ROOT, and define it in yourbashrcif it is not. -
Standalone: A standalone installation may be a good idea if you are on your personal workstation and not on a cluster. Install LAPACKE-dev libraries by:
(sudo) apt-get install liblapacke-dev
Mechanism generation
After the successful compilation, the C subroutines generated by the pyJac for the analytical Jacobian should be compiled and linked to the OpenFOAM case folder.
In this repository, we provide the input files and C subroutines generated by pyJac for GRI-3.0, Yao, and DRM-19 mechanisms. If you want to use one of these mechanisms, no other dependency is required (except a CMake installation).
- Go to the
utilities/folder, and executeAllrunas:
./Allrun -m
where gri30, yao, or drm19. This command will generate a folder called foam_mech, which contains the chemical mechanism input files and a library generated from the files generated by pyJac. You need to put this folder to your simulation case and link it through controlDict, as explained in the next section.
If you want to use a different chemical mechanism, you need to create the thermo input files in a format required by OpenFOAM, as well as analytical Jacobian C subroutines generated by pyJac. This process requires many dependencies, most notably Cantera and pyJac.
We have developed a tool called ct2foam that makes this process a lot easier. Please check ct2foam (and the pyjac2foam utility in it) if you are interested in using a different mechanism.
Usage
Once the compilation is successful, any case running with standard OpenFOAM can be easily converted to use DLBFOAM, following these steps:
- The DLBFoam should be linked to the solver. Add the following to your system/controlDict file:
libs
(
"libchemistryModel_DLB.so"
"libODE_DLB.so"
"$FOAM_CASE/constant/foam_mech/libc_pyjac.so"
);
the first two libraries link the DLBFoam and the optimized LAPACK solvers, while the last library links the C subroutines generated in the previous section for the analytical Jacobian.
- Select chemistry solver as
ode_LAPACKand the method asloadBalanced_pyJacin constant/chemistryProperties:
chemistryType
{
solver ode_LAPACK;
method loadBalanced_pyJac;
}
- Add the loadbalancing subdictionary to the same chemistryProperties file:
loadbalancing
{
active true;
log true;
}
- Set the solver flag under
odeCoeffstoseulex_LAPACKin order to use the optimized ODE solvers:
odeCoeffs
{
solver seulex_LAPACK;
absTol 1e-08;
relTol 1e-05;
}
- (Optional) Set the refmapping as active in chemistryProperties file if you want to use the reference mapping method (you have to add an empty
refmapping{}dict even if you do not use it):
refmapping
{
active true;
mixtureFractionProperties
{
oxidizerMassFractions
{
N2 0.77;
O2 0.23;
}
fuelMassFractions
{
NC12H26 1.0;
}
#include "$FOAM_CASE/constant/foam_mech/thermo.foam"
}
tolerance 1e-4; // mixture fraction tolerance
deltaT 2; // temperature tolerance
}
Reference mapping uses mixture fraction (Z) and maps a reference solution to reference cells satisfying a condition.
The entry above sets the Z=0 and Z=1 conditions from given mass fractions. For each CFD iteration it finds a reference solution where Z (Optional) When deltaT is explicitly set, the mapper also checks the temperature between reference solution and other reference cells and ensures: abs(Tcell-Tref) For a working example, check the tutorials given in tutorials folder. Please submit a GitHub issue if you found a bug in the program. If you need help with the software or have further questions, either open an issue or contact the contributors. If you use our model, please cite the publication describing its implementation, Refs. [1] and [2]. [1] B. Tekgül, P. Peltonen, H. Kahila, O. Kaario, V. Vuorinen, DLBFoam: An open-source dynamic load balancing model for fast reacting flow simulations in OpenFOAM, arXiv preprint arXiv:2011.07978 (2020). [2] I. Morev, B. Tekgül, M. Gadalla, A. Shahanaghi, J. Kannan, S. Karimkashi, O. Kaario, V. Vuorinen, Fast reactive flow simulations using analytical Jacobian and dynamic load balancing in OpenFOAM, arXiv preprint arXiv:2105.12070 (2021).
Contributors
Getting help and reporting bugs
Citation
References
BibTex
@article{tekgul2020dlbfoam,
title={DLBFoam: An open-source dynamic load balancing model for fast reacting flow simulations in OpenFOAM},
author={Tekg{\"u}l, Bulut and Peltonen, Petteri and Kahila, Heikki and Kaario, Ossi and Vuorinen, Ville},
journal={arXiv preprint arXiv:2011.07978},
year={2020}
}
BibTex
@article{morev2021fast,
title={Fast reactive flow simulations using analytical Jacobian and dynamic load balancing in OpenFOAM},
author={Morev, Ilya and Tekg{\"u}l, Bulut and Gadalla, Mahmoud and Shahanaghi, Ali and Kannan, Jeevananthan and Karimkashi, Shervin and Kaario, Ossi and Vuorinen, Ville},
journal={arXiv preprint arXiv:2105.12070},
year={2021}
}
1 Nov 14, 2021
118 Nov 21, 2022
763 Jan 9, 2023
126 Nov 27, 2022
11 Nov 14, 2022
969 Jan 3, 2023
12 Sep 5, 2021
7 Apr 29, 2022
32 Dec 23, 2022
121 Dec 31, 2022
95 Jan 3, 2023
14 Dec 24, 2022
19 Dec 21, 2022
5 Sep 24, 2022
7 Jul 6, 2022
0 Mar 1, 2022
2 Feb 19, 2022
6 Dec 14, 2022
3 Dec 14, 2022