HPC support for reservoir simulation

HPC support for reservoir simulation  [2 PhD Projects]

Physics-faithful reservoir simulations are vital for the Oil & Gas industry, both for effective extraction of oil/gas and also for CO2 sequestration. The latter is a mechanism for reducing the amount of carbon dioxide in the atmosphere. To achieve physical faithfulness of the simulations, it will require a combination of complicated mathematical models, advanced numerical algorithms and high/extremely-high computational resolutions. These simulations, either executed independently or in an “ensemble” setting, will pose a formidable computational challenge, thus the need for efficiently using supercomputers.

Challenges

Using supercomputers for efficient numerical simulations, such as reservoir simulations, is a highly non-trivial topic. Achieving high-performance computing (HPC) requires handling at least three challenges as follows:

    • The overall computational work must be appropriately partitioned into pieces, which is the first step for using supercomputers because they are built upon the principle of parallel processing.
    • The chosen numerical algorithm for a reservoir simulation to run on a supercomputer must be parallelizable and, moreover, scalable with an increasing number of processing units.
    • The software implementation must be able to effectively utilize the capabilities of the processing units and the interconnect between them on a supercomputer.

For reservoir simulations, in particular, the above three challenges are intermingled together. First, the underground reservoirs are in general of irregular 3D shapes, thus unstructured computational meshes must be adopted to resolve the geometric irregularities. Second, partitioning unstructured meshes alone (without considering the actual computations to be executed on top of those) is already a demanding task. The added difficulty for reservoir simulations is that advanced numerical strategies are required to solve the involved mathematical equations. These complex algorithms, when parallelized, may require a different goal for the mesh partitioning task. For example, the mesh entities that are strongly coupled numerically will benefit from being assigned to the same partition, instead of being divided among multiple partitions. Therefore, the mesh partitioning task related to parallel reservoir simulations must balance between the standard partitioning criteria and the resulting effectiveness of the parallelized numerical algorithm. To make the situation even more challenging, the actual utilization of a parallel computer for reservoir simulations is also dependent on the hardware details, as well as the specific reservoir case at hand.

 

The goals of this SIRIUS project are therefore:

    1. To investigate the necessary balance between the standard criteria for partitioning unstructured meshes and the reservoir-specific criteria that are important for the parallelized numerical algorithm. Based on this investigation, we want to extend the standard mesh partitioning problem;
    2. To ensure a high level of performance of parallel reservoir simulations by avoiding non-contributing computations that can arise specifically from parallelization;
    3. To obtain a quantitative understanding of the communication overhead that is associated with any parallel implementation of a reservoir simulation, for the possibility of a better mapping of the different pieces to the actual hardware processing units;
    4. To avoid unnecessary waste of the computational effort by automatically identifying good choices of the inherent parameters of the numerical algorithm, so that accuracy is maintained while the numerical algorithm progresses faster than when less optimal parameters are used.

Our Approach

During the mesh partitioning task, an unstructured computational mesh is first translated to a corresponding graph (so that existing graph partitioning software can be used). Inside this step, we have developed two new strategies to assign different weights to the edges in the resulting graph. More specifically, instead of the standard approach to assigning each edge with an equal weight, we strategically assign each edge with a so-called transmissibility value, or a scaled value based on it. These non-uniform edge weights are motivated by the different levels of numerical interaction between the mesh entities (each edge in the graph corresponds to the numerical interaction between two mesh entities). An example of the different partitioning results can be found in Figure 1. The benefit is that the mesh entities having a strong interaction between themselves are more likely to be kept on the same partition, instead of being divided across partitions. As a consequence, the parallelized numerical algorithm can benefit from such heterogeneity-aware mesh partitioning.

After a computational mesh is partitioned, each partition needs to be expanded with a layer of “ghost cells”, which are needed to facilitate the necessary data exchanges between the neighboring partitions. It is noted that computations on these ghost cells are unnecessary, because the values are to be provided by the neighbors. In case software code for standard sequential (non-parallel) computing is blindly used, wasteful computations will take place on these ghost cells. To avoid the non-contributing computations, we have re-ordered the cells on each partition such that the ghost cells are placed together at the end. Combined with a careful modification of the software code, we are thus able to eliminate the non-contributing computations. The benefit for the resulting parallel reservoir computation can be illustrated by Figure 2. The corresponding software modifications, together with the heterogeneity-aware mesh partitioning strategies discussed above, have already been incorporated into the OPM software framework (https://opm-project.org), so that the entire oil & gas research and industry community can use them in real-world reservoir modeling work.

Figure 1. Three mesh partitioning results, where the left plot arises from the standard “equal-edge-weight” approach, whereas the middle and right plots arise from assigning heterogeneous edge weights. The middle partitioning result gives the best parallel performance of a typical resulting reservoir simulation.

Figure 2. The effect of avoiding non-contributing computations for three basic computing kernels needed in a parallel reservoir simulation (top left and bottom left and right plots). The orange bars (marked with “GL”) show the time usage after eliminating the non-contributing computations, while the original time usage is shown by the blue bars.

Data exchanges via communication between the neighboring partitions are needed to ensure that the processing units on a parallel computer can join force in a parallel reservoir simulation. However, the time used by the inter-neighbor data exchanges can be substantial, especially when the different partitions have different amounts of communication. This heterogeneity with respect to the per-partition communication cost can be further magnified by the heterogeneity of the interconnect typically found on modern parallel computers. To obtain a full understanding of the communication-cost heterogeneity, we have developed a novel modeling approach based on a fair competition among the partitions, where the partitions with lower communication volumes can finish before the more heavily communication-loaded partitions. The result is a new, per-partition communication prediction model can be used to pinpoint communication “hotspots” and/or to propose an improved mapping of the different partitions to the different processing units of a parallel computer. The per-partition prediction accuracy can be illustrated a concrete example that involves 128 partitions, as shown in Figure 3.

Any advanced numerical algorithm used for reservoir simulations typically involves many internal parameters that can have an impact on its numerical effectiveness. It is however far from straightforward to choose ideal values for these internal parameters, because they may be dependent on the concrete reservoir case at hand, sometimes also dependent on the actual parallel computer used and/or the number of processing units used. To achieve further time saving of reservoir simulations, we have developed a new search strategy that is based on stochastic sampling while simultaneously being constrained with some rule-based guidance. At the same time, the automated search process will guard itself against inappropriate parameter choices such as those leading to numerical inaccuracy. An example result of the automated parameter search is shown in Figure 4.

Figure 3. An example result of using a new prediction model of the per-partition communication cost, where blue bars indicate the actual time usage and the green bars indicate the model prediction.

Figure 4. The result of an automated parameter search process, where five progressively identified paramemter sets lead to increasingly more efficient parallel reservoir simulations (reflected in fewer iterations needed). The different colors correspond to different number of processing units used on a parallel computer.

Results

Documentation

The approaches adopted for heterogeneity-aware mesh partitioning and eliminating non-contributing computations are documented in the publications below. The approaches used for detailed per-partition communication modeling and automated parameter search will be published later in separate research papers.

Source Code

The OPM software framework (https://opm-project.org) has incorporated the modifications in the implementation needed to facilitate heterogeneity-aware mesh partitioning and eliminating non-contributing computations. These software improvements are thus available for the entire research and industrial community of oil & gas.

Publications​

Thune,X. Caiand A. B. Rustad. On the impact of heterogeneity-aware mesh partitioning and non-contributing computation removal on parallel reservoir simulations. Journal of Mathematics in Industry, vol. 11, 2021. DOI: 10.1186/s13362-021-00108-5.

F. Rasmussen,T. H. Sandve,K. Bao, A. Lauser, J. Hove, B. Skaflestad, R. Klöfkorn, M. Blatt, A. B. Rustad, O. Sævareid, K. A. Lie and A. Thune. The Open Porous Media Flow reservoir simulator. Computers & Mathematics with Applications, vol. 81, pages 159-185, 2021. DOI: 10.1016/j.camwa.2020.05.014.

H. Sandve,A. B. Rustad,A. Thune, B. Nazarian, S. Gasda and A. F. Rasmussen. Simulators for the Gigaton Storage Challenge. A Benchmark Study on the Regional Smeaheia Model.   EAGE GeoTech 2022 Sixth EAGE Workshop on CO2 Geological Storage, 2022, DOI: 10.3997/2214-4609.20224033.

Team

SIRIUS:

PhD candidates: Andreas Thune (Simula/UiO), Erik Sæternes (Simula/UiO)

Supervisors: Xing Cai (Simula/UiO), Tor Skeie (UiO/Simula), Alf Birger Rustad (Equinor), Ernst Gunnar Gran (NTNU/Simula)

Partners

Acknowledgements

SIRIUS Centre for Scalable Data Access (Research Council of Norway, project 237889) has provided the base funding for the two PhD candidates. Additional funding has been provided by Equinor and Simula.