Finite Element Large Eddy Simulation

Principal Investigator

Hugh Blackburn

Building, Construction and Engineering,

CSIRO Division

The project is concerned with developing techniques for large eddy simulation
of bluff-body flows and
flows in complex geometries. These flows have a wide range of application
in mechanical, civil and offshore engineering, and at present the only available
approaches for engineering investigations are Reynolds-averaged simulations
or physical experiments. The Reynolds-averaged approach is very often inaccurate
even in modelling the time-mean flow, and fluctuating quantities are very
poorly predicted. Experimental methods are becoming increasingly expensive,
while computer simulations, using large eddy simulation, are just becoming
feasible for these kinds of problems. Finite element based methods required
for complex geometries have not yet been extensively employed for large
eddy simulation, due to the algorithmic difficulties encountered on typical
supercomputer architectures, and the complexity of sub-grid scale stress
models suitable for arbitrary geometries.

Co-Investigators

Dean Niclasen

Mechanical Engineering,

Monash University

Projects

g89 - VPP

What are the results to date and the future of
this work?

Following the initial application both investigators attended the ANUSF
course on vectorization and tuning for the VPP. As a result of information
and assistance received there and subsequently, porting and vectorization
of existing low-order finite element and high order (spectral) finite element
unsteady Navier­Stokes codes was carried out. At that time the low-order
code had 30% vectorization on test problems, while the spectral element
code achieved 50% for typical 3D jobs. Further vectorization of the spectral
element code was somewhat impeded by absence of a native C++ compiler ­
runtime information provided by system sampling utilities reports time spent
in top-level routines only, and debugging tools fail to work ­ C++ is
used only for the top level of the code: compute-intensive parts are carried
by FORTRAN & C routines.

It was then planned to use MPI as a basis for parallelization of the
spectral element code, since the spatial discretization employed (2DxFourier)
is well suited to the coarse-grained architecture of the VPP. This work
has now been successfully carried out, together with further vectorization
of gradient operator kernels to improve performance in computation of nonlinear
terms and PCG solution methods. Coding of eddy-viscosity type LES has also
been incorporated and is undergoing testing.

Appendix B -

The algorithm employed has proven to be quite efficient in concurrent
execution, as anticipated: solutions for each Fourier mode can (if possible)
execute on a separate processor, with interprocessor communication required
only during computation of nonlinear terms in the Navier­Stokes equations.

Sample speedup comparisons for a medium-sized (120000-node/100MB) 3D
problem, based on elapsed times:

#procs

time

speedup

efficiency

CPUtime

%Vector

1

972

1

1

880

46

2

540

1.8

.90

873

40

4

340

2.9

.71

1098

32

Overall performance for these runs was approximately 2.5 times that of
a 4-processor 250Mhz DEC Alphaserver. Since the comparison were carried
out, per-node performance has been significantly enhanced by vectorization
of 2D field-gradient operators which then consumed of the order 50% of execution
time: typical performance figures for the kernel routines are now between
500 & 1000 Mflops. Concurrent efficiency is likely to improve as message
sizes increase for larger problems, possibly also with a new memory-transpose
algorithm if required.

In terms of algorithm development, it is planned to develop a new inter-processor
memory transpose based on recursive doubling in order to remove potential
bottlenecks, if this proves to be a significant problem as the number of
PEs employed increases.

Futher grants of time will be consumed mainly in parallel production
runs, initially to investigate DNS & LES of fluid-structure interaction
problems.

What computational techniques are used?

The method is a spectral element/Fourier incompressible flow solver,
which employs a time-splitting algorithm based on backwards differentiation.
Two-dimensional planes of data are decomposed into high-order ("spectral")
elements which employ orthogonal functions, such as Legendre or Chebyshev
polynomials, as basis functions. Extension to three spatial dimensions is
achieved using Fourier expansions, which also allows most of the operations
in each timestep to be carried out concurrently. Nonlinear terms in the
Navier-Stokes equations are are computed pseudo-spectrally in physical space,
and each of the products requires an inter-processor data transpose of velocity
field components. The remaining two-dimensional Poisson/Helmholtz problems
which comprise the time-split are carried out concurrently. Owing to the
use of orthogonal basis functions in all directions, spatial converegence
of the method depends exponentially on the number of grid points used.