<p><aclass="reference internal"href="Section_accelerate.html"><em>Return to Section accelerate overview</em></a></p>
<divclass="section"id="user-omp-package">
<h1>5.USER-OMP package<aclass="headerlink"href="#user-omp-package"title="Permalink to this headline">¶</a></h1>
<p>The USER-OMP package was developed by Axel Kohlmeyer at Temple
University. It provides multi-threaded versions of most pair styles,
nearly all bonded styles (bond, angle, dihedral, improper), several
Kspace styles, and a few fix styles. The package currently
uses the OpenMP interface for multi-threading.</p>
<p>Here is a quick overview of how to use the USER-OMP package:</p>
<ulclass="simple">
<li>use the -fopenmp flag for compiling and linking in your Makefile.machine</li>
<li>include the USER-OMP package and build LAMMPS</li>
<li>use the mpirun command to set the number of MPI tasks/node</li>
<li>specify how many threads per MPI task to use</li>
<li>use USER-OMP styles in your input script</li>
</ul>
<p>The latter two steps can be done using the “-pk omp” and “-sf omp”
<aclass="reference internal"href="Section_start.html#start-7"><span>command-line switches</span></a> respectively. Or
the effect of the “-pk” or “-sf” switches can be duplicated by adding
the <aclass="reference internal"href="package.html"><em>package omp</em></a> or <aclass="reference internal"href="suffix.html"><em>suffix omp</em></a> commands
<p>You must also use the <aclass="reference internal"href="package.html"><em>package omp</em></a> command to enable the
USER-OMP package, unless the “-sf omp” or “-pk omp”<aclass="reference internal"href="Section_start.html#start-7"><span>command-line switches</span></a> were used. It specifies how many
threads per MPI task to use, as well as other options. Its doc page
explains how to set the number of threads via an environment variable
if desired.</p>
<p><strong>Speed-ups to expect:</strong></p>
<p>Depending on which styles are accelerated, you should look for a
reduction in the “Pair time”, “Bond time”, “KSpace time”, and “Loop
time” values printed at the end of a run.</p>
<p>You may see a small performance advantage (5 to 20%) when running a
USER-OMP style (in serial or parallel) with a single thread per MPI
task, versus running standard LAMMPS with its standard
(un-accelerated) styles (in serial or all-MPI parallelization with 1
task/core). This is because many of the USER-OMP styles contain
similar optimizations to those used in the OPT package, as described
above.</p>
<p>With multiple threads/task, the optimal choice of MPI tasks/node and
OpenMP threads/task can vary a lot and should always be tested via
benchmark runs for a specific simulation running on a specific
machine, paying attention to guidelines discussed in the next
sub-section.</p>
<p>A description of the multi-threading strategy used in the USER-OMP
package and some performance examples are <aclass="reference external"href="http://sites.google.com/site/akohlmey/software/lammps-icms/lammps-icms-tms2011-talk.pdf?attredirects=0&d=1">presented here</a></p>
<p><strong>Guidelines for best performance:</strong></p>
<p>For many problems on current generation CPUs, running the USER-OMP
package with a single thread/task is faster than running with multiple
threads/task. This is because the MPI parallelization in LAMMPS is
often more efficient than multi-threading as implemented in the
USER-OMP package. The parallel efficiency (in a threaded sense) also
varies for different USER-OMP styles.</p>
<p>Using multiple threads/task can be more effective under the following
circumstances:</p>
<ulclass="simple">
<li>Individual compute nodes have a significant number of CPU cores but
the CPU itself has limited memory bandwidth, e.g. for Intel Xeon 53xx
(Clovertown) and 54xx (Harpertown) quad core processors. Running one
MPI task per CPU core will result in significant performance
degradation, so that running with 4 or even only 2 MPI tasks per node
is faster. Running in hybrid MPI+OpenMP mode will reduce the
inter-node communication bandwidth contention in the same way, but
offers an additional speedup by utilizing the otherwise idle CPU
cores.</li>
<li>The interconnect used for MPI communication does not provide
sufficient bandwidth for a large number of MPI tasks per node. For
example, this applies to running over gigabit ethernet or on Cray XT4
or XT5 series supercomputers. As in the aforementioned case, this
effect worsens when using an increasing number of nodes.</li>
<li>The system has a spatially inhomogeneous particle density which does
not map well to the <aclass="reference internal"href="processors.html"><em>domain decomposition scheme</em></a> or
<aclass="reference internal"href="balance.html"><em>load-balancing</em></a> options that LAMMPS provides. This is
because multi-threading achives parallelism over the number of
particles, not via their distribution in space.</li>
<li>A machine is being used in “capability mode”, i.e. near the point
where MPI parallelism is maxed out. For example, this can happen when
using the <aclass="reference internal"href="kspace_style.html"><em>PPPM solver</em></a> for long-range
electrostatics on large numbers of nodes. The scaling of the KSpace
calculation (see the <aclass="reference internal"href="kspace_style.html"><em>kspace_style</em></a> command) becomes
the performance-limiting factor. Using multi-threading allows less
MPI tasks to be invoked and can speed-up the long-range solver, while
increasing overall performance by parallelizing the pairwise and
bonded calculations via OpenMP. Likewise additional speedup can be
sometimes be achived by increasing the length of the Coulombic cutoff
and thus reducing the work done by the long-range solver. Using the
<aclass="reference internal"href="run_style.html"><em>run_style verlet/split</em></a> command, which is compatible
with the USER-OMP package, is an alternative way to reduce the number
of MPI tasks assigned to the KSpace calculation.</li>
</ul>
<p>Additional performance tips are as follows:</p>
<ulclass="simple">
<li>The best parallel efficiency from <em>omp</em> styles is typically achieved
when there is at least one MPI task per physical processor,
i.e. socket or die.</li>
<li>It is usually most efficient to restrict threading to a single
socket, i.e. use one or more MPI task per socket.</li>
<li>Several current MPI implementation by default use a processor affinity
setting that restricts each MPI task to a single CPU core. Using
multi-threading in this mode will force the threads to share that core
and thus is likely to be counterproductive. Instead, binding MPI
tasks to a (multi-core) socket, should solve this issue.</li>
</ul>
<divclass="section"id="restrictions">
<h2>Restrictions<aclass="headerlink"href="#restrictions"title="Permalink to this headline">¶</a></h2>
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.