<liclass="toctree-l2"><aclass="reference internal"href="#general-strategies">5.2. General strategies</a></li>
<liclass="toctree-l2"><aclass="reference internal"href="#packages-with-optimized-styles">5.3. Packages with optimized styles</a></li>
<liclass="toctree-l2"><aclass="reference internal"href="#comparison-of-various-accelerator-packages">5.4. Comparison of various accelerator packages</a><ul>
<li>5.4 <aclass="reference internal"href="#acc-4"><span>Comparison of various accelerator packages</span></a></li>
</ul>
<p>The <aclass="reference external"href="http://lammps.sandia.gov/bench.html">Benchmark page</a> of the LAMMPS
web site gives performance results for the various accelerator
packages discussed in Section 5.2, for several of the standard LAMMPS
benchmark problems, as a function of problem size and number of
compute nodes, on different hardware platforms.</p>
<divclass="section"id="measuring-performance">
<spanid="acc-1"></span><h2>5.1. Measuring performance<aclass="headerlink"href="#measuring-performance"title="Permalink to this headline">¶</a></h2>
<p>Before trying to make your simulation run faster, you should
understand how it currently performs and where the bottlenecks are.</p>
<p>The best way to do this is run the your system (actual number of
atoms) for a modest number of timesteps (say 100 steps) on several
different processor counts, including a single processor if possible.
Do this for an equilibrium version of your system, so that the
100-step timings are representative of a much longer run. There is
typically no need to run for 1000s of timesteps to get accurate
timings; you can simply extrapolate from short runs.</p>
<p>For the set of runs, look at the timing data printed to the screen and
log file at the end of each LAMMPS run. <aclass="reference internal"href="Section_start.html#start-8"><span>This section</span></a> of the manual has an overview.</p>
<p>Running on one (or a few processors) should give a good estimate of
the serial performance and what portions of the timestep are taking
the most time. Running the same problem on a few different processor
counts should give an estimate of parallel scalability. I.e. if the
simulation runs 16x faster on 16 processors, its 100% parallel
efficient; if it runs 8x faster on 16 processors, it’s 50% efficient.</p>
<p>The most important data to look at in the timing info is the timing
breakdown and relative percentages. For example, trying different
options for speeding up the long-range solvers will have little impact
if they only consume 10% of the run time. If the pairwise time is
dominating, you may want to look at GPU or OMP versions of the pair
style, as discussed below. Comparing how the percentages change as
you increase the processor count gives you a sense of how different
operations within the timestep are scaling. Note that if you are
running with a Kspace solver, there is additional output on the
breakdown of the Kspace time. For PPPM, this includes the fraction
spent on FFTs, which can be communication intensive.</p>
<p>Another important detail in the timing info are the histograms of
atoms counts and neighbor counts. If these vary widely across
processors, you have a load-imbalance issue. This often results in
inaccurate relative timing data, because processors have to wait when
communication occurs for other processors to catch up. Thus the
reported times for “Communication” or “Other” may be higher than they
really are, due to load-imbalance. If this is an issue, you can
uncomment the MPI_Barrier() lines in src/timer.cpp, and recompile
LAMMPS, to obtain synchronized timings.</p>
<hrclass="docutils"/>
</div>
<divclass="section"id="general-strategies">
<spanid="acc-2"></span><h2>5.2. General strategies<aclass="headerlink"href="#general-strategies"title="Permalink to this headline">¶</a></h2>
<divclass="admonition note">
<pclass="first admonition-title">Note</p>
<pclass="last">this section 5.2 is still a work in progress</p>
</div>
<p>Here is a list of general ideas for improving simulation performance.
Most of them are only applicable to certain models and certain
bottlenecks in the current performance, so let the timing data you
generate be your guide. It is hard, if not impossible, to predict how
much difference these options will make, since it is a function of
problem size, number of processors used, and your machine. There is
no substitute for identifying performance bottlenecks, and trying out
various options.</p>
<ulclass="simple">
<li>rRESPA</li>
<li>2-FFT PPPM</li>
<li>Staggered PPPM</li>
<li>single vs double PPPM</li>
<li>partial charge PPPM</li>
<li>verlet/split run style</li>
<li>processor command for proc layout and numa layout</li>
<li>load-balancing: balance and fix balance</li>
</ul>
<p>2-FFT PPPM, also called <em>analytic differentiation</em> or <em>ad</em> PPPM, uses
2 FFTs instead of the 4 FFTs used by the default <em>ik differentiation</em>
PPPM. However, 2-FFT PPPM also requires a slightly larger mesh size to
achieve the same accuracy as 4-FFT PPPM. For problems where the FFT
cost is the performance bottleneck (typically large problems running
on many processors), 2-FFT PPPM may be faster than 4-FFT PPPM.</p>
<p>Staggered PPPM performs calculations using two different meshes, one
shifted slightly with respect to the other. This can reduce force
aliasing errors and increase the accuracy of the method, but also
doubles the amount of work required. For high relative accuracy, using
staggered PPPM allows one to half the mesh size in each dimension as
compared to regular PPPM, which can give around a 4x speedup in the
kspace time. However, for low relative accuracy, using staggered PPPM
gives little benefit and can be up to 2x slower in the kspace
time. For example, the rhodopsin benchmark was run on a single
processor, and results for kspace time vs. relative accuracy for the
different methods are shown in the figure below. For this system,
staggered PPPM (using ik differentiation) becomes useful when using a
relative accuracy of slightly greater than 1e-5 and above.</p>
<spanid="acc-3"></span><h2>5.3. Packages with optimized styles<aclass="headerlink"href="#packages-with-optimized-styles"title="Permalink to this headline">¶</a></h2>
<p>Accelerated versions of various <aclass="reference internal"href="pair_style.html"><em>pair_style</em></a>,
<aclass="reference internal"href="fix.html"><em>fixes</em></a>, <aclass="reference internal"href="compute.html"><em>computes</em></a>, and other commands have
been added to LAMMPS, which will typically run faster than the
standard non-accelerated versions. Some require appropriate hardware
to be present on your system, e.g. GPUs or Intel Xeon Phi
coprocessors.</p>
<p>All of these commands are in packages provided with LAMMPS. An
overview of packages is give in <aclass="reference internal"href="Section_packages.html"><em>Section packages</em></a>.</p>
<p>These are the accelerator packages
currently in LAMMPS, either as standard or user packages:</p>
<p>Note that the first 4 steps can be done as a single command, using the
src/Make.py tool. This tool is discussed in <aclass="reference internal"href="Section_start.html#start-4"><span>Section 2.4</span></a> of the manual, and its use is
illustrated in the individual accelerator sections. Typically these
steps only need to be done once, to create an executable that uses one
or more accelerator packages.</p>
<p>The last 4 steps can all be done from the command-line when LAMMPS is
launched, without changing your input script, as illustrated in the
individual accelerator sections. Or you can add
<aclass="reference internal"href="package.html"><em>package</em></a> and <aclass="reference internal"href="suffix.html"><em>suffix</em></a> commands to your input
script.</p>
<divclass="admonition note">
<pclass="first admonition-title">Note</p>
<pclass="last">With a few exceptions, you can build a single LAMMPS executable
with all its accelerator packages installed. Note however that the
USER-INTEL and KOKKOS packages require you to choose one of their
hardware options when building for a specific platform. I.e. CPU or
Phi option for the USER-INTEL package. Or the OpenMP, Cuda, or Phi
option for the KOKKOS package.</p>
</div>
<p>These are the exceptions. You cannot build a single executable with:</p>
<ulclass="simple">
<li>both the USER-INTEL Phi and KOKKOS Phi options</li>
<li>the USER-INTEL Phi or Kokkos Phi option, and either the USER-CUDA or GPU packages</li>
</ul>
<p>See the examples/accelerate/README and make.list files for sample
Make.py commands that build LAMMPS with any or all of the accelerator
packages. As an example, here is a command that builds with all the
GPU related packages installed (USER-CUDA, GPU, KOKKOS with Cuda),
including settings to build the needed auxiliary USER-CUDA and GPU
<spanid="acc-4"></span><h2>5.4. Comparison of various accelerator packages<aclass="headerlink"href="#comparison-of-various-accelerator-packages"title="Permalink to this headline">¶</a></h2>
<divclass="admonition note">
<pclass="first admonition-title">Note</p>
<pclass="last">this section still needs to be re-worked with additional KOKKOS
and USER-INTEL information.</p>
</div>
<p>The next section compares and contrasts the various accelerator
options, since there are multiple ways to perform OpenMP threading,
run on GPUs, and run on Intel Xeon Phi coprocessors.</p>
<p>All 3 of these packages accelerate a LAMMPS calculation using NVIDIA
hardware, but they do it in different ways.</p>
<p>As a consequence, for a particular simulation on specific hardware,
one package may be faster than the other. We give guidelines below,
but the best way to determine which package is faster for your input
script is to try both of them on your machine. See the benchmarking
section below for examples where this has been done.</p>
<p><strong>Guidelines for using each package optimally:</strong></p>
<ulclass="simple">
<li>The GPU package allows you to assign multiple CPUs (cores) to a single
GPU (a common configuration for “hybrid” nodes that contain multicore
CPU(s) and GPU(s)) and works effectively in this mode. The USER-CUDA
package does not allow this; you can only use one CPU per GPU.</li>
<li>The GPU package moves per-atom data (coordinates, forces)
back-and-forth between the CPU and GPU every timestep. The USER-CUDA
package only does this on timesteps when a CPU calculation is required
(e.g. to invoke a fix or compute that is non-GPU-ized). Hence, if you
can formulate your input script to only use GPU-ized fixes and
computes, and avoid doing I/O too often (thermo output, dump file
snapshots, restart files), then the data transfer cost of the
USER-CUDA package can be very low, causing it to run faster than the
GPU package.</li>
<li>The GPU package is often faster than the USER-CUDA package, if the
number of atoms per GPU is smaller. The crossover point, in terms of
atoms/GPU at which the USER-CUDA package becomes faster depends
strongly on the pair style. For example, for a simple Lennard Jones
system the crossover (in single precision) is often about 50K-100K
atoms per GPU. When performing double precision calculations the
crossover point can be significantly smaller.</li>
<li>Both packages compute bonded interactions (bonds, angles, etc) on the
CPU. This means a model with bonds will force the USER-CUDA package
to transfer per-atom data back-and-forth between the CPU and GPU every
timestep. If the GPU package is running with several MPI processes
assigned to one GPU, the cost of computing the bonded interactions is
spread across more CPUs and hence the GPU package can run faster.</li>
<li>When using the GPU package with multiple CPUs assigned to one GPU, its
performance depends to some extent on high bandwidth between the CPUs
and the GPU. Hence its performance is affected if full 16 PCIe lanes
are not available for each GPU. In HPC environments this can be the
case if S2050/70 servers are used, where two devices generally share
one PCIe 2.0 16x slot. Also many multi-GPU mainboards do not provide
full 16 lanes to each of the PCIe 2.0 16x slots.</li>
</ul>
<p><strong>Differences between the two packages:</strong></p>
<ulclass="simple">
<li>The GPU package accelerates only pair force, neighbor list, and PPPM
calculations. The USER-CUDA package currently supports a wider range
of pair styles and can also accelerate many fix styles and some
compute styles, as well as neighbor list and PPPM calculations.</li>
<li>The USER-CUDA package does not support acceleration for minimization.</li>
<li>The USER-CUDA package does not support hybrid pair styles.</li>
<li>The USER-CUDA package can order atoms in the neighbor list differently
from run to run resulting in a different order for force accumulation.</li>
<li>The USER-CUDA package has a limit on the number of atom types that can be
used in a simulation.</li>
<li>The GPU package requires neighbor lists to be built on the CPU when using
exclusion lists or a triclinic simulation box.</li>
<li>The GPU package uses more GPU memory than the USER-CUDA package. This
is generally not a problem since typical runs are computation-limited
rather than memory-limited.</li>
</ul>
<divclass="section"id="examples">
<h3>5.4.1. Examples<aclass="headerlink"href="#examples"title="Permalink to this headline">¶</a></h3>
<p>The LAMMPS distribution has two directories with sample input scripts
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.