respectively. Or the effect of the “-pk” or “-sf” switches can be
duplicated by adding the <aclass="reference internal"href="package.html"><em>package kokkos</em></a> or <aclass="reference internal"href="suffix.html"><em>suffix kk</em></a> commands respectively to your input script.</p>
<p>Or you can follow these steps:</p>
<p>CPU-only (run all-MPI or with OpenMP threading):</p>
<p>KOKKOS_DEVICE sets the parallelization method used for Kokkos code
(within LAMMPS). KOKKOS_DEVICES=OpenMP means that OpenMP will be
used. KOKKOS_DEVICES=Pthreads means that pthreads will be used.
KOKKOS_DEVICES=Cuda means an NVIDIA GPU running CUDA will be used.</p>
<p>If KOKKOS_DEVICES=Cuda, then the lo-level Makefile in the src/MAKE
directory must use “nvcc” as its compiler, via its CC setting. For
best performance its CCFLAGS setting should use -O3 and have a
KOKKOS_ARCH setting that matches the compute capability of your NVIDIA
hardware and software installation, e.g. KOKKOS_ARCH=Kepler30. Note
the minimal required compute capability is 2.0, but this will give
signicantly reduced performance compared to Kepler generation GPUs
with compute capability 3.x. For the LINK setting, “nvcc” should not
be used; instead use g++ or another compiler suitable for linking C++
applications. Often you will want to use your MPI compiler wrapper
for this setting (i.e. mpicxx). Finally, the lo-level Makefile must
also have a “Compilation rule” for creating <ahref="#id1"><spanclass="problematic"id="id2">*</span></a>.o files from <ahref="#id3"><spanclass="problematic"id="id4">*</span></a>.cu files.
See src/Makefile.cuda for an example of a lo-level Makefile with all
of these settings.</p>
<p>KOKKOS_USE_TPLS=hwloc binds threads to hardware cores, so they do not
migrate during a simulation. KOKKOS_USE_TPLS=hwloc should always be
used if running with KOKKOS_DEVICES=Pthreads for pthreads. It is not
necessary for KOKKOS_DEVICES=OpenMP for OpenMP, because OpenMP
provides alternative methods via environment variables for binding
threads to hardware cores. More info on binding threads to cores is
given in <spanclass="xref std std-ref">this section</span>.</p>
<p>KOKKOS_ARCH=KNC enables compiler switches needed when compling for an
Intel Phi processor.</p>
<p>KOKKOS_USE_TPLS=librt enables use of a more accurate timer mechanism
on most Unix platforms. This library is not available on all
platforms.</p>
<p>KOKKOS_DEBUG is only useful when developing a Kokkos-enabled style
within LAMMPS. KOKKOS_DEBUG=yes enables printing of run-time
debugging information that can be useful. It also enables runtime
bounds checking on Kokkos data structures.</p>
<p>KOKKOS_CUDA_OPTIONS are additional options for CUDA.</p>
<p>For more information on Kokkos see the Kokkos programmers’ guide here:
/lib/kokkos/doc/Kokkos_PG.pdf.</p>
<p><strong>Run with the KOKKOS package from the command line:</strong></p>
<p>The mpirun or mpiexec command sets the total number of MPI tasks used
by LAMMPS (one or multiple per compute node) and the number of MPI
tasks used per node. E.g. the mpirun command in MPICH does this via
its -np and -ppn switches. Ditto for OpenMPI via -np and -npernode.</p>
<p>When using KOKKOS built with host=OMP, you need to choose how many
OpenMP threads per MPI task will be used (via the “-k” command-line
switch discussed below). Note that the product of MPI tasks * OpenMP
threads/task should not exceed the physical number of cores (on a
node), otherwise performance will suffer.</p>
<p>When using the KOKKOS package built with device=CUDA, you must use
exactly one MPI task per physical GPU.</p>
<p>When using the KOKKOS package built with host=MIC for Intel Xeon Phi
coprocessor support you need to insure there are one or more MPI tasks
per coprocessor, and choose the number of coprocessor threads to use
per MPI task (via the “-k” command-line switch discussed below). The
product of MPI tasks * coprocessor threads/task should not exceed the
maximum number of threads the coproprocessor is designed to run,
otherwise performance will suffer. This value is 240 for current
generation Xeon Phi(TM) chips, which is 60 physical cores * 4
threads/core. Note that with the KOKKOS package you do not need to
specify how many Phi coprocessors there are per node; each
coprocessors is simply treated as running some number of MPI tasks.</p>
<p>You must use the “-k on”<aclass="reference internal"href="Section_start.html#start-7"><span>command-line switch</span></a> to enable the KOKKOS package. It
takes additional arguments for hardware settings appropriate to your
system. Those arguments are <aclass="reference internal"href="Section_start.html#start-7"><span>documented here</span></a>. The two most commonly used
options are:</p>
<divclass="highlight-python"><divclass="highlight"><pre>-k on t Nt g Ng
</pre></div>
</div>
<p>The “t Nt” option applies to host=OMP (even if device=CUDA) and
host=MIC. For host=OMP, it specifies how many OpenMP threads per MPI
task to use with a node. For host=MIC, it specifies how many Xeon Phi
threads per MPI task to use within a node. The default is Nt = 1.
Note that for host=OMP this is effectively MPI-only mode which may be
fine. But for host=MIC you will typically end up using far less than
all the 240 available threads, which could give very poor performance.</p>
<p>The “g Ng” option applies to device=CUDA. It specifies how many GPUs
per compute node to use. The default is 1, so this only needs to be
specified is you have 2 or more GPUs per compute node.</p>
<p>The “-k on” switch also issues a “package kokkos” command (with no
additional arguments) which sets various KOKKOS options to default
values, as discussed on the <aclass="reference internal"href="package.html"><em>package</em></a> command doc page.</p>
<p>Use the “-sf kk”<aclass="reference internal"href="Section_start.html#start-7"><span>command-line switch</span></a>,
which will automatically append “kk” to styles that support it. Use
the “-pk kokkos”<aclass="reference internal"href="Section_start.html#start-7"><span>command-line switch</span></a> if
you wish to change any of the default <aclass="reference internal"href="package.html"><em>package kokkos</em></a>
optionns set by the “-k on”<aclass="reference internal"href="Section_start.html#start-7"><span>command-line switch</span></a>.</p>
<p>Note that the default for the <aclass="reference internal"href="package.html"><em>package kokkos</em></a> command is
to use “full” neighbor lists and set the Newton flag to “off” for both
pairwise and bonded interactions. This typically gives fastest
performance. If the <aclass="reference internal"href="newton.html"><em>newton</em></a> command is used in the input
script, it can override the Newton flag defaults.</p>
<p>However, when running in MPI-only mode with 1 thread per MPI task, it
will typically be faster to use “half” neighbor lists and set the
Newton flag to “on”, just as is the case for non-accelerated pair
styles. You can do this with the “-pk”<aclass="reference internal"href="Section_start.html#start-7"><span>command-line switch</span></a>.</p>
<p><strong>Or run with the KOKKOS package by editing an input script:</strong></p>
<p>The discussion above for the mpirun/mpiexec command and setting
appropriate thread and GPU values for host=OMP or host=MIC or
device=CUDA are the same.</p>
<p>You must still use the “-k on”<aclass="reference internal"href="Section_start.html#start-7"><span>command-line switch</span></a> to enable the KOKKOS package, and
specify its additional arguments for hardware options appopriate to
your system, as documented above.</p>
<p>Use the <aclass="reference internal"href="suffix.html"><em>suffix kk</em></a> command, or you can explicitly add a
“kk” suffix to individual styles in your input script, e.g.</p>
<p>The performance of KOKKOS running in different modes is a function of
your hardware, which KOKKOS-enable styles are used, and the problem
size.</p>
<p>Generally speaking, the following rules of thumb apply:</p>
<ulclass="simple">
<li>When running on CPUs only, with a single thread per MPI task,
performance of a KOKKOS style is somewhere between the standard
(un-accelerated) styles (MPI-only mode), and those provided by the
USER-OMP package. However the difference between all 3 is small (less
than 20%).</li>
<li>When running on CPUs only, with multiple threads per MPI task,
performance of a KOKKOS style is a bit slower than the USER-OMP
package.</li>
<li>When running on GPUs, KOKKOS is typically faster than the USER-CUDA
and GPU packages.</li>
<li>When running on Intel Xeon Phi, KOKKOS is not as fast as
the USER-INTEL package, which is optimized for that hardware.</li>
</ul>
<p>See the <aclass="reference external"href="http://lammps.sandia.gov/bench.html">Benchmark page</a> of the
LAMMPS web site for performance of the KOKKOS package on different
hardware.</p>
<p><strong>Guidelines for best performance:</strong></p>
<p>Here are guidline for using the KOKKOS package on the different
hardware configurations listed above.</p>
<p>Many of the guidelines use the <aclass="reference internal"href="package.html"><em>package kokkos</em></a> command
See its doc page for details and default settings. Experimenting with
its options can provide a speed-up for specific calculations.</p>
<p><strong>Running on a multi-core CPU:</strong></p>
<p>If N is the number of physical cores/node, then the number of MPI
tasks/node * number of threads/task should not exceed N, and should
typically equal N. Note that the default threads/task is 1, as set by
the “t” keyword of the “-k”<aclass="reference internal"href="Section_start.html#start-7"><span>command-line switch</span></a>. If you do not change this, no
additional parallelism (beyond MPI) will be invoked on the host
CPU(s).</p>
<p>You can compare the performance running in different modes:</p>
<ulclass="simple">
<li>run with 1 MPI task/node and N threads/task</li>
<li>run with N MPI tasks/node and 1 thread/task</li>
<li>run with settings in between these extremes</li>
</ul>
<p>Examples of mpirun commands in these modes are shown above.</p>
<p>When using KOKKOS to perform multi-threading, it is important for
performance to bind both MPI tasks to physical cores, and threads to
physical cores, so they do not migrate during a simulation.</p>
<p>If you are not certain MPI tasks are being bound (check the defaults
for your MPI installation), binding can be forced with these flags:</p>
<p>For binding threads with the KOKKOS OMP option, use thread affinity
environment variables to force binding. With OpenMP 3.1 (gcc 4.7 or
later, intel 12 or later) setting the environment variable
OMP_PROC_BIND=true should be sufficient. For binding threads with the
KOKKOS pthreads option, compile LAMMPS the KOKKOS HWLOC=yes option, as
discussed in <aclass="reference internal"href="Section_start.html#start-3-4"><span>Section 2.3.4</span></a> of the
manual.</p>
<p><strong>Running on GPUs:</strong></p>
<p>Insure the -arch setting in the machine makefile you are using,
e.g. src/MAKE/Makefile.cuda, is correct for your GPU hardware/software
(see <aclass="reference internal"href="Section_start.html#start-3-4"><span>this section</span></a> of the manual for
details).</p>
<p>The -np setting of the mpirun command should set the number of MPI
tasks/node to be equal to the # of physical GPUs on the node.</p>
<p>Use the “-k”<aclass="reference internal"href="Section_start.html#start-7"><span>command-line switch</span></a> to
specify the number of GPUs per node, and the number of threads per MPI
task. As above for multi-core CPUs (and no GPU), if N is the number
of physical cores/node, then the number of MPI tasks/node * number of
threads/task should not exceed N. With one GPU (and one MPI task) it
may be faster to use less than all the available cores, by setting
threads/task to a smaller value. This is because using all the cores
on a dual-socket node will incur extra cost to copy memory from the
2nd socket to the GPU.</p>
<p>Examples of mpirun commands that follow these rules are shown above.</p>
<divclass="admonition note">
<pclass="first admonition-title">Note</p>
<pclass="last">When using a GPU, you will achieve the best performance if your
input script does not use any fix or compute styles which are not yet
Kokkos-enabled. This allows data to stay on the GPU for multiple
timesteps, without being copied back to the host CPU. Invoking a
non-Kokkos fix or compute, or performing I/O for
<aclass="reference internal"href="thermo_style.html"><em>thermo</em></a> or <aclass="reference internal"href="dump.html"><em>dump</em></a> output will cause data
to be copied back to the CPU.</p>
</div>
<p>You cannot yet assign multiple MPI tasks to the same GPU with the
KOKKOS package. We plan to support this in the future, similar to the
GPU package in LAMMPS.</p>
<p>You cannot yet use both the host (multi-threaded) and device (GPU)
together to compute pairwise interactions with the KOKKOS package. We
hope to support this in the future, similar to the GPU package in
LAMMPS.</p>
<p><strong>Running on an Intel Phi:</strong></p>
<p>Kokkos only uses Intel Phi processors in their “native” mode, i.e.
not hosted by a CPU.</p>
<p>As illustrated above, build LAMMPS with OMP=yes (the default) and
MIC=yes. The latter insures code is correctly compiled for the Intel
Phi. The OMP setting means OpenMP will be used for parallelization on
the Phi, which is currently the best option within Kokkos. In the
future, other options may be added.</p>
<p>Current-generation Intel Phi chips have either 61 or 57 cores. One
core should be excluded for running the OS, leaving 60 or 56 cores.
Each core is hyperthreaded, so there are effectively N = 240 (4*60) or
N = 224 (4*56) cores to run on.</p>
<p>The -np setting of the mpirun command sets the number of MPI
tasks/node. The “-k on t Nt” command-line switch sets the number of
threads/task as Nt. The product of these 2 values should be N, i.e.
240 or 224. Also, the number of threads/task should be a multiple of
4 so that logical threads from more than one MPI task do not run on
the same physical core.</p>
<p>Examples of mpirun commands that follow these rules are shown above.</p>
<divclass="section"id="restrictions">
<h2>Restrictions<aclass="headerlink"href="#restrictions"title="Permalink to this headline">¶</a></h2>
<p>As noted above, if using GPUs, the number of MPI tasks per compute
node should equal to the number of GPUs per compute node. In the
future Kokkos will support assigning multiple MPI tasks to a single
GPU.</p>
<p>Currently Kokkos does not support AMD GPUs due to limits in the
available backend programming models. Specifically, Kokkos requires
extensive C++ support from the Kernel language. This is expected to
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.