VI-HPS Tuning Workshop

Wednesday, March 5 – Friday, March 7, 2008

SunHPC 2008 Seminar

Monday, March 3 – Tuesday, March 4, 2008

Contents

• Introduction
• Programming Tools
• Soft- and Hardware Platforms
• Documentation
• Agenda and Slides
• Costs
• Registration and Feedback
• Logistics

Introduction

• the Jülich Supercomputing Centre (JSC) of the Forschungszentrum Jülich,
• the Center for Information Services and High Performance Computing (ZIH) of the Technische Universität Dresden, and
• the Innovative Computing Laboratory of the University of Tennessee (ICL).

took place in Aachen in March 2008.

Besides esteemed tools experts of the VI-HPS partners, we will be happy to welcome performance experts from Sun Microsystems as special guests, contributing their experience and assistance in using the Sun performance tools.

The mission of the VI-HPS is to improve the quality and accelerate the development process of complex simulation programs in science and engineering. For this purpose, integrated state-of-the-art programming tools for high-performance computing are developed that assist domain scientists in diagnosing programming errors and optimizing the performance of their applications. As training and support is an essential component of the VI-HPS activities, we are happy to invite application programmers to bring in their codes, to learn more about state-of-the-are programming tools for high-performance computing and get expert assistance in debugging, tuning and parallelization using MPI and OpenMP.

Performance tuning is still often a matter of some experimentation, but we can give you advice on a best effort basis. Application developers can expect to learn about the execution performance of their applications: this insight can be helpful even where it doesn’t directly lead to performance improvements. To maximize the efficiency of the workshop, we would like to ask you to prepare a test case that reflects a typical production run, but does not take too long to execute. In the ideal case, a run should not take more than 5 to 10 minutes to finish.  It is also important to have an easy way of verifying that the results of this test run are correct.

Wel provided access to and support for the programming tools and the computing platforms listed below:

The Programming Tools

Developing correct and scalable parallel programs is hard and good programming tools may increase the programmer’s productivity considerably. But developing these tools themselves is an art as well and this tuning workshop will be a unique opportunity for application programmers to directly interact with tool developers when working on their application codes.

In order to get some basic insight into the functioning of performance tools I like to point to an introduction into „Profiling and Tracing in Linux“ by Sameer Shende (pdf) which basically holds for other operating systems as well. A summary for first reading is provided here.

The tools which were presented and available in the context of this workshop were

• Sun Performance Analyzer is a proprietary tool from Sun Microsystems which is freely available for Linux and Solaris together with the Sun Studio compiler suite. The tool primarily targets towards serial and shared memory parallel program analysis and has some limited features for MPI program analysis.
• Acumem Virtual Performance Expert (VPE) is a commercial tool from Acumem to analyze the cache behaviour of program.
• ompP is a free profiling tool for OpenMP applications currently developed in the ICL.
  User Guide (pdf)
• OPARI is a free source-to-source translation tool for instrumenting OpenMP codes developed at the JSC. OPARI is used in conjunction with ompP, KOJAK and VampirTrace
• VampirTrace is an free MPI tracing library provided by ZIH generating trace files in the OTF open trace format. Such a tracefile can be displayed with the VAMPIR tool
  User Guide (pdf)
• VAMPIR is a commercial framework and graphical analysis tool developed at the ZIH to display and analyze OTF trace files generated by VampirTrace and other tools.
• KOJAK / SCALASCA for the performance analysis of MPI applications with a strong focus on improved scalability. Like KOJAK, tracefiles are analyzed to automatically detect and classify performance properties which then are displayed by the CUBE graphical utility.
  With the most recent versions, both tools have been merged into SCALASCA1.0.
  Quick Reference Guide (pdf)
• PAPI is a free interface to hardware performance counters developed at ICL which can be used by ompP, KOJAK, SCALASCA and VampirTrace.
  User Guide (pdf)
• MARMOT is a free correctness checking tools for MPI programs developed at ZIH and HLRS
• Sun Thread Analyzer a proprietary tool from Sun Microsystems which is freely available for Linux and Solaris together with the Sun Studio compiler suite. The tool checks for data races and deadlocks in shared memory parallel programs.

HPC December Workshop 2010 Part II

Array Building Blocks Tutorial (Intel Ct)

Thursday, December 9 – Friday, December 10, 2010

Date	Time	Location
Day 1: Thursday, December 9	09:00 – 17:15	Center for Computing and  Communication RWTH Aachen University Seffenter Weg 23 52074 Aachen Seminar Rooms 1 & 2  (Kopernikusstr.)
Day 2: Friday, December 10	09:00 – 15:30

Introduction

Intel® Array Building Blocks (ArBB) supports a high-level, generalized and portable programming model for data-parallel programming. It simplifies the efficient parallelization of computations over large data sets. Programmers do not need to focus on the implementation details of their data-parallel program, but instead can express a program’s algorithms in terms of operations on collections of data. ArBB’s deterministic semantics avoid race conditions and deadlocks by design, improving reliability and maintainability, and can be used for both rapid prototyping and production-stable codes.

Learning Material

• Tutorial Slides and Documentation ( only accessible for RWTH members, you need to login with your RWTH L2P-TiM-Account)
  • 00 Introduction.pdf
  • 01 Introduction to ArBB, Part 1.pdf
  • 02 Introduction to ArBB, Part 2.pdf
  • 03 ArBB Execution Engine.pdf
  • 04 ArBB Advanced Programming.pdf

Registration

closed

There will be a Social Dinner on Wednesday December 08 at 7p.m at Restaurant Elisenbrunnen.

HPC December Workshop Part I

Be sure to also consider part I of our december workshop: The Tuning Workshop

Contact

The UltraSPARC T1 („Niagara“) based Sun Fire T2000 Server

The Sun Fire T2000 Server

Since december 2005 a Sun Fire T2000 server with a 1GHz UltraSparc T1 „Niagra“ processor from Sun Microsystems is installed at the RWTH Aachen Universities Center for Computing and Communication (CCC). This system has 8 GByte main memory, and is running Solaris 10.

After the upgrade of all big Sun Fire servers of the Center with dual core UltraSPARC IV chips late 2004 and after the recent purchase of 4 new dual-core Opteron based V40z servers, the installation of the UltraSPARC T1 based Sun Fire T2000 system marks an important step towards the employment of new types of micro-processors with chip multi-processing (CMP) and chip multi-threading (CMT) technologies which will most likely dominate the market of HPC systems in the future.

An „integer-only“ Machine in a Center of Excellence for Engineering Sciences and Computational Fluid Dynamics?

Why installing a machine which is only capable of delivering some 100 MFlop/s in a compute environment dominated by technical applications?

On the first sight, this does not seem to fit well. But we want to be prepared for future technologies. For sure future multi-threading processors will be capable of executing floating point operations at the same rate as the Niagara processors executes integer opterations today. So we want to look at the questions of how to use this kind of architecture properly. Will this kind of machines be able to suite our needs in future?

The stagnation of the performance growth of single processors is very bad news for the HPTC community. As a consequence parallelization is getting even more important. For many engineering and scientific applications parallelization is not at all trivial. Unfortunately in our environment we do not see many codes which are „embarrassingly parallel“. Therefore we think that in future parallelization has to be on multiple levels. Hybrid parallelization using MPI plus OpenMP and autoparallelization, nested parallelization with MPI and with OpenMP will be needed to keep even more processors busy and to cut down the turnaround time of large simulation jobs.

So we want to investigate in how far the well-known techniques of MPI and OpenMP programming work on this brand new chip. Therefore we are looking at several benchmarks and applications, which are not dominated by floating point operations.

First Experiences using the Niagara Processor – A Word of Caution! Of course we are very curious about first performance results using this brand new processor architecture. And most likely others are curious to see our first benchmark results, too. But please handle these results with care. Performance results can only be as good as the people who run the experiments, and we are new to this system. They can be only as good as the compiler supports the architecture, and the Sun Studio compiler version 11 is the first version to support the Niagara processor. There were several people working on the used systems at a time. We try not to interfere with each other, but you never know. Also time is always short – we might have overlooked something. So take all these numbers as preliminary! We have been using the compiler switches -fast [-xtarget=ultraT1] [-g] [-xarch=v9b] throughout the tests unless otherwise noted. The picture shows the machines which have been used for many of the comparisons below. The Sun Fire T2000 is the 2U silver box on top of two blue Sun Fire E2900 boxes below. Indeed, after receiving the hint from Sun Microsystems to change a system parameter, we found out that this parameter might heavily impact the performance of the Sun Fire T2000. So we basically have to repeat all our measurements! We are also awaiting another patch for the system software … So far we added set consistent_coloring=2 in the /etc/systems file. Have a look at the altered performance curve of John-the-Ripper

A Look at the Memory Performance of the Sun Fire T2000

A tiny serial program measures the memory latency using pointer chasing by a long sequency of identical instructions:

p = (char **)*p;

A look into the disassembly reveals that in fact one load instructions follows another:

... 
ld [%g5], %g2 
ld [%g2], %g1 
ld [%g1], %o7 
ld [%o7], %o5 
ld [%o5], %o4 ...

The content of the memory location just read from memory delivers the address of the following load instruction. Thus these load instructions cannot be overlapped. The memory latency measured is: 107 ns

Now how about memory bandwidth? The code segment which is timed is

long *x, *xstart, *xend, mask; 
... 
for ( x = xstart; x < xend; x++ ) *x ^= mask;

So each loop iteration involves one load and one store of a variable of type long. The memory footprint of this loop is always much larger than the level 2 cache, so each load and store operations goes to the main memory. The memory bandwidth which has been measured depends on the size of the long type, which is 4 bytes when compiled for 32 bit addressing and 8 bytes when compiled for 64 bit addressing: 463 MB/s and 873 MB/s respectively.

Now the idea of the multi-threading architecture, as has been explained in a presentation given by Partha Tirumalai and Ruud van der Pas during the SunHPC colloquium in Aachen in October 2004, is bridging the growing gap between processor and memory speed by overlapping the stall time of one thread when waiting for data from memory with the activity of other threads running on the same hardware, thus leading to a much better utilitsation of the silicon.

So an obvious experiment is to run the same kernel measuring memory latency and bandwidth several times in parallel in order to find out in how far multiple threads running on the same processor or even on the same core on this processor interfere which each other. For this purpose we took the same tiny program kernels, parallelized it using MPI and used explicit processor binding to carefully place processes onto the processor cores. The given numbers are for 64-bit addressing mode.

For comparison we include measurement results on the Sun Fire E2900 which is equipped with 12 UltraSPARC IV processors running at 1200 MHz.

measuring memory latency and bandwidth with a parallel kernel program

The experiment nicely shows that the memory performance scales quite well. The memory latency increases only up to 129 ns when running 32 processes. For up to 8 processes it is profitable to distribute them across all eight cores instead of filling part of the cores with processes and leaving others empty. The only surprising exception is the 16 process case, where it seems to be more profitable to start 4 threads on 4 cores each leaving the other 4 cores empty.
The kernel program challenges the memory bandwidth considerably and reveils that in such a case the bandwidth might get a limiting factor for performance. The maximum total bandwidth which could be measured is about 12.4 times higher than the bandwidth which can be dedicated to a single process. It seems that the bandwidth is sufficiently scalable for up to eight processes, but may become a limiting factor for more processes in such extreme cases with no data locality.

The lat_mem_rd benchmark, which is part of the LMbench suite, can be used to look a bit closer into memory latency and the memory hierarchy. The pointer chasing mechanism works like described above, but we vary the stride between successive memory accesses and also the memory footprint.

The figure shows the memory performance as measured by the serial lat_mem_rd program for various strides

At first we ran the original serial version varying the memory footprint between 1KB and 8 MB and choosing a stride of 8, 16, 32, 64 and 128 bytes. As long as the memory footprint is below 8 KB, all access can be satisfied in the L1 cache and the „memory“ latency (the average latency of each load instructiong) is 3 ns. When the memory footprint is between 8 KB and 3 MB, the accesses are all satisfied in the L2 cache and the „memory“ latency is about 22 ns, with the only exception when the stride is only 8 bytes. In this case every second load instructions hits the cache line which has previously been fetched into the L1 cache, as the L1 cache line is 16 bytes long. With a stride of 16 bytes or larger, each load instruction misses the L1 cache. If the memory footprint is larger than 3 MB the latency of fetching a cache line of 64 bytes into the L2 cache takes 107 ns and if the stride is less than 64 bytes, this cache line is reused leading to lower average latencies.

Also this kernel program was parallelized using MPI and ran on the Niagara processor with a varying number of processes and with the strides 8, 64 and 8192. All MPI processes execute the same pointer chasing loop simultaneously of course on their private piece of memory. We plot the average latencies of all processes for each measurement.

The figure shows the memory performance as measured by the mpi version of the lat_mem_rd program for a stride of 8 bytes and various number of MPI processes. This was measured after we changed the system parameters.

With a stride of 8 Bytes there is a lot of cache line reuse of course. But the most striking information which is given by the above figure is that for a large memory footprint the latency does not really get worse when the number of MPI processes increases. The L2 cache which is shared by all cores leads to shift of the slope when the number of processes running simultaneously increases, which has to be expected.

The latency for a small memory footprint raises for 16 or more threads. This might be consequence of the sharing of the L1 caches between all threads of a single core. Here further investigations would be necessary to understand this effect. This is also the case for larger strides as can be seen below.

The figure shows the memory performance as measured by the mpi version of the lat_mem_rd program for a stride of 64 bytes and various number of MPI processes. This was measured before we changed the system parameters, note the difference!

The figure shows the memory performance as measured by the mpi version of the lat_mem_rd program for a stride of 64 bytes and various number of MPI processes. This was measured after we changed the system parameters, note the difference!

The same is true for a stride of 64 bytes, which leads to cache misses for each load operation. The latency is well below 140 ns in all cases. Again the effect of sharing a common L2 cache is clearly visible.

A stride of 8 KB may lead to all kind of nasty effects which nead further investigation. It can be expected that TLB misses have played an important role in slowing down the latency in this serios of measurments.

The EPCC OpenMP Micro Benchmark

The EPCC OpenMP Micro Benchmark carefully measures the performance of all major OpenMP primitives. The first test focusses on the OpenMP directives and the second test take a closer look at the performance of parallel loops using the various OpenMP schedule kinds. The Sun Fire T2000 shows a very similar behaviour in all aspects than the Sun Fire E2900 system, just the T2000 is clearly faster by a factor of roughly two! This correlates nicely to the difference in the memory latency.

NAS Parallel Benchmark NPB 3.2 OpenMP – Integer Sort

The NAS Parallel Benchmark Suite is very well-know in the HPC community. It is available in serial, OpenMP and MPI versions and contains one program focussing on integer performance: is. We ran this program for 4 different test cases W, A, B, C which work datasets of increasing size.

Comparing the performance of the Sun Fire T2000 system with the Sun Fire E2900 system, it can clearly be seen that the large caches of the UltraSPARC IV processors are profitable for the smaller testcases W, A, and B, whereas for the largest test case C a single Niagara processor running 16 threads outperforms 12 UltraSPARC IV processors. This was measured after we changed the system parameters, which did not make big difference for this case.

Integer Stream Benchmark

The OpenMP Stream Benchmark was changed to do integer instead of floating point operations. Both a Fortran and a C++ version where used for measurements.

Overall the UltraSPARC T1 system scales well, in some cases even up to 32 threads.

Parallel Partitioning of Graphs with ParMETIS

The ParMETIS program package is frequently used for partitioning unstructured graphs, in order to minimize the communication of large-scale numerical simulations.We selected a test case (mtest) which is not dominated by floating point operations. It is used to compute a k-way partitioning of a mesh containung elements of different types on p processors.

The measurements cannot be used to judge scalability, as the partitioning depends on the number of processors used. Here we compare the performance of a 24-way Sun Fire E2900 system, equipped with 12 dual-core UltraSPARC IV processors running at 1200 MHz versus the 1 GHz single-socket SunFire T2000 system:

When using the same number of MPI processes the UltraSPARC IV-based machine solves these problems about two times faster than the Niagara-based system. Remember, we are comparing a 12-socket SMP machine with a single socket system.

Password Cracking with „John the Ripper“

„John the Ripper“ is a popular password crack program used by system administrators to search for weak user passwords.
Hardware Counter analysis reveals that it does not use many floating point instructions and thus it might be a good candidate for the Sun Fire T2000 as a parallel version is publicly available too.
We employed mpich2 to compile and run this parallel version, as the test mode of „John“ uses the SIGALARM signal which is suppressed by Sun’s MPI implementation (HPC ClusterTools V6).

The performance measure is checks per second and we only looked at traditional DES encryption so far.

The figure depicts that the MPI version of John scales very well on the Sun Fire E2900 whereas on the Sun Fire T2000 it scales up to 8 MPI processes and then it drops down again. This was measured before we changed the system parameters…

Looking at the absolute performance, the Sun Fire E2900 clearly outperforms the Sun Fire T2000.This was measured before we changed the system parameters…

This figure depicts the scalability of „John“ on the Sun Fire T2000 after we changed the system parameters…

Why does „John the Ripper“ not scale on the Niagara processor?
Now, if the US IV chip does not suffer from stalls because a program displays a very good data locality such that almost always data can be kept in the caches, than multiple US IV processors will of course outperform a single Niagara processor. And indeed …

… hardware counter measurements reveal that instruction and data (level 1) cache misses badly hurt the Niagara processor. The number of data cache misses increases, when more processes are running per core, as they have to share a common L1 cache of 8 KB, whereas a single core on the UltraSPARC has a data cache of 64 KB. Plotted are the number of misses per MPI process for the whole test run. Please note that the scales differ by an order of magnitude between the Sun Fire T2000 (left y-axis) and the Sun Fire E2900 (right y-axis)! This was measured before we changed the system parameters…

The UltraSPARC IV (US IV) and the Niagara processor are both able to initiate 8 instructions per cycle, the US IV being a dual core 4-issue superscalar processor and the Niagara with 1 instruction per cycle for each of the eight cores.
Now comparing the number of checks per processor chip the 1 GHz Niagara chip even performs a little better than the 1.2 GHz US IV: 1283599 checks/sec versus 1187115 checks/sec.

What if the Niagara could count using floating point numbers …

In many of the PDE solvers running on our machines a CG type of linear equation solver is used in the heart of the computations. The most time consuming part of such a solver frequently is a sparse matrix vector multiplication, which accesses the main memory using index lists. Now how would a processor like the Niagara perform, if it had a floating point unit per core, like it has integer units today. In order to find out, we just changed the data type in one of our sparse matrix vector multiplication codes from double to long long int knowing, that the results of course would not be very meaningful. Well, just an experiment. This is what we get:

Sparse matrix vector multiplication on the SF E2900 and the SF T2000 when using 64 bit floating point or integer numbers. Whereas the Niagara only reaches up to about 100 MFlop/s it matches the speed of 12 UltraSPARC IV processors when using integers.

Summary

The single-thread performance of the UltraSPARC IV processor running at 1.2 GHz is about 2 -3 times higher than the single-thread performance of the 1 GHz Niagara chip for many of the programs which we looked at so far and which do not execute many floating point operations.
But the lower memory latency even for a high number of threads and the high memory bandwidth lead to a good performance for multi-threaded programs. This is particularly impressive for the class C integer sort benchmark of the NPB OpenMP collection.
Also the ParMETIS results are impressive, when taking into account that the performance of a single Niagara chip is compared to up to 12 UltraSPARC IV chips.

Links

• • Sun Microsystems
    • Sun Fire T1000
    • Sun Fire T2000
    • UltraSPARC T1
    • Throughput Computing
    • OpenSPARC T1
    • Developing and Tuning Applications on UltraSPARC T1 Chip Multithreading Systems

• Center for Computing and Communication of the RWTH Aachen University
  • High Performance Computing
  • Sun Fire SMP Cluster
  • The RWTH HPC Primer
• Further Links
  • AnandTech: LMBench 3.0a on Niagara
  • AnandTech: SUN’s UltraSparc T1 – the Next Generation Server CPUs

You want to get Hands-on Experience with the Niagara Processor?

Users having accounts on the machines of the CCC can seemlessly use the Sun Fire T2000 system upon request. The system is fully compatible to the other UltraSPARC IV based systems. It is running Solaris 10 and the Sun Studio compilers. Nevertheless we recommend that you load the latest Sun Studio 11 compiler into your environment by

module switch studio studio/11

and then recompile your application using the compiler flag

-xtarget=ultraT1

which will be expanded to

-xarch=v8plus -xcache=8/16/4/4:3072/64/12/32 -xchip=ultraT1

by the compiler (Keep in mind that the rightmost compiler option dominates if you for example want to add the -fast or the -xarch=v9b flag !)

The Solaris 10 operating systems deals with the system like a full blown 32-way shared memory machine. Keep in mind, that floating point intense application will run very slow! The machine is not built for such applications!

Dieter an Mey
Aachen, February 2008

High Performance Computing (HPC)

As described in the short introduction into HPC the focus of HPC is to reduce the time to solution of computational problems. The target is an efficient implementation of a suitable algorithm in a suitable programming language leading to an executable program which exploits the suitable machinery efficiently. Today, this usually results in a cache friendly parallel program written in C, C++ or Fortran.

Currently there are  two dominating parallelization paradigms:

• OpenMP for shared-memory-parallelization on multiprocessor computers (SMP), with all processors accessing a common memory
• Message Passing with MPI for distributed-memory parallelization for clusters of computers connect via a fast network

Performance Tuning Targets

What can be accomplished by parallelization and performance tuning? In an ideal case parallelization would lead to a speed-up which scales linearly with the number of processors employed compared to the original serial program running on a single processor. A list of naive expectations often will be made:

• A serial program is expected to run close to the processor’s theoretical peak performance.
• An OpenMP program running on an SMP with M processors (or processor cores) is expected to run M times faster with M OpenMP threads compared to the serial program running on one processor (core).
• An MPI program running on an SMP cluster consisting of N nodes with each node having M processors (or processor cores) is expected to run N*M times faster with N*M MPI processes
• A hybrid (OpenMP + MPI) parallel program running on an SMP cluster consisting of N nodes with each node having M processors (or processor cores) is expected to run N*M times faster with N MPI processes and M OpenMP threads per MPI process (or with for example N*2 MPI processes and M/2 OpenMP threads per MPI process).

What if a program’s performance does not meet these expectations? Indeed, there are good reasons why these expectations most likely will not be met:

• While today’s processor typically have a very high peak performance, memory performance lags behind and in many cases the processors cannot be fed with data fast enough to maintain a high computational speed. As a rule of thumb, modern processors typically achieve some five percent of their peak performance. But it may get worse, if the way data is accessed in memory is very unfortunate. In a lucky case, if a program displays a nice (temporal and spatial) locality and thus is very cache friendly performance may get close to some 20 to 50 percent of theoretical peak performance.
• Multiprocessor computers, multicore processors and even more multithreaded processor cores share a lot of resources. In an SMP the processors share paths to the common memory, cores in a multicore processor frequently share caches and pins, multithreaded processor cores share virtually everything except for a couple of state registers. All this resource sharing may lead to considerable conflicts limiting the speed-up of parallel programs.
• In a compute cluster the network between these nodes may be a bottleneck, if the MPI program is communication intensive. In this case the performance of the network, its latency, bandwidth and topology, will be an important factor.

Thus, performance tuning has three major aspects:

• designing and tuning a sequential program to exploit the serial performance
• parallelization for shared memory to exploit the node performance
• parallelization using message passing to exploit the cluster performance

Now, what if a program has been parallelized using OpenMP and/or MPI but it still runs too slow? What if adding resources, increasing the number of OpenMP threads and/or MPI processes even slows down the program?

Parallelization always introduces synchronization and communication overhead. Additionally any code regions which have not been parallelized limit the scalability of the parallel program according to the Amdahl’s Law. Another root of bad scalability is Load Imbalance, when some processes jost do nothing while waiting for other becaming ready.

Tuning Sequential Programs

In order to improve the performance of a single process, the hardware architecture of a processor has to be taken into account. Over the last decades the processor speed has been improved considerably, whereas memory has primarily increased its capacity but not the access time (latency). As a consequence, processors frequently have to wait for data coming from memory and in many cases cannot run at full speed. In order to overcome this bottleneck, chip designers have added fast but small buffers (caches) early on. These caches work transparently to the user and can speed up programs considerably. Still many compute intense applications only run at some five percent of the theoretically peak performance of modern processors, leaving considerable headroom for tuning efforts. The advent of multicore processors aggravates this memory bottleneck furthermore.

In order to improve the cache performance it is beneficial to reuse data residing in a cache in a timely manner (temporal locality). As caches are organized in so-called “cache lines” of typically 64 bytes and data is transported between memory and processors in chunks of cache lines, it is profitable to really use all of the data of such a chunk once it resides in a cache (spatial locality).

Tuning OpenMP Programs

• As all threads of an OpenMP program access a common memory, tuning the single thread’s memory performance is essential for improving the performance of an OpenMP program as well. Taking care about the locality of memory accesses is essential as all modern computer architectures are (cc)NUMA.
• A nasty aspect of bad performance of OpenMP programs which is particularly hard to detect is called “false sharing”. When threads update different parts of the same cache line, cache lines may be forced to travel between the caches, effectively putting the caches out of commission.
• Adding OpenMP constructs also introduces administrative overhead to the original program. If this overhead is too large compared to the amount of parallelized work which is distributed to multiple threads, then using OpenMP may be counterproductive at all as it will slow down the application.
• Furthermore, critical regions may have to be introduced into the program in order to update shared memory locations by multiple threads and avoid data races. These critical regions serialize the program flow and may seriously decrease performance.
• Load imbalances may cause threads to idle or wait at synchronization points (barriers). OpenMP provides easy-to-use loop scheduling policies to overcome these problems in many cases, which may in turn insert additional overhead. Another approach to handle non-uniformly pieces of work is Tasking (OpenMP from v3.0 on).

Tuning MPI Programs

• The major source of inefficiency of MPI programs is communication overhead. How much time is spent in sending messages compared to useful calculation? An opportunity of hiding communication is by overlapping communication and computation.
• Global communication involving many or all MPI processes – for example calculating reductions, residuals, or error estimations – may include costly synchronizations. Sometimes the agglomeration of such reductions can reduce the overhead.
• Load imbalances may easily become a serious performance problem. Tuning tools providing statistical and visual information about the program execution can easily detect these problems, but solving them may require a lot of program and even algorithmical changes.

Performance Tuning Procedure

The application of tuning tools consists of three major steps:

• Instrumentation or modification of the program to generate performance data
• Collecting runtime information
• Evaluation, analysis and presentation of the collected data (typically after the program run, but some tools also present performance data dynamically at runtime).

• timing information (CPU-time and/or real-time) on a statement, loop, routine, or program level
• hardware performance counter information (cache hits and misses, various kinds of instruction counters, memory access counters, network accesses etc.)
• number of executions (e.g. floating point operation) on a per instruction, per loop, or per routine basis
• specific wait times when threads or processes are synchronized, waiting for messages etc.
• program counter (PC) and sometimes callstack information to relate performance data to the program location at the analysis phase

Program Instrumentation

Instrumentation of the program can be done on various levels:

• On the source code level,
  • the programmer or a source-to-source pre-processor can add statements to inquire and record timing or hardware counter information and call library routines belonging to some measurement system in order to collect additional application specific information.
  • The compiler can instrument the code (if it supports this task).
• At link time profiling versions of libraries can be used, or routines can be wrapped by adapter routines collecting performance information.
• The executable can be enhanced through binary instrumentation.
• The binary program can be executed under the control of some measuring tool, as simple as a timer and as complex as a tracing utility which has access to a comprehensive dynamic tracing framework provided by the operating system.

Collecting runtime information

How can performance information be collected? Two different methods can be distinguished:

• Profiling updates summary statistics of execution when an event occurs. It uses the occurrence of an event to keep track of statistics of performance metrics. These statistics are maintained at runtime as the process executes and are stored in profile data files when the program terminates.
  In profiling based on sampling, a hardware interval timer periodically interrupts the execution. Such interrupts can as well be triggered by performance counters, that measure events in hardware.
• Tracing records a detailed log of timestamped events and their attributes. It reveals the temporal aspect of the execution. This allows the user to see when and where routine transitions, communication and user defined events take place.

The program’s image file and symbol table can be used after the program run to calculate the time spent in each routine.

Evaluation, analysis and presentation of the runtime information and performance properties

The output of performance tools can be as simple as a textual summary of a small amount of runtime statistics and as overwhelming as a huge amount of data to be presented by graphical user interfaces. These user interfaces provide different views on the data including timeline visualization revealing the behavior of the application over time.

The purpose of a performance tool is not to overwhelm the user with the sheer amount of data but to pinpoint the most important performance problems.

Outlook

Today, performance analysis of “regular” parallel programs is well understood. But analyzing “irregular” parallel programs applying nested, hybrid or even recursive parallelization, changing the number processes or threads on-the-fly is much harder and the subject of active research. On the other hand, the number processor cores employed in large applications is constantly increasing leading to enormous amounts of performance data, causing problems when storing, retrieving, analyzing and presenting them in an adequate and efficient way.

References

1. Sameer Shende, Profiling and Tracing in Linux
   http://www.cs.uoregon.edu/research/paraducks/papers/linux99.pdf
2. Intel VTune Amplifier
3. Bernd Mohr,  Performance Analysis of MPI Programs with Vampir and Vampirtrace (PDF)

Dieter an Mey
Aachen, February 2008

High Performance Computing (HPC)

As described in the short introduction into HPC the focus of HPC is to reduce the time to solution of computational problems. The target is an efficient implementation of a suitable algorithm in a suitable programming language leading to an executable program which exploits the suitable machinery efficiently. Today, this usually results in a cache friendly parallel program written in C, C++ or Fortran.

Currently there are  two dominating parallelization paradigms:

• OpenMP for shared-memory-parallelization on multiprocessor computers (SMP), with all processors accessing a common memory
• Message Passing with MPI for distributed-memory parallelization for clusters of computers connect via a fast network

Performance Tuning Targets

What can be accomplished by parallelization and performance tuning? In an ideal case parallelization would lead to a speed-up which scales linearly with the number of processors employed compared to the original serial program running on a single processor. A list of naive expectations often will be made:

• A serial program is expected to run close to the processor’s theoretical peak performance.
• An OpenMP program running on an SMP with M processors (or processor cores) is expected to run M times faster with M OpenMP threads compared to the serial program running on one processor (core).
• An MPI program running on an SMP cluster consisting of N nodes with each node having M processors (or processor cores) is expected to run N*M times faster with N*M MPI processes
• A hybrid (OpenMP + MPI) parallel program running on an SMP cluster consisting of N nodes with each node having M processors (or processor cores) is expected to run N*M times faster with N MPI processes and M OpenMP threads per MPI process (or with for example N*2 MPI processes and M/2 OpenMP threads per MPI process).

What if a program’s performance does not meet these expectations? Indeed, there are good reasons why these expectations most likely will not be met:

• While today’s processor typically have a very high peak performance, memory performance lags behind and in many cases the processors cannot be fed with data fast enough to maintain a high computational speed. As a rule of thumb, modern processors typically achieve some five percent of their peak performance. But it may get worse, if the way data is accessed in memory is very unfortunate. In a lucky case, if a program displays a nice (temporal and spatial) locality and thus is very cache friendly performance may get close to some 20 to 50 percent of theoretical peak performance.
• Multiprocessor computers, multicore processors and even more multithreaded processor cores share a lot of resources. In an SMP the processors share paths to the common memory, cores in a multicore processor frequently share caches and pins, multithreaded processor cores share virtually everything except for a couple of state registers. All this resource sharing may lead to considerable conflicts limiting the speed-up of parallel programs.
• In a compute cluster the network between these nodes may be a bottleneck, if the MPI program is communication intensive. In this case the performance of the network, its latency, bandwidth and topology, will be an important factor.

Thus, performance tuning has three major aspects:

• designing and tuning a sequential program to exploit the serial performance
• parallelization for shared memory to exploit the node performance
• parallelization using message passing to exploit the cluster performance

Now, what if a program has been parallelized using OpenMP and/or MPI but it still runs too slow? What if adding resources, increasing the number of OpenMP threads and/or MPI processes even slows down the program?

Parallelization always introduces synchronization and communication overhead. Additionally any code regions which have not been parallelized limit the scalability of the parallel program according to the Amdahl’s Law. Another root of bad scalability is Load Imbalance, when some processes jost do nothing while waiting for other becaming ready.

Tuning Sequential Programs

In order to improve the performance of a single process, the hardware architecture of a processor has to be taken into account. Over the last decades the processor speed has been improved considerably, whereas memory has primarily increased its capacity but not the access time (latency). As a consequence, processors frequently have to wait for data coming from memory and in many cases cannot run at full speed. In order to overcome this bottleneck, chip designers have added fast but small buffers (caches) early on. These caches work transparently to the user and can speed up programs considerably. Still many compute intense applications only run at some five percent of the theoretically peak performance of modern processors, leaving considerable headroom for tuning efforts. The advent of multicore processors aggravates this memory bottleneck furthermore.

In order to improve the cache performance it is beneficial to reuse data residing in a cache in a timely manner (temporal locality). As caches are organized in so-called “cache lines” of typically 64 bytes and data is transported between memory and processors in chunks of cache lines, it is profitable to really use all of the data of such a chunk once it resides in a cache (spatial locality).

Tuning OpenMP Programs

• As all threads of an OpenMP program access a common memory, tuning the single thread’s memory performance is essential for improving the performance of an OpenMP program as well. Taking care about the locality of memory accesses is essential as all modern computer architectures are (cc)NUMA.
• A nasty aspect of bad performance of OpenMP programs which is particularly hard to detect is called “false sharing”. When threads update different parts of the same cache line, cache lines may be forced to travel between the caches, effectively putting the caches out of commission.
• Adding OpenMP constructs also introduces administrative overhead to the original program. If this overhead is too large compared to the amount of parallelized work which is distributed to multiple threads, then using OpenMP may be counterproductive at all as it will slow down the application.
• Furthermore, critical regions may have to be introduced into the program in order to update shared memory locations by multiple threads and avoid data races. These critical regions serialize the program flow and may seriously decrease performance.
• Load imbalances may cause threads to idle or wait at synchronization points (barriers). OpenMP provides easy-to-use loop scheduling policies to overcome these problems in many cases, which may in turn insert additional overhead. Another approach to handle non-uniformly pieces of work is Tasking (OpenMP from v3.0 on).

Tuning MPI Programs

• The major source of inefficiency of MPI programs is communication overhead. How much time is spent in sending messages compared to useful calculation? An opportunity of hiding communication is by overlapping communication and computation.
• Global communication involving many or all MPI processes – for example calculating reductions, residuals, or error estimations – may include costly synchronizations. Sometimes the agglomeration of such reductions can reduce the overhead.
• Load imbalances may easily become a serious performance problem. Tuning tools providing statistical and visual information about the program execution can easily detect these problems, but solving them may require a lot of program and even algorithmical changes.

Performance Tuning Procedure

The application of tuning tools consists of three major steps:

• Instrumentation or modification of the program to generate performance data
• Collecting runtime information
• Evaluation, analysis and presentation of the collected data (typically after the program run, but some tools also present performance data dynamically at runtime).

• timing information (CPU-time and/or real-time) on a statement, loop, routine, or program level
• hardware performance counter information (cache hits and misses, various kinds of instruction counters, memory access counters, network accesses etc.)
• number of executions (e.g. floating point operation) on a per instruction, per loop, or per routine basis
• specific wait times when threads or processes are synchronized, waiting for messages etc.
• program counter (PC) and sometimes callstack information to relate performance data to the program location at the analysis phase

Program Instrumentation

Instrumentation of the program can be done on various levels:

• On the source code level,
  • the programmer or a source-to-source pre-processor can add statements to inquire and record timing or hardware counter information and call library routines belonging to some measurement system in order to collect additional application specific information.
  • The compiler can instrument the code (if it supports this task).
• At link time profiling versions of libraries can be used, or routines can be wrapped by adapter routines collecting performance information.
• The executable can be enhanced through binary instrumentation.
• The binary program can be executed under the control of some measuring tool, as simple as a timer and as complex as a tracing utility which has access to a comprehensive dynamic tracing framework provided by the operating system.

Collecting runtime information

How can performance information be collected? Two different methods can be distinguished:

• Profiling updates summary statistics of execution when an event occurs. It uses the occurrence of an event to keep track of statistics of performance metrics. These statistics are maintained at runtime as the process executes and are stored in profile data files when the program terminates.
  In profiling based on sampling, a hardware interval timer periodically interrupts the execution. Such interrupts can as well be triggered by performance counters, that measure events in hardware.
• Tracing records a detailed log of timestamped events and their attributes. It reveals the temporal aspect of the execution. This allows the user to see when and where routine transitions, communication and user defined events take place.

The program’s image file and symbol table can be used after the program run to calculate the time spent in each routine.

Evaluation, analysis and presentation of the runtime information and performance properties

The output of performance tools can be as simple as a textual summary of a small amount of runtime statistics and as overwhelming as a huge amount of data to be presented by graphical user interfaces. These user interfaces provide different views on the data including timeline visualization revealing the behavior of the application over time.

The purpose of a performance tool is not to overwhelm the user with the sheer amount of data but to pinpoint the most important performance problems.

Outlook

Today, performance analysis of “regular” parallel programs is well understood. But analyzing “irregular” parallel programs applying nested, hybrid or even recursive parallelization, changing the number processes or threads on-the-fly is much harder and the subject of active research. On the other hand, the number processor cores employed in large applications is constantly increasing leading to enormous amounts of performance data, causing problems when storing, retrieving, analyzing and presenting them in an adequate and efficient way.

References

1. Sameer Shende, Profiling and Tracing in Linux
   http://www.cs.uoregon.edu/research/paraducks/papers/linux99.pdf
2. Intel VTune Amplifier
3. Bernd Mohr,  Performance Analysis of MPI Programs with Vampir and Vampirtrace (PDF)

SunHPC 2008 Seminar

Monday, March 3 – Tuesday, March 4, 2008

followed by the first

VI-HPS Tuning Workshop

Wednesday, March 5 – Friday, March 7, 2008

Contents

• Introduction
• Participants
• Agenda
• Costs
• Registration and Feedback
• Logistics

Introduction

The SunHPC 2008 Seminar is the 8th event in a series of successful introductions into application performance tuning organized by the RWTH Aachen University and Sun Microsystems. We keep continuously changing the format a little bit over the years. This time it is combined with the first VI-HPS Tuning Workshop.

The SunHPC 2008 Seminar starts with a short overview on serial application performance tuning. This is followed by a detailed tutorial on shared memory parallelization.  After an extensive introduction into concepts related to parallelization, automatic parallelization as well as the OpenMP programming model are covered in great detail.
Through an instructor led lab there is an opportunity to try things out yourself using examples made available.

The general philosophy of the first part of the workshop is to build up understanding of key concepts that are relevant to obtain good application performance. Once this is achieved, it is much easier to use the development environment in the best possible way.

The Sun compilers, the Sun Performance Analyzer and the Sun Thread Analyzer are covered in detail. It is shown how these tools can be used to get optimal productivity and performance out of  UltraSPARC T2- and Opteron-based Sun systems.
We also briefly touch upon several third party software products, which augment the programmer’s tool suite on the Sun systems.

SunHPC 2008 Seminar

Monday, March 3 – Tuesday, March 4, 2008 

followed by the first

VI-HPS Tuning Workshop

Wednesday, March 5 – Friday, March 7, 2008

Contents

• Introduction
• Participants
• Agenda
• Costs
• Registration and Feedback
• Logistics

Introduction

HPC Tuning Workshop Tue, Dec 5  – Thu, Dec 7, 2017 IT Center RWTH Aachen University sponsored by:

Introduction

This year’s aiXcelerate HPC Tuning Workshop will focus on two topics:

1. MPI+OPA: Tuning of parallel programs employing message passing with MPI and running on the Intel OmniPath fabric.
2. KNL: Tuning for Intel’s Xeon Phi new many core processor

The nodes of Claix, the latest RWTH Aachen University’s Supercomputer, are equipped with Intel Broadwell processors and connected through a network with the Intel OmniPath architecture (OPA). Claix has recenlty been complemented by 16 nodes containing an Intel’s Xeon Phi 7210 processor – code named Knights Landing (KNL). At the Jülich Supercomputer Centre a cluster called „Booster“ will soon be installed which will contain Intel’s Xeon Phi many core 7250-F processors and an Intel OmniPath fabric. The KNL nodes at Aachen can well be used to prepare and tune programs for Booster.

Researchers from FZ Jülich and RWTH with high demand for compute power can apply for resources on these systems as part of the JARA-HPC partition (>>> more… ).
Researchers from all over Germany can apply for resources on Claix (>>> more… ).

The workshop will consist of presentations open to a broader auditorium and of a hands-on tuning workshop for a limited number of selected compute projects.
Presentations will be given in English.

We are proud to announce Michael Klemm and Christopher Dahnken, two HPC performance experts from Intel, who will give presentations and support you with your tuning efforts. They will be complemented by experts of the HPC Team of the IT Center at RWTH Aachen University.

Materials

1. Agenda
2. Michael Klemm (Intel): MPI & ITAC
3. Dirk Schmidl (RWTH): NonIntel Tools
4. Hristo Iliev (RWTH): Open MPI
5. Paul Kapinos (RWTH): RWTH Compute Cluster environment
6. Christopher Dahnken (Intel): Intel Xeon Phi Architecture
7. Christopher Dahnken (Intel): OpenMP SIMD
8. Michael Klemm (Intel): VTune, also for MPI

aiXcelerate – presentations

1. MPI+OPA: On Tue morning, Dec 5, we will start at 10:00 with presentations on MPI programming and tuning using Intel MPI on the OmniPath fabric (OPA).
   Please  register here for the presentations on MPI+OPA
2. KNL: On Wed morning, Dec 6, we will start at 10:00 with presentations on Xeon Phi (KNL) programming and tuning using Intel compiler and tools.
   Please  register here for the presentations on KNL

aiXcelerate – tuning workshop

In the afternoon after the presentations on Tue and Wed and also throughout Thu, Dec 7, we provide you with hands-on opportunity to adapt, analyze and tune your performance critical applications on Claix, with its Broadwell and Xeon Phi processors connected through the OmniPath Fabric.

Participation in the presentations is a prerequisite for participation in tuning workshop.

Attendees are kindly requested to prepare and bring in their own code. It is assumed that you have a good working knowledge with MPI and/or OpenMP, and C/C++ or Fortran, whatever your compute project employs. To maximize the efficiency of the workshop, we would like to ask you to prepare one or more test cases that reflect typical production runs, but do not take too long to execute – in the ideal case, a run should not take more than 5 to 10 minutes to finish. Members of the local HPC team will support  every accepted project in porting the code to Claix. By the start of the workshop the code should be ready to be analyzed.

If you plan to participate in the tuning activities please contact us by sending an email to hpcevent@itc.rwth-aachen.de.

Workshop participants are invited to take part in the social dinner on Wednesday evening in Restaurant Palladion .-please note if you like to take part in your registration. Thank you.

login-knl.hpc.itc.rwth-aachen.de

#BSUB -P hpclab
#BSUB -R knl
#BSUB -m c64m208k

HPC Tuning Workshop Tue, Dec 5  – Thu, Dec 7, 2017  IT Center  RWTH Aachen University  sponsored by:

Introduction

This year’s aiXcelerate HPC Tuning Workshop will focus on two topics:

1. MPI+OPA: Tuning of parallel programs employing message passing with MPI and running on the Intel OmniPath fabric.
2. KNL: Tuning for Intel’s Xeon Phi new many core processor

The nodes of Claix, the latest RWTH Aachen University’s Supercomputer, are equipped with Intel Broadwell processors and connected through a network with the Intel OmniPath architecture (OPA). Claix has recenlty been complemented by 16 nodes containing an Intel’s Xeon Phi 7210 processor – code named Knights Landing (KNL). At the Jülich Supercomputer Centre a cluster called „Booster“ will soon be installed which will contain Intel’s Xeon Phi many core 7250-F processors and an Intel OmniPath fabric. The KNL nodes at Aachen can well be used to prepare and tune programs for Booster.

Researchers from FZ Jülich and RWTH with high demand for compute power can apply for resources on these systems as part of the JARA-HPC partition (>>> more… ).
Researchers from all over Germany can apply for resources on Claix (>>> more… ).

The workshop will consist of presentations open to a broader auditorium and of a hands-on tuning workshop for a limited number of selected compute projects.
Presentations will be given in English.

We are proud to announce Michael Klemm and Christopher Dahnken, two HPC performance experts from Intel, who will give presentations and support you with your tuning efforts. They will be complemented by experts of the HPC Team of the IT Center at RWTH Aachen University.

Materials

1. Agenda
2. Michael Klemm (Intel): MPI & ITAC
3. Dirk Schmidl (RWTH): NonIntel Tools
4. Hristo Iliev (RWTH): Open MPI
5. Paul Kapinos (RWTH): RWTH Compute Cluster environment
6. Christopher Dahnken (Intel): Intel Xeon Phi Architecture
7. Christopher Dahnken (Intel): OpenMP SIMD
8. Michael Klemm (Intel): VTune, also for MPI

aiXcelerate – presentations

1. MPI+OPA: On Tue morning, Dec 5, we will start at 10:00 with presentations on MPI programming and tuning using Intel MPI on the OmniPath fabric (OPA).
   Please  register here for the presentations on MPI+OPA
2. KNL: On Wed morning, Dec 6, we will start at 10:00 with presentations on Xeon Phi (KNL) programming and tuning using Intel compiler and tools.
   Please  register here for the presentations on KNL

aiXcelerate – tuning workshop

In the afternoon after the presentations on Tue and Wed and also throughout Thu, Dec 7, we provide you with hands-on opportunity to adapt, analyze and tune your performance critical applications on Claix, with its Broadwell and Xeon Phi processors connected through the OmniPath Fabric.

Participation in the presentations is a prerequisite for participation in tuning workshop.

Attendees are kindly requested to prepare and bring in their own code. It is assumed that you have a good working knowledge with MPI and/or OpenMP, and C/C++ or Fortran, whatever your compute project employs. To maximize the efficiency of the workshop, we would like to ask you to prepare one or more test cases that reflect typical production runs, but do not take too long to execute – in the ideal case, a run should not take more than 5 to 10 minutes to finish. Members of the local HPC team will support  every accepted project in porting the code to Claix. By the start of the workshop the code should be ready to be analyzed.

If you plan to participate in the tuning activities please contact us by sending an email to hpcevent@itc.rwth-aachen.de.

Workshop participants are invited to take part in the social dinner on Wednesday evening in Restaurant Palladion .-please note if you like to take part in your registration. Thank you.

login-knl.hpc.itc.rwth-aachen.de

#BSUB -P hpclab
#BSUB -R knl
#BSUB -m c64m208k

HPC Tuning Workshop Mon, Dec 3  – Wed, Dec 5, 2018 IT Center RWTH Aachen University sponsored by:

Introduction

aiXcelerate HPC Tuning Workshop had the focus on the Intel Skylake (SKK) Microarchitecture, SIMD programming and performance tuning using Likwid, Intel VTune/Amplifier and Intel Advisor.

The nodes of CLAIX-2018, the latest RWTH Aachen University’s Supercomputer, were equipped with Intel Skylake processors and connected through a network with the Intel OmniPath architecture (OPA). It was a major extension of CLAIX-2016 system equipped with Intel  Broadwell processors and connected through OPA, too.

Researchers from FZ Jülich and RWTH with high demand for compute power can apply for resources on these systems as part of the JARA-HPC partition (>>> more… ).
Researchers from all over Germany can apply for resources on Claix (>>> more… ).

The workshop consisted of presentations open to a broader auditorium and of a hands-on tuning workshop for a limited number of selected computing projects.
Presentations was  given in English.

We awere proud to announce Michael Klemm and Christopher Dahnken, two HPC performance experts from Intel, who gave presentations and support your tuning efforts. Furthermore, a presentation on detecting performance limiting factors with hardware monitoring using likwid was given by Thomas Gruber (born Röhl) main developer of the Likwid tool. Experts of the HPC Team of the IT Center at RWTH Aachen University assisted in your tuning activietes as well.

Materials

1. Agenda
2. Sandra Wienke (RWTH): aiXcelerate Welcome & CLAIX-2018 Overview
3. Marcus Wagner, Paul Kapinos (RWTH): SLURM and Modules for CLAIX-2018
4. Jonas Hahnfeld (RWTH): Performance Monitoring on CLAIX
5. Michael Klemm (Intel): Skylake Architecture
6. Michael Klemm (Intel): Skylake Performance Considerations
7. Michael Klemm (Intel): Intel VTune Amplifier
8. Thomas Gruber (FAU Erlangen): LIKWID, Detecting Performance Limiting Factors with Hardware Monitoring
   

Registration

Closed

aiXcelerate 2018 – Presentations

1. Dec 3, 11:00-13:00 – On Mon morning, Dec 3, we started at 11:00 with presentations on the new CLAIX-2018 cluster and the Skylake processor microarchitecture
2. Dec 3, 14:00-15:00 – After lunch on Monday, we continued with presentations on SIMD programming and performance optimization.
3. Dec 4, 11:00-13:00 – On Tuesday morning, Dec 4, we started at 11:00 with presentations on Likwid, Intel VTune/Amplifier and Advisor.

aiXcelerate 2018 – Tuning Workshop

After the presentations on Mon Dec 3, on Tue Dec4, and also throughout Wed, Dec 5, we provided you with hands-on opportunity to adapt, analyze and tune your performance critical applications on CLAIX, with its Skylake (and Broadwell) processors connected with the Intel OmniPath (OPA) fabric.

Participation in the presentations was a prerequisite for participation in tuning workshop.

Attendees were kindly requested to prepare and bring in their own code. It was assumed that you had a good working knowledge with MPI and/or OpenMP, and C/C++ or Fortran, whatever your compute project employs. To maximize the efficiency of the workshop, we asked the participants to prepare one or more test cases that reflected typical production runs, but did not take too long to execute – in the ideal case, a run didnot  take more than 5 to 10 minutes to finish. Members of the local HPC team supported every accepted project in porting the code to Claix. By the start of the workshop the code was asked be ready to be analyzed.

Furthermore, each participant (or participating group) was kindly asked to shortly present their project (1-2 slides) at the beginning of the tuning workshop and also shortly present the outcome of the tuning efforts at the end of the workshop.

Participants who were intested in the tuning activities contacted us by sending an email to hpcevent@itc.rwth-aachen.de.

Workshop participants were invited to take part in the social dinner on Tuesday, Dec 4 in the restaurant Palladion.

HPC Tuning Workshop Mon, Dec 3  – Wed, Dec 5, 2018  IT Center  RWTH Aachen University  sponsored by:

Introduction

aiXcelerate HPC Tuning Workshop had the focus on the Intel Skylake (SKK) Microarchitecture, SIMD programming and performance tuning using Likwid, Intel VTune/Amplifier and Intel Advisor.

The nodes of CLAIX-2018, the latest RWTH Aachen University’s Supercomputer, were equipped with Intel Skylake processors and connected through a network with the Intel OmniPath architecture (OPA). It was a major extension of CLAIX-2016 system equipped with Intel  Broadwell processors and connected through OPA, too.

Researchers from FZ Jülich and RWTH with high demand for compute power can apply for resources on these systems as part of the JARA-HPC partition (>>> more… ).
Researchers from all over Germany can apply for resources on Claix (>>> more… ).

The workshop consisted of presentations open to a broader auditorium and of a hands-on tuning workshop for a limited number of selected computing projects.
Presentations was  given in English.

We awere proud to announce Michael Klemm and Christopher Dahnken, two HPC performance experts from Intel, who gave presentations and support your tuning efforts. Furthermore, a presentation on detecting performance limiting factors with hardware monitoring using likwid was given by Thomas Gruber (born Röhl) main developer of the Likwid tool. Experts of the HPC Team of the IT Center at RWTH Aachen University assisted in your tuning activietes as well.

Materials

1. Agenda
2. Sandra Wienke (RWTH): aiXcelerate Welcome & CLAIX-2018 Overview
3. Marcus Wagner, Paul Kapinos (RWTH): SLURM and Modules for CLAIX-2018
4. Jonas Hahnfeld (RWTH): Performance Monitoring on CLAIX
5. Michael Klemm (Intel): Skylake Architecture
6. Michael Klemm (Intel): Skylake Performance Considerations
7. Michael Klemm (Intel): Intel VTune Amplifier
8. Thomas Gruber (FAU Erlangen): LIKWID, Detecting Performance Limiting Factors with Hardware Monitoring
   

Registration

Closed

aiXcelerate 2018 – Presentations

1. Dec 3, 11:00-13:00 – On Mon morning, Dec 3, we started at 11:00 with presentations on the new CLAIX-2018 cluster and the Skylake processor microarchitecture
2. Dec 3, 14:00-15:00 – After lunch on Monday, we continued with presentations on SIMD programming and performance optimization.
3. Dec 4, 11:00-13:00 – On Tuesday morning, Dec 4, we started at 11:00 with presentations on Likwid, Intel VTune/Amplifier and Advisor.

aiXcelerate 2018 – Tuning Workshop

After the presentations on Mon Dec 3, on Tue Dec4, and also throughout Wed, Dec 5, we provided you with hands-on opportunity to adapt, analyze and tune your performance critical applications on CLAIX, with its Skylake (and Broadwell) processors connected with the Intel OmniPath (OPA) fabric.

Participation in the presentations was a prerequisite for participation in tuning workshop.

Attendees were kindly requested to prepare and bring in their own code. It was assumed that you had a good working knowledge with MPI and/or OpenMP, and C/C++ or Fortran, whatever your compute project employs. To maximize the efficiency of the workshop, we asked the participants to prepare one or more test cases that reflected typical production runs, but did not take too long to execute – in the ideal case, a run didnot  take more than 5 to 10 minutes to finish. Members of the local HPC team supported every accepted project in porting the code to Claix. By the start of the workshop the code was asked be ready to be analyzed.

Furthermore, each participant (or participating group) was kindly asked to shortly present their project (1-2 slides) at the beginning of the tuning workshop and also shortly present the outcome of the tuning efforts at the end of the workshop.

Participants who were intested in the tuning activities contacted us by sending an email to hpcevent@itc.rwth-aachen.de.

Workshop participants were invited to take part in the social dinner on Tuesday, Dec 4 in the restaurant Palladion.