in cooperation with

 

Parallel Programming in Computational Engineering and Science (PPCES)

March 11th – 15th, 2024

About PPCES

This one week event will continue the tradition of previous annual week-long events that take place in Aachen every spring since 2001. We will cover the basics of parallel programming using OpenMP and MPI in Fortran and C/C++ and a first step towards performance tuning. Furthermore, we will embrace current topics in machine & deep learning. Hands-on exercises for each topic will be included.

The contents of the courses are generally applicable but will be specialized towards the compute cluster CLAIX which is the current system installed at the RWTH’s IT Center. It might be helpful to read through the information which is provided during the HPC introduction on February 5th and 6th 2024. This is especially true if you want to actively use CLAIX after this event.

Topics

OpenMP is a widely used approach for programming shared memory architectures, supported by most compilers nowadays. We will cover the basics of the programming paradigm as well as some advanced topics such as programming NUMA machines. The nodes of the RWTH Compute Cluster contain an increasing number of cores and thus we consider shared memory programming a vital alternative for applications that cannot be easily parallelized with MPI. We also expect a growing number of application codes to combine MPI and OpenMP for clusters of nodes with a growing number of cores.

The Message Passing Interface (MPI) is the de-facto standard for programming large HPC systems. We will introduce the basic concepts and give an overview of some advanced features. Also covered is hybrid parallelization, i.e., the combination of MPI and shared memory programming, which is gaining popularity as the number of cores per cluster node grows.

Machine & Deep Learning: We provide a fundamental introduction into machine and deep learning approaches as well as data processing techniques that support dataset preparation or model selection for training and inference. It covers the basic concepts for supervised and unsupervised learning such as classification, regression, and clustering to get a feeling which technique is appropriate for a particular problem. Additionally, we conduct several hands-on exercises with common frameworks such as scikit-learn and PyTorch on our recent HPC infrastructure that is equipped with powerful Intel Sapphire Rapid CPUs and NVIDIA H100 GPUs. In these exercises, we demonstrate how to define models, construct dataset and training pipelines or utilize monitoring and visualization of the training results, e.g., with Tensorboard. For the deep learning exercises, we start by training on a single GPU first. If that is not enough, we also show how to scale up distributed training onto multiple compute nodes or GPUs.

Organization

• There is no seminar fee.
• Presentations will be given in English. Slides will be available during or after the event.
• This is an in-person event and will be held on the RWTH IT Center premises (see below). Coffee, tea, and soda are sponsored by NEC throughout the event.
• You can/must register per topic, i.e., OpenMP (Mon + Tue), MPI (Wed + Thur), and ML/DL (Fri).
• For more details on the target audience, prerequistes and gained skills, see the “Skills” sections below (per topic).

Guest Speakers

We are happy to announce our guest speaker: Ruud van der Pas (Oracle Linux Engineering)

Agenda

Agenda is still subject to small changes.

Day 1+2: OpenMP

Monday, March 11	Day 1: OpenMP Part I
09:00 – 09:10	Welcome & Organization OpenMP Intro	Christian Terboven (RWTH)
09:10 – 10:10	Introduction into Parallel Computing	Ruud van der Pas (Oracle Linux Engineering)
10:40 – 12:00	OpenMP Basics – Part 1 Overview Parallel Region Worksharing Scoping Compilers + Exercises	Christian Terboven (RWTH)
13:30 – 14:30	OpenMP Basics – Part 2 (incl. Lab)	Christian Terboven (RWTH)
14:30 – 15:30	OpenMP Basics – Part 3 (incl. Lab) Motivation Tasking Model Taskloop Cutoff Dependencies	Christian Terboven (RWTH)
16:00 – 17:00	OpenMP Basics – Part 4 (incl. Lab) NUMA Affinity Hybrid	Christian Terboven (RWTH)

 

Day 3+4: MPI

Wednesday, March 13	Day 3: MPI Part I
09:00 – 10:30	MPI Basics MPI Concepts (incl. Lab)	Tobias Dollenbacher (RWTH), Joachim Jenke (RWTH)
11:00 – 12:30	Blocking Point-to-Point Communication – Part 1 (incl. Lab)	Tobias Dollenbacher (RWTH)
14:00 – 15:30	Blocking Point-to-Point Communication – Part 2 (incl. Lab)	Tobias Dollenbacher (RWTH)
16:00 – 17:00	Non-blocking Point-to-Point Communication (incl. Lab)	Joachim Jenke (RWTH)

 

Thursday, March 14	Day 4: MPI Part II
09:00 – 10:30	Derived Datatypes (incl. Lab)	Marc-André Hermanns (RWTH)
11:00 – 12:30	Blocking Collective Communication (incl. Lab)	Marc-André Hermanns (RWTH)
14:00 – 15:30	Communicator Basics (incl. Lab)	Tobias Dollenbacher (RWTH)
16:00 – 17:00	Hybrid Programming Wrap-Up	Jannis Klinkenberg (RWTH)

Day 5: Machine & Deep Learning

Friday, March 15	Day 5: Machine Learning
09:00 – 10:30	Introduction to AI and Machine Learning, and Preprocessing Techniques – Part 1 01_ML-DL_Agenda 02_ML-DL_Introduction_AI_ML	Jannis Klinkenberg (RWTH)
11:00 – 12:30	Introduction to AI and Machine Learning, and Preprocessing Techniques – Part 2 03_ML-DL_Execution_Options ML-DL_Hands-on	Jannis Klinkenberg (RWTH)
14:00 – 15:30	Introduction to (Distributed) Deep Learning 04_ML-DL_Introduction_DL 05_ML-DL_PyTorch 06_ML-DL_Distributed_DL 99_ML-DL_Further_Links_and_Literature	Jannis Klinkenberg (RWTH)
16:00 – 17:00	ML-DL_Hands-on	Jannis Klinkenberg (RWTH)

Skills – OpenMP

Course level

beginner

Target audience

• HPC users
• HPC developers

Prerequisites

• Able to understand C/C++ (or Fortran) source codes
• Comfortable with C/C++ or Fortran programming (for lab sessions)
• Comfortable using a Linux environment (for lab sessions)

Gained skills

• Broad knowledge of OpenMP, including the machine, memory and execution model
• General understanding when and how to use the different worksharing, synchronization and tasking constructs and directives
• Capability to apply OpenMP to existing codes and to use the scoping correctly
• Understanding of how OpenMP parallel programs interact with the system, in particular of NUMA architectures and SIMD microarchitectures

Skills – MPI

Course level

beginner

Target audience

• HPC developers

Prerequisites

• Able to understand C/C++ (or Fortran) source codes
• Comfortable with C/C++ or Fortran programming (for lab sessions)
• Comfortable using a Linux environment (for lab sessions)

Gained skills

• General understanding of the classic MPI “world model” of process initialization
• Overview of point-to-point and collective communication an their use cases
• Understand the use and definition of MPI derived datatypes
• Knowledge of basic MPI terminology
• Knowledge of blocking and non-blocking procedure calls and when to use them
• Using MPI together with OpenMP in basic hybrid scenarios

Skills – ML/DL

Course level

• beginner to intermediate

Target audience

• HPC/AI users
• HPC/AI developers

Prerequisites

• Basic knowledge of Python
• Comfortable using a JupyterHub or Linux environment (for lab sessions)
• Comfortable with Python programming and a little Bash scripting (for lab sessions)

Gained skills

• High-level overview of data proprocessing, AI, machine and deep learning techniques and how they relate to each other
• Understanding how to apply shallow ML techniques to solve classification and clustering tasks
• Understanding how to execute PyTorch deep learning and distributed deep learning on HPC infrastructures

Material Downloads

OpenMP Labs

• tbd.

MPI Labs

• ppces2024-MPI-labs-C.tar.gz
• ppces2024-MPI-labs-Fortran.tar.gz
• ppces2024-MPI-labs-all.tar.gz

ML/DL Labs

• ppces2024-ML-DL-labs.tar.gz

Further Links

Need support or want to ask questions?

• HPC Consultation Hour (once per month)
• AI/ML/DL Consultation Hour (once per week)

Want to apply for compute time (proposals)?

• Overview of Compute Project Categories
• Visit WestAI website and send mail to contact@westai.de

Registration

Registration link: https://eveeno.com/ppces_2024 (until March 4th, 2024)

During registration, you can select on which days you want to participate.

Sponsoring

Catering is sponsored by NEC.