Kategorie: ‘RWTH-HPC’
Changes to default partition for Claix jobs that do not define it.
Dear HPC Users,
With the introduction of Claix 2025, we have changed the way default partitions are defined for jobs that provide none.
Prior to this change, the default partition was statically defined for all Slurm project accounts. A job submitted without a partition would fall back to this hard-coded default partition.
This has now changed. We now take a best-effort approach to define the default partition for each job dynamically, based on the login / submit node from which the job was submitted, the account used and the resources requested.
Jobs submitted from Claix23 machines will set a default c23ms partition for CPU only jobs and c23g partition for GPU jobs.
The same will now happen with jobs submitted from Claix25 machines (c25ms and c25g).
IH projects and login nodes will also be taken into account.
We still ask for users to try and explicitly define partitions for their jobs to avoid any undesired behaviors.
The best practice is therefore still to define partitions in the jobs explicitly.
This change should not affect the majority of jobs or how they are billed.
Important Changes for HPC Accounts – Usage of the NHR community AAI
We are unifying the account experience for our HPC services. As part of this, your local account must be connected to the National High Performance Computing (NHR) Community Authentication and Authorization Infrastructure (CAAI).
This linking is mandatory to keep access to your HPC account. Please link your account by June 31, 2026.
TL;DR
The following section summarizes the actions required to keep your HPC account active. Detailed explanations and instructions are provided throughout this email.
– External users with an existing RWTH account: Please link your home organization identity with your RWTH account as described on IT Center Help [1], if you have not already done so.
– All users (internal and external): Please log in once to the RWTH RegApp [2] via your home organization (SSO) to keep access to your HPC account.
– Applicants for computing time (JARDS): Use the new home organization login as described in the JARDS manual [3].
What has changed and why?
Previously, you logged in to the RWTH RegApp directly with your RWTH account to manage your HPC account.
Last year, we introduced the NHR CAAI as an identity proxy in front of the RegApp:
1. You now first log in via NHR CAAI using your home organization credentials.
2. You are then forwarded to the RWTH RegApp.
This allows external users to use our HPC systems seamlessly with their home organization identity.
If you have not used the RWTH RegApp since this change, your RegApp account is probably not yet mapped to your NHR CAAI account. We need this mapping to maintain a unified and consistent user account lifecycle.
What is required for external users with an RWTH account?
If you are not a member of RWTH Aachen University, but you do have an RWTH account:
1. Link your home organization account to your RWTH account as described on IT Center Help [1], if you have not done this yet.
2. Afterwards, follow the steps for all users below.
This ensures that your home organization login and your RWTH account are correctly associated.
What must all users (internal and external) do?
All users (RWTH-internal and external) have to:
1. Go to the RWTH RegApp [2].
2. When redirected to NHR CAAI, select your home organization (e.g., RWTH, your external institution, your ORCID or your Google account).
3. Log in with your home organization’s Single Sign-On (SSO) credentials.
During this process, an NHR CAAI account is created (if necessary) and mapped to your existing RWTH RegApp account.
Verify that it worked
We strongly recommend that you verify the result:
1. Open the RWTH RegApp [2].
2. Go to: “Registered services” → “RWTH High Performance Computing”.
3. Check that the displayed HPC username matches your actual HPC username.
If the RWTH High Performance Computing service is missing, or if the username is incorrect, please contact the IT-ServiceDesk (servicedesk@itc.rwth-aachen.de).
Will this affect my current HPC usage?
For now, your ability to connect to and authenticate on the HPC systems is not affected by the state of your RegApp account, until we start deactivating unlinked accounts in six months.
This means:
– If anything goes wrong during the procedure, you can continue to use the HPC service for the time being.
– We will notify users who have not completed the migration.
– We will also monitor and investigate cases where the migration appears to have failed.
If you have any reason to believe that you have lost access to your RWTH RegApp account due to this migration, please contact the IT-ServiceDesk (servicedesk@itc.rwth-aachen.de).
Did anything change for the computing time application as well?
Until now, you have been able to log in to JARDS via an email callback in order to apply for computing time. The JARDS login was also switched to a federated SSO login via CAAI on April 2, 2026.
We recommend that you use this new login method from now on and sign in to JARDS using the account of your home institution.
To keep your existing JARDS account when switching to the new home-organization login (SSO), your home organization’s Identity Provider (IdP) must transmit the same email as you previously used for the JARDS email callback. To ensure this, please follow these instructions:
1. Log in here: https://login.nhr-verein.de
2. Navigate to “Index → My E‑Mail Addresses”.
3. If the email is not the same as you used as JARDS email callback, add the email.
4. Confirm the email you receive.
5. Log in to JARDS via SSO.
Until October 1, 2026, you can continue to apply for compute time (“JARDS.application”) either via the existing email callback (not recommended) or via the new SSO login (recommended). For all reviewers (“JARDS.review”), the email callback will remain available even after October 1, 2026.
If you have any questions or problems, the team at the IT-ServiceDesk (servicedesk@itc.rwth-aachen.de) will be happy to help you.
Kind regards,
Your HPC Team
[1] https://help.itc.rwth-aachen.de/service/rhr4fjjutttf/article/9a42e86854ab43049d0f27b17e3ff601/
[2] https://regapp.itc.rwth-aachen.de
[3] https://jards.itc.rwth-aachen.de/jards/WEB/application/showloginhelp.php
GPU TDP reduced on CLAIX 2023 nodes
As of the 30.06.2026 at 11:30 the maximum TDP of the H100 GPUs on Claix 2023 nodes has been reduced from the previous value of 700W to 600W.
We implemented this lower TDP to reduce the thermal load on the central cooling system. This central cooling system was operating hotter and causing some specific GPU nodes to trigger thermal protection shutdowns that in turn kill any running jobs on said nodes.
This change is noteworthy for users that perform performance measurements and would like to compare their previous measurements to new ones under the reduced TDP.
For normal usage of the GPUs of CLAIX 2023, this change should not affect validity of results and should provide close to stock times to solution.
Users should not expect longer runtimes in most cases. For extremely intense computations, please recheck if your workflows require more runtime and adjust your batch scripts accordingly. We nevertheless, do not expect this change to affect most users of GPUs and is of no consequence to non-GPU users.
ML compute quota now shown in GPU-h
We would like to inform users of the following changes to r_wlm_usage.
Users can now see that the r_wlm_usage and r_wlm_usage -q commands display different quotas for MPI (CPU Only) and ML (GPUs) projects. The r_wlm_usage table will display for the same project, two distinct sections as follows:
=== HPC partitions (CPU Core-h) ===
….
=== ML partitions (GPU-h) ===
….
These changes are mostly in how the values are displayed as GPU-h. The used quotas have not changed and we expect no negative effects to previous, current or future quota utilization. Already used and allocated quotas within ML (GPU) systems were converted with the pre-existing ratio of 24CPU-h per 1 GPU-h. This change is to help distinguish between CPU and GPU usage within the NHR directives.
Additionally the default GPU-h quota for the ‘default’ project of every account running on GPU nodes has been set to 10 GPU-h.
This means that users can use their ‘default’ CPU-h quota without using the GPU-h quota and vice versa. The same rules of over-quota apply to both types.
Firefox Replaced on Dialog Nodes
The Firefox web browser was uninstalled from all HPC cluster dialog nodes due to lacking security updates for the given system configuration. Moreover, Firefox was deployed the modules tree to maintain its availability until the update problems using the OS packages can be fully resolved. However, this strategy requires to load the “Firefox” module using the command line first and executing it then on the command line as well.
To workaround these limitations, we are providing a wrapper script and a desktop definition, such that you should be able to continue using Firefox seamlessly without any further actions required.
You can track any disruptions or security advisories that may occur due to the aforementioned change in the RWTH High Performance Computing (HPC) category on our status reporting portal.
Runtime limits for GPU Jobs
On the 06.03.2026 we changed how many GPU nodes from the c23g partition are allowed to run user GPU Jobs with runtimes longer than 24 hours.
The change has the following main goals:
- Increase the throughput and reduce the waiting times for GPU jobs with runtimes lower than 24 hours.
- Encourage users to submit shorter jobs and make use of resilience methods like checkpointing if necessary.
- Reduce the maintenance downtime of GPU nodes.
The change effectively limits the amount of long running GPU jobs to only half the GPU nodes in the c23g partition. Long running GPU jobs are user GPU jobs with runtimes longer than 24 hours. GPU jobs with less than 24 hours of runtime are considered short and will be scheduled to all available GPU nodes of the c23g partition.
It is understood that the waiting times of long running GPU jobs will increase and we therefore encourage users to change their workflows to accommodate shorter running GPU jobs.
This change is necessary to improve QoS for users of the c23g partition and to allow for faster maintenance works on the GPU nodes.
Security Update and Cluster Maintenance for Munge
Summary: A security vulnerability (CVE) has been identified in the Munge software, which could potentially allow unauthorized access to the Munge key used for user authentication in Slurm jobs. In response to this issue, a new version of Munge has been released.
Details:
- Issue Identified: A security flaw in Munge that may expose the Munge key.
- Action Taken: Deployment of a new version of Munge to address the vulnerability.
- Maintenance Scheduled: The cluster will undergo maintenance to exchange the current Munge key.
User Instructions:
- Job Review: All users are requested to review their job lists. If any unknown jobs appear, please delete them and notify us immediately.
- Sensitive Data Handling: Users with extremely sensitive data in their directories can request a temporary suspension of job submissions for their accounts. Accounts will be re-enabled after maintenance is complete.
We appreciate your cooperation and understanding as we work to enhance our system’s security.
You can track any disruptions or security advisories that may occur due to the aforementioned change in the RWTH High Performance Computing (HPC) category on our status reporting portal.
Slurm Update for Claix
We updated Slurm to a newer and more stable Version: 25.05.5
This upgrade fixed issues we had with our scheduling system and internal rights-management database.
We improved how Slurm calculates priority for pending Jobs based on user feedback and internal metrics. Here is a short summary with basic details.
In short:
– Job waiting times will be more predictable and intuitive.
– Longer waiting times will increase the priority of a pending jobs.
– Jobs will still be able to access resources quickly if recent resource usage quotas are low.
Details:
– A pending job will not be delayed by any other new jobs (see its expected start time increase into the future) after 24 hours of waiting.
– Note that other software or hardware malfunctions might still cause delays in jobs, but new jobs will no longer cause this after 24 hours.
– A pending job with low fair-share factor might still be delayed by new jobs with higher fair-share factors during the first 24 hours of waiting.
– The project used for a job (default or otherwise) will determine its fair-share priority factor based on recent resource usage.
– Projects that have already used their “fair-share” of resources, will have a lower fair-share priority factor than projects with lower recent resource usage.
– Priorities and fair-share priority factors only matter for comparing jobs waiting for the same resources (e.g: partitions).
New Project Limits
As you know, we operate our cluster in 1-cluster-concept. This also means that the system is supported by multiple funding sources with different budgets. The NHR share (Tier-2) is the largest. Unfortunately, the project quota for RWTH-s, RWTH-thesis and RWTH-lecture (Tier-3) in the machine learning segment (GPU partition) is currently overbooked. Therefore, the following changes will take effect immediately for all new project applications (already granted projects are not affected):
- RWTH-s projects: reduction of the maximum project size to 4000 GPU-h per project per year
- RWTH-lecture and -thesis projects: reduction of the maximum project size to 1500 GPU-h per project
In order to ensure that all users can still successfully carry out their research projects, we propose the following approach for future proposals:
- If you do not need more than 4000 GPU-h, please apply for RWTH-s as before.
- If you do not need more than 10000 GPU-h and would like to conduct a project in the field of machine learning or artificial intelligence, please apply for a WestAI project.
- If you do not need more than 10000 GPU-h, you are welcome to apply once (!) for an NHR Starter project. Please note that this category is intended as preparation for an NHR Normal or Large project. An extension is therefore only possible with a full proposal and should be for more than 10000 GPU-h. NHR Starter projects are allocated centrally by the NHR office; if you choose this route, please select RWTH as your preferred center.
- If you need more than 10000 GPU-h, please apply for resources within an NHR Normal or Large project. To ensure appropriate use of the system, NHR projects undergo a science-led peer-review process.
We are convinced that this step is in the interest of all users in order to avoid overloading the GPU partition (ML segment).
Furthermore, please note that in future, NHR Normal projects may request up to 20 million core-h.
Please refer to the project catalogue for a complete overview (including a decision flow chart),
Please contact servicedesk@itc.rwth-aachen.de for any further questions.
Slurm GPU HPC resource allocation changing on the 01.11.2025
The CLAIX HPC systems will be changing the way GPU resources are requested and allocated starting on the 01.11.2025.
Users submitting Slurm Jobs will no longer be able to request arbitrary amounts of CPU and Memory resources when using GPUs on GPU nodes.
Requesting an entire GPU node’s memory or all CPUs, but only a single GPU will no longer be possible.
Each GPU within a GPU node will have a corresponding strict maximum of CPUs and Memory that can be requested.
To obtain more than the strict maximum of CPUs or Memory per GPU, more GPUs will need to be requested too.
The specific limits per GPU on GPU nodes will be eventually documented separately.
Users are expected to modify their submission scripts or methods accordingly.
This change is driven by our efforts to update the HPC resource billing mechanism to comply with NHR HPC directives.
NHR requires that computing projects apply for CPU and GPU resources independently.
NHR also requires that HPC Centers track the use of these CPU and GPU resources.
The independent resources are then accounted for by Slurm jobs within our CLAIX nodes.
Therefore CPU nodes will only track CPUs (and equivalent memory) and GPU nodes will only track GPUs used.
The quota tools will eventually reflect this too.
