Getting Started with OpenMP#

Resources: OpenMP Official Site

OpenMP is a widely used API for shared-memory parallel programming, this guide shows how to leverage it effectively on HPC systems such as Muscadine. We’ll cover the basics, practical examples, and optimization strategies.

What is OpenMP?#

OpenMP (Open Multi-Processing) is a standard API for parallel programming in shared-memory environments. It allows you to parallelize loops, sections, and tasks using compiler directives (#pragma in C/C++).

Key features:

Incremental parallelization: Parallelize sections of existing serial code.
Portable: Supported on most HPC clusters.
Scalable: Suitable for multi-core nodes on HPC systems.

Note

OpenMP does NOT scale across nodes. OpenMP is useful when optimizing existing code or when used in conjunction with MPI.

Setting Up Your Environment#

Before writing OpenMP programs, ensure your a compatible compiler is loaded:

module load gcc
# or
module load aocc

Check OpenMP support:

gcc -fopenmp -o test_openmp test_openmp.c

Your First OpenMP Program#

Parallelizing a simple for loop in C:

#include <stdio.h>
#include <omp.h>

int main() {
    int i;
    int n = 16;

    #pragma omp parallel for
    for (i = 0; i < n; i++) {
        printf("Thread %d processing iteration %d\n", omp_get_thread_num(), i);
    }

    return 0;
}

Compile and run:

gcc -fopenmp -O2 -o hello_omp hello_omp.c
export OMP_NUM_THREADS=4
./hello_omp

Output: Shows iterations distributed across threads.

OpenMP Directives#

Parallel Loops#

#pragma omp parallel for schedule(static, 4)
for (int i = 0; i < N; i++) {
    array[i] = compute(i);
}

schedule(static, 4) splits work into chunks of 4 iterations per thread.
schedule(dynamic) useful when iteration times vary.

Parallel Sections#

#pragma omp parallel sections
{
    #pragma omp section
    compute_part1();

    #pragma omp section
    compute_part2();
}

Reduction#

double sum = 0.0;

#pragma omp parallel for reduction(+:sum)
for (int i = 0; i < N; i++) {
    sum += array[i];
}

Reduction avoids race conditions in summing or accumulating values.

Best Practices for HPC#

Set OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK: Use the --cpus-per-task slurm directive to control threading
Avoid false sharing: Align data to prevent multiple threads from writing to the same cache line.
Use schedule(dynamic) for irregular workloads: Minimizes load imbalance.
Profile before tuning: Use tools like gprof, perf, or HPC-specific profilers.
Hybrid MPI + OpenMP: Combine MPI across nodes with OpenMP within (numa) nodes for optimal scaling.

Advanced HPC Tips#

Thread affinity: Bind threads to cores to reduce cache thrashing:
```
export OMP_PROC_BIND=close
export OMP_PLACES=cores
```
Nested parallelism: Enable with omp_set_nested(1) if your code spawns parallel regions inside others.
Memory considerations: Ensure your data structures fit within NUMA domains to maximize memory bandwidth.

Tip

Start small: parallelize the most computationally heavy loops first, and gradually extend OpenMP pragmas across your HPC code. Measure scaling at each step.