Hello,
I'd like to continue this topic. The context of that is that I am testing distcc to perform distributed compilation across many hosts which are in fact pods (containers) in k8s env. What I observe is that when all compilation which run on certain container cause memory pressure (memory utilization hits memory limit defined by cgroupv2 for container, swap is disabled) then cpu load for cc1 processes significantly increase (container looks like hanged) and all compilations may last indefinitely. OOM killer is triggered when we hit the memory limit (that is expected). So some memory has been released and other compilations can be continued until we reach memory limit again, however in that point OOM killer is not invoked again. I observe that memory utilization almost equal to memory limit and this is permanent state. It behaves like gcc is aware of lacking memory and trying garbage collect continuously in order to free some memory to continue. Is it possible that gcc somehow prevents exceeding memory limit? My expectation in such scenario is that OOM killer is invoked each time when we hit memory limit. Any insights into whether this is GCC-internal behaviour or if it's more likely tied to Linux's / kernel memory management would be appreciated.
In my test I run 10 compilations of dummy_1.c in parallel (one compilation per cpu) where each compilation utilize ~1GB of RAM in peak. Command which I invoke:
# gcc -std=c99 -Wall -Wextra -pedantic -O3 -funroll-loops -finline-functions -fexpensive-optimizations -S -c "$src_file" -o "/tmp/test/build/${src_file%.c}.o"
BR,
Krystian
From: Krystian Kazmierczak (Nokia)
Sent: Wednesday, April 30, 2025 2:49 PM
To: 'gcc-help@xxxxxxxxxxx' <gcc-help@xxxxxxxxxxx>
Subject: GCC compilation performance under RAM starvation
Hello,
I'm observing that gcc compilation slows down significantly when the system experiences RAM starvation. I understand that some slowdown is expected under memory pressure, but I'm trying to determine whether this level of degradation is expected GCC behavior or is primarily due to Linux kernel memory management.
What I observe:
* During large builds (make -jN), when total RAM is nearly or fully exhausted, compilation slows drastically. Same source file compile ~20s when we have free memory comparing to 40s-60s when we reach the threshold of memory (if not killed by OOM meanwhile).
* Sometimes the Linux kernel's OOM killer is triggered, mostly by other processes not gcc itself, then gcc is selected to be killed e.g.:
[Fri Apr 18 16:14:49 2025] python invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=815
[Fri Apr 18 16:14:49 2025] Memory cgroup out of memory: Killed process 108353 (cc1plus) total-vm:5550480kB, anon-rss:5421224kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:10876kB oom_score_adj:815
* The system has swap disabled.
Questions:
* Is this performance degradation considered normal for gcc under memory pressure?
* Is there any GCC-specific tuning or flags that can help in constrained memory environments? (I'm aware of ggc-min-expand and ggc-min-heapsize but these in most cases leads to lower memory consumption at the cost of higher user time).
* I want to understand whether this observation is only an outcome of kernel's memory management or gcc itself is aware of reaching limit of RAM and then somehow "slows down".
System Info:
* GCC version: gcc (GCC) 13.1.0
* OS: AlmaLinux release 9.3
Any insights into whether this is GCC-internal behaviour or if it's more likely tied to Linux's memory management would be appreciated.
BR,
Krystian
#include <stdio.h>
#include <stdlib.h>
// **Generate Millions of Lines via Recursive Macros (Preprocessor Slowdown)**
#define RECURSIVE_MACRO(N) RECURSIVE_MACRO_##N
#define RECURSIVE_MACRO_1 1
#define RECURSIVE_MACRO_2 RECURSIVE_MACRO_1 + RECURSIVE_MACRO_1
#define RECURSIVE_MACRO_4 RECURSIVE_MACRO_2 + RECURSIVE_MACRO_2
#define RECURSIVE_MACRO_8 RECURSIVE_MACRO_4 + RECURSIVE_MACRO_4
#define RECURSIVE_MACRO_16 RECURSIVE_MACRO_8 + RECURSIVE_MACRO_8
#define RECURSIVE_MACRO_32 RECURSIVE_MACRO_16 + RECURSIVE_MACRO_16
#define RECURSIVE_MACRO_64 RECURSIVE_MACRO_32 + RECURSIVE_MACRO_32
#define RECURSIVE_MACRO_128 RECURSIVE_MACRO_64 + RECURSIVE_MACRO_64
#define RECURSIVE_MACRO_256 RECURSIVE_MACRO_128 + RECURSIVE_MACRO_128
#define RECURSIVE_MACRO_512 RECURSIVE_MACRO_256 + RECURSIVE_MACRO_256
#define RECURSIVE_MACRO_1024 RECURSIVE_MACRO_512 + RECURSIVE_MACRO_512
#define RECURSIVE_MACRO_2048 RECURSIVE_MACRO_1024 + RECURSIVE_MACRO_1024
#define RECURSIVE_MACRO_4096 RECURSIVE_MACRO_2048 + RECURSIVE_MACRO_2048
#define RECURSIVE_MACRO_8192 RECURSIVE_MACRO_4096 + RECURSIVE_MACRO_4096
#define RECURSIVE_MACRO_16384 RECURSIVE_MACRO_8192 + RECURSIVE_MACRO_8192
#define RECURSIVE_MACRO_32768 RECURSIVE_MACRO_16384 + RECURSIVE_MACRO_16384
#define RECURSIVE_MACRO_65536 RECURSIVE_MACRO_32768 + RECURSIVE_MACRO_32768
#define RECURSIVE_MACRO_131072 RECURSIVE_MACRO_65536 + RECURSIVE_MACRO_65536
#define RECURSIVE_MACRO_262144 RECURSIVE_MACRO_131072 + RECURSIVE_MACRO_131072
#define RECURSIVE_MACRO_524288 RECURSIVE_MACRO_262144 + RECURSIVE_MACRO_262144
#define RECURSIVE_MACRO_1048576 RECURSIVE_MACRO_524288 + RECURSIVE_MACRO_524288
#define RECURSIVE_MACRO_2097152 RECURSIVE_MACRO_1048576 + RECURSIVE_MACRO_1048576
// **Generate Large Symbol Table**
volatile long long large_array[5000][5000];
// **Recursive Inline Function for Slow Compilation**
inline long long recursive_function(int n) {
return (n == 0) ? 1 : n * recursive_function(n - 1);
}
// **Generate Thousands of Huge Inline Functions**
#define INLINE_FUNC(N) \
inline int inline_func_##N() { \
return inline_func_##N##_helper(100); \
} \
inline int inline_func_##N##_helper(int x) { \
return x * 2 + N; \
}
INLINE_FUNC(1)
INLINE_FUNC(2)
INLINE_FUNC(3)
INLINE_FUNC(4)
INLINE_FUNC(5)
INLINE_FUNC(6)
INLINE_FUNC(7)
INLINE_FUNC(8)
INLINE_FUNC(9)
INLINE_FUNC(10)
INLINE_FUNC(11)
INLINE_FUNC(12)
INLINE_FUNC(13)
INLINE_FUNC(14)
INLINE_FUNC(15)
INLINE_FUNC(16)
INLINE_FUNC(17)
INLINE_FUNC(18)
INLINE_FUNC(19)
INLINE_FUNC(20)
INLINE_FUNC(21)
INLINE_FUNC(22)
INLINE_FUNC(23)
INLINE_FUNC(24)
INLINE_FUNC(25)
INLINE_FUNC(26)
INLINE_FUNC(27)
INLINE_FUNC(28)
INLINE_FUNC(29)
INLINE_FUNC(30)
INLINE_FUNC(31)
INLINE_FUNC(32)
INLINE_FUNC(33)
INLINE_FUNC(34)
INLINE_FUNC(35)
INLINE_FUNC(36)
INLINE_FUNC(37)
INLINE_FUNC(38)
INLINE_FUNC(39)
INLINE_FUNC(40)
// **Force Huge Loop Unrolling**
void expensive_loop() {
volatile int x = 0;
#pragma GCC unroll 10000
for (long long i = 0; i < 10000; i++) { // 10M Unrolled Iterations
x += i % 100;
}
}
int main() {
printf("Starting long compilation program...\n");
// **Compile-Time Computation**
printf("Recursive Macro Computed Value: %d\n", RECURSIVE_MACRO_2097152);
// **Call Expensive Recursive Function**
printf("Recursive Function Result: %lld\n", recursive_function(15));
// **Call Multiple Inlined Functions**
printf("Inline Function Result: %d\n", inline_func_10());
// **Execute Huge Loop**
expensive_loop();
printf("Done!\n");
return 0;
}
