Repository navigation
Wrong memory ceiling in cgroup v2 environments #47259
Description
Since #27508 Node.js is able to automatically limit its heap size based on cgroup limits instead of the host's physical memory. This is very important for containerized workloads, where we have many pods with smaller memory limits running on a node with a large amount of memory. Node.js has to consider the cgroup limit and not the amount of physical memory, in order to avoid allocating more than allowed and getting OOM killed.
This seems to have broken with cgroup v2, because libuv's uv_get_constrained_memory does not support cgroup v2, at least in the currently released version 1.44.2. It has already been implemented in libuv/libuv#3744 in September 2022, but there hasn't been a release since.
For us the issue manifested itself in many Node.js pods getting OOM killed after a Kubernetes cluster upgrade to 1.25 where cgroup v2 support is introduced. Because the solution depends on the next libuv release I don't expect a fix here, this report is intended more as a tracking issue for people affected by it.
At the moment I can see several options:
- Wait and work around it by manually setting
--max-old-space-size - Backport cgroup v2 support from
libuv - Switch to a different mechanism for detecting available memory (seems unattractive given we already have the
libuvdependency)
Version
v13.0.0 and upward
Platform
5.15.0-1034-azure
Subsystem
src/api/environment.cc
What steps will reproduce the bug?
I reproduced this by comparing the behavior on two AKS clusters, one on 1.24.9 (cgroup v1) and the other on 1.25.5 (cgroup v2). Both had a single node with 4 GiB memory. But you should be able to observe the behavior in any environment using cgroup v2, you can verify that is the case by running stat -fc %T /sys/fs/cgroup/ which should output cgroup2fs.
My repro:
- Create a pod with a limit of
600Mi - From within that pod run
node -e "console.log(v8.getHeapStatistics())"and check the output
How often does it reproduce? Is there a required condition?
Always when using cgroup v2
What is the expected behavior? Why is that the expected behavior?
heap_size_limit should be less than the cgroup maximum reported by cat /sys/fs/cgroup/memory.max (v2) or cat /sys/fs/cgroup/memory/memory.limit_in_bytes (v1). In my specific repro with the 600 MiB pod limit, heap_size_limit was 312 MiB in the cgroup v1 environment.
What do you see instead?
heap_size_limit is more than the cgroup maximum reported by cat /sys/fs/cgroup/memory.max, it seems influenced by the host's physical memory instead. In my specific repro with the 600 MiB pod limit, heap_size_limit was 1.96 GiB in the cgroup v2 environment, which will eventually lead to the pod being OOM killed.
Activity
bnoordhuis commentedon Mar 26, 2023
This is a known issue (as you mention) and blocked on the next libuv release, tracked in libuv/libuv#3887.
The release will likely happen in the next few weeks but keep in mind it takes a while to make its way into node's LTS lines.
I'm going to go ahead and close this but I can convert it to a discussion if you have follow-up questions.
martindisch commentedon Mar 26, 2023
That's great, thank you. We can keep it like this, that way we have somewhere to point people to that are encountering it as well.
deps: upgrade to libuv 1.45.0
deps: upgrade to libuv 1.45.0
test: unflake test-vm-timeout-escape-nexttick
tools: update LICENSE and license-builder.sh
deps: upgrade to libuv 1.45.0
test: unflake test-vm-timeout-escape-nexttick
tools: update LICENSE and license-builder.sh
sudeep1607 commentedon Jun 7, 2023
@targos Will this be released for Node 18 LTS as well?
We are seeing same behaviour on Node 18 LTS too
sandeeplamb commentedon Jun 20, 2023
Do you think it can cause the Node also down in kubernetes cluster as no resources left on node to run system processes and then kubelet goes down which ultimately removes the node also from worker nodes group in cluster?
martindisch commentedon Jun 20, 2023
I can't tell you for sure what happens in your specific k8s setup, but it's certainly possible that your pods using more memory has additional adverse effects that could impact your node and node pool.
4 remaining items
deps: upgrade to libuv 1.45.0
test: unflake test-vm-timeout-escape-nexttick
tools: update LICENSE and license-builder.sh
deps: upgrade to libuv 1.45.0
test: unflake test-vm-timeout-escape-nexttick
tools: update LICENSE and license-builder.sh
deps: upgrade to libuv 1.45.0
deps: upgrade to libuv 1.45.0
deps: upgrade to libuv 1.45.0
deps: upgrade to libuv 1.45.0
nubunto commentedon Oct 11, 2023
I am seeing CPU throttling in my case. I already set
max_old_space_sizeto be 80% of the configured amount available in the memory requests. Could it be related? Anything else I can do to fix this?martindisch commentedon Oct 11, 2023
I don't see how it could be related. This issue is only aboute Node's ability to set its heap size based on the environment. By the way, I'm happy to report that the fix has made it into recent versions of Node.js and we no longer need to set
max_old_space_sizein our systems.bnoordhuis commentedon Oct 11, 2023
FYI, the v18.18.1 release (#50066) rolls back libuv to a version without the cgroupsv2 fixes.
martindisch commentedon Oct 11, 2023
Well it was fun while it lasted :) Thanks for the heads-up, would've walked right into that one otherwise.
linux: ignore cgroup v1 memory soft limit (#5360)