Repository navigation
FinalizationGroup in combination with buffers seems to allocate memory without releasing it #30853
Description
- Version: v12.13.1
- Platform: Linux apechkurov-laptop 4.15.0-72-generic fs: fix fd leak on early readstream destroy #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
- Subsystem: Buffer, FinalizationGroup API
Recently I was working on an experimental Buffer pool implementation based on FinalizationGroup API (see #30683) and encountered a weirdness that I'd like clarify. It seems that node doesn't free memory when FinalizationGroup is used to track Buffers (as "holdings") under certain conditions. This issue may be related with off-heap memory allocator or OS memory behavior, but I'd like to confirm that.
Here is the most simple reproducer that I could find:
'use strict';
const fg = new FinalizationGroup(finalizer);
// 8 ticks, 1GB per each => 8GB total
const ticks = 8;
const bufsPerTick = 1024;
const size = 1024 * 1024;
let slices = [];
let tick = 0;
setInterval(() => {
tick += 1;
if (tick === ticks) console.log('Registered all slices');
if (tick > ticks) {
slices = [];
return;
}
slices = [];
for (let i = 0; i < bufsPerTick; i++) {
const buf = Buffer.alloc(size);
const slice = buf.slice();
slices.push(slice);
fg.register(slice, buf);
}
}, 500);
let finalized = 0;
function finalizer(iter) {
for (const _ of iter) {
finalized += 1;
if (finalized === ticks * bufsPerTick)
console.log('All finalizer callbacks are triggered');
}
}When this script is run under --harmony-weak-refs flag, node process consumes about 2GB of physical memory on my machine (and about 2.5GB of virtual memory) and that value doesn't decrease even after 10 minutes.
On the other hand, if you comment the fg.register(slice, buf); line, you'll see that resident memory consumption eventually goes down to ~32KB (virtual is ~600MB).
Once again, I'm not sure if that's a bug, but I'd like to understand the reason of such behavior.
Activity
Jamesernator commentedon Dec 10, 2019
There's a decent chance it's on the v8 side, check this thread for implementation status: https://bugs.chromium.org/p/v8/issues/detail?id=8179
Also try on Node 13, it's possible the issue has already been fixed.
puzpuzpuz commentedon Dec 10, 2019
Thanks @Jamesernator. I've tried running the script on
v13.3.1-nightly20191204355b48bd06and still see the same behavior withFinalizationGroup.registerbeing used.What's even more interesting, with commented
fg.register(slice, buf);line, I can see about 900MB of resident memory consumption and it doesn't go down. It's totally different from memory consumption of v12.13.1 in the same scenario.So, maybe it's not related with FG API itself and it's something different?
puzpuzpuz commentedon Feb 1, 2020
This issue may be related with suboptimal behavior of
glibc, which may have large RSS values in such benchmarks. See #21973puzpuzpuz commentedon Mar 26, 2020
Closing this one, as it doesn't seem to be related with node or v8. Feel free to comment if you also experience the same behavior.