Repository navigation
parallel/test-worker-nearheaplimit-deadlock is flaky #50761
Copy link
Copy link
Closed as not planned
Labels
flaky-testIssues and PRs involving tests that fail intermittently in CI.Issues and PRs involving tests that fail intermittently in CI.staleIssues and PRs marked stale due to inactivity and scheduled for automatic closure.Issues and PRs marked stale due to inactivity and scheduled for automatic closure.
Description
Refs: nodejs/reliability#720
It seems the failure started about 3 days ago, around the time that #48655 landed. I cannot reproduce it locally however.
One peculiar bit I noticed from the error logs:
16:17:07 1: 0x104c0fb6c node::Abort() [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 2: 0x104c0fd6c node::ModifyCodeGenerationFromStrings(v8::Local<v8::Context>, v8::Local<v8::Value>, bool) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 3: 0x104daf0fc v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 4: 0x104f8fb50 v8::internal::Heap::stack() [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 5: 0x104fab008 v8::internal::LocalFactory::NumberToStringCacheHash(v8::internal::Tagged<v8::internal::Smi>) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
See full stack trace
16:17:07 not ok 3261 parallel/test-worker-nearheaplimit-deadlock
16:17:07 ---
16:17:07 duration_ms: 193.09600
16:17:07 severity: crashed
16:17:07 exitcode: -6
16:17:07 stack: |-
16:17:07
16:17:07 <--- Last few GCs --->
16:17:07
16:17:07 [15425:0x128098000] 20 ms: Mark-Compact 1.7 (6.0) -> 1.7 (6.0) MB, 1.12 / 0.00 ms (average mu = 0.312, current mu = 0.000) background allocation failure; GC in old space requested
16:17:07 [15425:0x128098000] 21 ms: Mark-Compact 1.7 (6.0) -> 1.7 (6.0) MB, 1.42 / 0.00 ms (average mu = 0.208, current mu = 0.055) background allocation failure; GC in old space requested
16:17:07
16:17:07
16:17:07 <--- JS stacktrace --->
16:17:07
16:17:07 FATAL ERROR: LocalHeap: allocation failed Allocation failed - JavaScript heap out of memory
16:17:07 ----- Native stack trace -----
16:17:07
16:17:07 1: 0x104c0fb6c node::Abort() [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 2: 0x104c0fd6c node::ModifyCodeGenerationFromStrings(v8::Local<v8::Context>, v8::Local<v8::Value>, bool) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 3: 0x104daf0fc v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 4: 0x104f8fb50 v8::internal::Heap::stack() [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 5: 0x104fab008 v8::internal::LocalFactory::NumberToStringCacheHash(v8::internal::Tagged<v8::internal::Smi>) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 6: 0x104f641c8 v8::internal::FactoryBase<v8::internal::LocalFactory>::NewByteArray(int, v8::internal::AllocationType) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 7: 0x104e01058 v8::internal::Handle<v8::internal::ByteArray> v8::internal::baseline::BytecodeOffsetTableBuilder::ToBytecodeOffsetTable<v8::internal::LocalIsolate>(v8::internal::LocalIsolate*) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 8: 0x104e00f2c v8::internal::baseline::BaselineCompiler::Build(v8::internal::LocalIsolate*) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 9: 0x104e17050 v8::internal::GenerateBaselineCode(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SharedFunctionInfo>) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 10: 0x104e657e8 v8::internal::Compiler::CompileSharedWithBaseline(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Compiler::ClearExceptionFlag, v8::internal::IsCompiledScope*) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 11: 0x104e65a9c v8::internal::Compiler::CompileBaseline(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ClearExceptionFlag, v8::internal::IsCompiledScope*) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 12: 0x104dfaf3c v8::internal::baseline::BaselineBatchCompiler::EnqueueFunction(v8::internal::Handle<v8::internal::JSFunction>) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 13: 0x104f1ce68 v8::internal::TieringManager::OnInterruptTick(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::CodeKind) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 14: 0x105338bd0 v8::internal::(anonymous namespace)::BytecodeBudgetInterruptWithStackCheck(v8::internal::Isolate*, v8::internal::Arguments<(v8::internal::ArgumentsType)0>&, v8::internal::CodeKind) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 15: 0x105337694 v8::internal::Runtime_BytecodeBudgetInterruptWithStackCheck_Ignition(int, unsigned long*, v8::internal::Isolate*) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 16: 0x1056d3954 Builtins_CEntry_Return1_ArgvOnStack_NoBuiltinExit [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 17: 0x1057aa5c4 Builtins_JumpLoopHandler [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 18: 0x105648b84 Builtins_InterpreterEntryTrampoline [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 19: 0x15000bb84
16:17:07 20: 0x15000ce8c
16:17:07 21: 0x105648b84 Builtins_InterpreterEntryTrampoline [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 22: 0x105648b84 Builtins_InterpreterEntryTrampoline [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 23: 0x1056468ac Builtins_JSEntryTrampoline [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 24: 0x105646594 Builtins_JSEntry [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 25: 0x104ef1740 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 26: 0x104ef0cc0 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 27: 0x104dc8b28 v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 28: 0x104bf2a50 node::builtins::BuiltinLoader::CompileAndCall(v8::Local<v8::Context>, char const*, node::Realm*) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 29: 0x104c822d4 node::PrincipalRealm::BootstrapRealm() [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 30: 0x104c80298 node::Realm::RunBootstrapping() [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 31: 0x104b40d98 node::CreateEnvironment(node::IsolateData*, v8::Local<v8::Context>, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, node::EnvironmentFlags::Flags, node::ThreadId, std::__1::unique_ptr<node::InspectorParentHandle, std::__1::default_delete<node::InspectorParentHandle> >) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 32: 0x104cc486c node::worker::Worker::Run() [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 33: 0x104cc7f50 node::worker::Worker::StartThread(v8::FunctionCallbackInfo<v8::Value> const&)::$_3::__invoke(void*) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
16:17:07 34: 0x181467878 _pthread_start [/usr/lib/system/libsystem_pthread.dylib]
16:17:07 35: 0x1814625e0 thread_start [/usr/lib/system/libsystem_pthread.dylib]
16:17:07 ...
It seems strange that the ModifyCodeGenerationFromStrings callback can get called from FatalProcessOutOfMemory. Even if this gets called for OOM somehow, that shouldn't do much unless source maps are enabled, in which case it would call into JavaScript and that's definitely going to crash in OOM.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
flaky-testIssues and PRs involving tests that fail intermittently in CI.Issues and PRs involving tests that fail intermittently in CI.staleIssues and PRs marked stale due to inactivity and scheduled for automatic closure.Issues and PRs marked stale due to inactivity and scheduled for automatic closure.
Activity
legendecas commentedon Nov 22, 2023
I think the incorrect backtrace is a problem related to calling backtrace in a [[noreturn]] function: #50849. As a matter of fact,
FatalProcessOutOfMemorydidn’t callModifyCodeGenerationFromStringCallbackafter all.mhdawson commentedon Nov 23, 2023
again - https://ci.nodejs.org/job/node-test-commit-osx-arm/nodes=osx11/14638/
src: fix backtrace with tail [[noreturn]] abort
src: make ModifyCodeGenerationFromStrings more robust
src: fix backtrace with tail [[noreturn]] abort
src: make ModifyCodeGenerationFromStrings more robust
joyeecheung commentedon Mar 16, 2024
This still shows up in the CI occasionally. I happened to get a stack trace locally:
joyeecheung commentedon Mar 16, 2024
The worker heap size before message port creation is roughly 2.2MB. I wonder if we should bump it up a little to avoid failures like this. The near heap limit callback is only invoked during GC, but if we just set the heap size to 0, chances are the heap limit is going to be hit straight away and V8 won't even be able to do a GC and will just go straight to OOM.
On the other hand we could probably also just...schedule a throw on the main thread in the OOM handler of the worker?
joyeecheung commentedon Mar 16, 2024
Hmm actually not, because the OOM handler doesn't allow returning. I guess we could just bump the limit up a little, and/or ask V8 to not crash the entire process when it's only the V8 heap running out?
joyeecheung commentedon Mar 16, 2024
I wonder if we should just allow the test to crash. In that case, V8 runs out of memory because it's performing some operation that forbids GC and therefore can't invoke our callbacks. Technically that means
ERR_WORKER_OUT_OF_MEMORYis just not going to work 100% of the time, but for user heaps maybe it's less likely for the crash to occur.src: fix backtrace with tail [[noreturn]] abort
src: make ModifyCodeGenerationFromStrings more robust
joyeecheung commentedon Apr 2, 2024
This may be potentially fixed by https://chromium-review.googlesource.com/c/v8/v8/+/5352288
github-actions commentedon May 27, 2026
This issue has been marked as stale due to 210 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.
github-actions commentedon Jun 27, 2026
This issue has been automatically closed after 30 days of inactivity following its stale status (no activity for a total of 240 days).
If this is still relevant, feel free to reopen it or leave a comment with additional details so we can continue the discussion.