Segmentation fault in v8::internal::compiler::(anonymous namespace)::MayAlias (10.15.2 from Debian Buster) #31484

Activity

1. addaleax

   added

   v8 engineIssues and PRs related to the V8 dependency.Issues and PRs related to the V8 dependency.

   on Jan 23, 2020
2. addaleax commented on Jan 23, 2020

   

   addaleax

   on Jan 23, 2020

   Member

   More actions

   Is there any chance you could share a reproduction? Can you try other Node.js versions and compare the result (including, ideally, verifying that the 10.15.2 binary from https://nodejs.org/en/ or e.g. nvm also exhibits the same issue)?
3. TimWolla commented on Jan 23, 2020

   

   TimWolla

   on Jan 23, 2020

   Author

   More actions

   Is there any chance you could share a reproduction?

   No. As I said it's a fairly complex application, containing a bunch of internal business logic. I started trying to rip out JavaScript code that I assume not to be executed within the staging system to reduce the footprint of the application, but so far without any real success. The biggest issue that I am not yet able to reliably trigger the segmentation fault. With an unmodified application it takes roughly 30 minutes to 1 hour to crash with the current traffic on that system.

   I can offer to poke around within the core file. I might be able to provide the core file to a nodejs core developer in private. I would need to check with my manager and to clean up the configuration to not leak any private information, such as TLS certificates, though.

   Can you try other Node.js versions and compare the result (including, ideally, verifying that the 10.15.2 binary from https://nodejs.org/en/ or e.g. nvm also exhibits the same issue)?

   Yes, I can try that. I just downloaded https://nodejs.org/download/release/v10.15.2/node-v10.15.2-linux-x64.tar.gz onto the machine, adjusted the unit file and restarted the service. For my record: As of 23:53:34 UTC the new binary is running with an unmodified application.
4. TimWolla commented on Jan 24, 2020

   

   TimWolla

   on Jan 24, 2020

   Last edited by TimWolla

   Author

   More actions

   For my record: As of 23:53:34 UTC the new binary is running with an unmodified application.

   That was quick. I started a traffic generator in parallel and the application crashed at 23:59:14 UTC.

   Find below the stack trace with the node binary downloaded from nodejs.org:

   Core was generated by `/original-node/node-v10.15.2-linux-x64/bin/node dist/server.js'.
   Program terminated with signal SIGSEGV, Segmentation fault.
   #0  0x0000000000cfe112 in v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*) [clone .part.54] ()
   [Current thread is 1 (Thread 0x7f269affd700 (LWP 24775))]
   (gdb) bt
   #0  0x0000000000cfe112 in v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*) [clone .part.54] ()
   #1  0x0000000000d0188c in v8::internal::compiler::LoadElimination::AbstractField::Kill(v8::internal::compiler::LoadElimination::AliasStateInfo const&, v8::internal::MaybeHandle<v8::internal::Name>, v8::internal::Zone*) const ()
   #2  0x0000000000d03e84 in v8::internal::compiler::LoadElimination::ReduceStoreField(v8::internal::compiler::Node*) ()
   #3  0x0000000000d08ba5 in v8::internal::compiler::LoadElimination::Reduce(v8::internal::compiler::Node*) ()
   #4  0x0000000000d4a9c8 in v8::internal::compiler::(anonymous namespace)::SourcePositionWrapper::Reduce(v8::internal::compiler::Node*) ()
   #5  0x0000000000c7ac5e in v8::internal::compiler::GraphReducer::Reduce(v8::internal::compiler::Node*) ()
   #6  0x0000000000c7b051 in v8::internal::compiler::GraphReducer::ReduceTop() ()
   #7  0x0000000000c7b4e1 in v8::internal::compiler::GraphReducer::ReduceNode(v8::internal::compiler::Node*) ()
   #8  0x0000000000d56bb6 in v8::internal::compiler::LoadEliminationPhase::Run(v8::internal::compiler::PipelineData*, v8::internal::Zone*) ()
   #9  0x0000000000d57b10 in v8::internal::compiler::PipelineImpl::OptimizeGraph(v8::internal::compiler::Linkage*) ()
   #10 0x0000000000d57b50 in v8::internal::compiler::PipelineCompilationJob::ExecuteJobImpl() ()
   #11 0x0000000000c109b1 in v8::internal::OptimizedCompilationJob::ExecuteJob() ()
   #12 0x0000000000c0b3f0 in v8::internal::OptimizingCompileDispatcher::CompileTask::RunInternal() ()
   #13 0x0000000000bc65f6 in v8::internal::CancelableTask::Run() ()
   #14 0x0000000000962bb9 in node::BackgroundRunner(void*) ()
   #15 0x00007f26a1292fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
   #16 0x00007f26a11c14cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
   

   Edit: After saving the core file and restarting the traffic generator the application crashed at 00:03:13 UTC.
5. bnoordhuis commented on Jan 24, 2020

   

   bnoordhuis

   on Jan 24, 2020

   Member

   More actions

   Can you open the core dump with gdb and post the output of info registers and disassemble? Thanks.
6. TimWolla commented on Jan 24, 2020

   

   TimWolla

   on Jan 24, 2020

   Author

   More actions

   Sure. Disassembly cut at some location after the current instruction that still fit on my terminal. I can give more disassembly if it is helpful.

   From the Debian binary

   (gdb) info registers
   rax            0x4670001           73859073
   rbx            0x0                 0
   rcx            0x0                 0
   rdx            0x27                39
   rsi            0x0                 0
   rdi            0x221fc90           35781776
   rbp            0x7fe93fffe130      0x7fe93fffe130
   rsp            0x7fe93fffe110      0x7fe93fffe110
   r8             0x4                 4
   r9             0x7fe93407a250      140639577023056
   r10            0x7fe93407eea8      140639577042600
   r11            0x1                 1
   r12            0x221fc90           35781776
   r13            0x1f2e570           32695664
   r14            0x213d3f8           34853880
   r15            0x7fe934079698      140639577020056
   rip            0x7fe9488dcb6b      0x7fe9488dcb6b <v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*)+43>
   eflags         0x10206             [ PF IF RF ]
   cs             0x33                51
   ss             0x2b                43
   ds             0x0                 0
   es             0x0                 0
   fs             0x0                 0
   gs             0x0                 0
   k0             0x0                 0
   k1             0x0                 0
   k2             0x0                 0
   k3             0x0                 0
   k4             0x0                 0
   k5             0x0                 0
   k6             0x0                 0
   k7             0x0                 0
   (gdb) disassemble
   Dump of assembler code for function v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*):
      0x00007fe9488dcb40 <+0>:	push   %rbp
      0x00007fe9488dcb41 <+1>:	mov    %rsp,%rbp
      0x00007fe9488dcb44 <+4>:	push   %r12
      0x00007fe9488dcb46 <+6>:	push   %rbx
      0x00007fe9488dcb47 <+7>:	sub    $0x10,%rsp
      0x00007fe9488dcb4b <+11>:	mov    %fs:0x28,%rax
      0x00007fe9488dcb54 <+20>:	mov    %rax,-0x18(%rbp)
      0x00007fe9488dcb58 <+24>:	xor    %eax,%eax
      0x00007fe9488dcb5a <+26>:	mov    $0x1,%eax
      0x00007fe9488dcb5f <+31>:	cmp    %rsi,%rdi
      0x00007fe9488dcb62 <+34>:	je     0x7fe9488dcbc0 <v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*)+128>
      0x00007fe9488dcb64 <+36>:	mov    0x8(%rdi),%rax
      0x00007fe9488dcb68 <+40>:	mov    %rsi,%rbx
   => 0x00007fe9488dcb6b <+43>:	mov    0x8(%rsi),%rsi
      0x00007fe9488dcb6f <+47>:	mov    %rdi,%r12
      0x00007fe9488dcb72 <+50>:	lea    -0x20(%rbp),%rdi
      0x00007fe9488dcb76 <+54>:	mov    %rax,-0x20(%rbp)
      0x00007fe9488dcb7a <+58>:	callq  0x7fe9484a74b0 <_ZNK2v88internal8compiler4Type5MaybeES2_@plt>
      0x00007fe9488dcb7f <+63>:	test   %al,%al
      0x00007fe9488dcb81 <+65>:	je     0x7fe9488dcbc0 <v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*)+128>
      0x00007fe9488dcb83 <+67>:	mov    (%rbx),%rdx
      0x00007fe9488dcb86 <+70>:	movzwl 0x10(%rdx),%edx
      0x00007fe9488dcb8a <+74>:	cmp    $0x3a,%dx
      0x00007fe9488dcb8e <+78>:	je     0x7fe9488dcc20 <v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*)+224>
      0x00007fe9488dcb94 <+84>:	cmp    $0xca,%dx
      0x00007fe9488dcb99 <+89>:	je     0x7fe9488dcc40 <v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*)+256>
      0x00007fe9488dcb9f <+95>:	cmp    $0x27,%dx
      0x00007fe9488dcba3 <+99>:	je     0x7fe9488dcc20 <v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*)+224>
   

   from the nodejs.org binary

   (gdb) info registers
   rax            0x4670001           73859073
   rbx            0x331d740           53598016
   rcx            0x331d760           53598048
   rdx            0x4670001           73859073
   rsi            0x3125940           51534144
   rdi            0x331d740           53598016
   rbp            0x7f269affc300      0x7f269affc300
   rsp            0x7f269affc2e0      0x7f269affc2e0
   r8             0x340cfc0           54579136
   r9             0x340ce98           54578840
   r10            0x0                 0
   r11            0x1                 1
   r12            0x0                 0
   r13            0x3125940           51534144
   r14            0x331d740           53598016
   r15            0x7f269407f458      139803669034072
   rip            0xcfe112            0xcfe112 <v8::internal::compiler::(anonymous namespace)::MayAlias(v8::internal::compiler::Node*, v8::internal::compiler::Node*) [clone .part.54]+130>
   eflags         0x10202             [ IF RF ]
   cs             0x33                51
   ss             0x2b                43
   ds             0x0                 0
   es             0x0                 0
   fs             0x0                 0
   gs             0x0                 0
   k0             0x0                 0
   k1             0x0                 0
   k2             0x0                 0
   k3             0x0                 0
   k4             0x0                 0
   k5             0x0                 0
   k6             0x0                 0
   k7             0x0                 0
   (gdb) disassemble
   Dump of assembler code for function _ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54:
      0x0000000000cfe090 <+0>:	push   %rbp
      0x0000000000cfe091 <+1>:	mov    %rsp,%rbp
      0x0000000000cfe094 <+4>:	push   %r12
      0x0000000000cfe096 <+6>:	push   %rbx
      0x0000000000cfe097 <+7>:	mov    %rdi,%rbx
      0x0000000000cfe09a <+10>:	sub    $0x10,%rsp
      0x0000000000cfe09e <+14>:	mov    (%rsi),%rax
      0x0000000000cfe0a1 <+17>:	movzwl 0x10(%rax),%eax
      0x0000000000cfe0a5 <+21>:	cmp    $0x3a,%ax
      0x0000000000cfe0a9 <+25>:	je     0xcfe0f0 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+96>
      0x0000000000cfe0ab <+27>:	cmp    $0xca,%ax
      0x0000000000cfe0af <+31>:	je     0xcfe198 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+264>
      0x0000000000cfe0b5 <+37>:	cmp    $0x27,%ax
      0x0000000000cfe0b9 <+41>:	je     0xcfe0f0 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+96>
      0x0000000000cfe0bb <+43>:	mov    (%rdi),%rdx
      0x0000000000cfe0be <+46>:	movzwl 0x10(%rdx),%edx
      0x0000000000cfe0c2 <+50>:	cmp    $0x3a,%dx
      0x0000000000cfe0c6 <+54>:	je     0xcfe1e0 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+336>
      0x0000000000cfe0cc <+60>:	cmp    $0xca,%dx
      0x0000000000cfe0d1 <+65>:	je     0xcfe1c8 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+312>
      0x0000000000cfe0d7 <+71>:	cmp    $0x27,%dx
      0x0000000000cfe0db <+75>:	je     0xcfe1e0 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+336>
      0x0000000000cfe0e1 <+81>:	mov    $0x1,%eax
      0x0000000000cfe0e6 <+86>:	add    $0x10,%rsp
      0x0000000000cfe0ea <+90>:	pop    %rbx
      0x0000000000cfe0eb <+91>:	pop    %r12
      0x0000000000cfe0ed <+93>:	pop    %rbp
      0x0000000000cfe0ee <+94>:	retq   
      0x0000000000cfe0ef <+95>:	nop
      0x0000000000cfe0f0 <+96>:	movzbl 0x17(%rsi),%eax
      0x0000000000cfe0f4 <+100>:	and    $0xf,%eax
      0x0000000000cfe0f7 <+103>:	cmp    $0xf,%eax
      0x0000000000cfe0fa <+106>:	je     0xcfe200 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+368>
      0x0000000000cfe100 <+112>:	mov    0x20(%rsi),%r12
      0x0000000000cfe104 <+116>:	cmp    %r12,%rbx
      0x0000000000cfe107 <+119>:	mov    $0x1,%eax
      0x0000000000cfe10c <+124>:	je     0xcfe0e6 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+86>
      0x0000000000cfe10e <+126>:	mov    0x8(%rbx),%rax
   => 0x0000000000cfe112 <+130>:	mov    0x8(%r12),%rsi
      0x0000000000cfe117 <+135>:	lea    -0x20(%rbp),%rdi
      0x0000000000cfe11b <+139>:	mov    %rax,-0x20(%rbp)
      0x0000000000cfe11f <+143>:	callq  0xdce240 <_ZNK2v88internal8compiler4Type5MaybeES2_>
      0x0000000000cfe124 <+148>:	test   %al,%al
      0x0000000000cfe126 <+150>:	je     0xcfe0e6 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+86>
      0x0000000000cfe128 <+152>:	mov    (%r12),%rdx
      0x0000000000cfe12c <+156>:	movzwl 0x10(%rdx),%edx
      0x0000000000cfe130 <+160>:	cmp    $0x3a,%dx
      0x0000000000cfe134 <+164>:	je     0xcfe238 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+424>
      0x0000000000cfe13a <+170>:	cmp    $0xca,%dx
      0x0000000000cfe13f <+175>:	je     0xcfe210 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+384>
      0x0000000000cfe145 <+181>:	cmp    $0x27,%dx
      0x0000000000cfe149 <+185>:	je     0xcfe238 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+424>
      0x0000000000cfe14f <+191>:	mov    (%rbx),%rcx
      0x0000000000cfe152 <+194>:	movzwl 0x10(%rcx),%ecx
      0x0000000000cfe156 <+198>:	cmp    $0x3a,%cx
      0x0000000000cfe15a <+202>:	je     0xcfe171 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+225>
      0x0000000000cfe15c <+204>:	cmp    $0xca,%cx
      0x0000000000cfe161 <+209>:	je     0xcfe270 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+480>
      0x0000000000cfe167 <+215>:	cmp    $0x27,%cx
      0x0000000000cfe16b <+219>:	jne    0xcfe0e6 <_ZN2v88internal8compiler12_GLOBAL__N_18MayAliasEPNS1_4NodeES4_.part.54+86>
      0x0000000000cfe171 <+225>:	movzbl 0x17(%rbx),%eax
   

   Not sure why the disassembly shows mangled function names.
7. bnoordhuis commented on Jan 24, 2020

   

   bnoordhuis

   on Jan 24, 2020

   Member

   More actions

   Thanks. So in both cases it crashes because the second function argument is a nullptr. What happens when you start node with --noturbo_load_elimination?

   Is checking with the latest v12.x or v13.x an option for you? v10.x is still at V8 6.8 and that's positively ancient by now.

   The bug may have been fixed in a newer release. v8/v8@b28637b seems like a good candidate.
8. TimWolla commented on Jan 24, 2020

   

   TimWolla

   on Jan 24, 2020

   Author

   More actions

   What happens when you start node with --noturbo_load_elimination?

   Started the unmodified application at 13:40:45 UTC using /original-node/node-v10.15.2-linux-x64/bin/node --noturbo_load_elimination dist/server.js.

   Is checking with the latest v12.x or v13.x an option for you?

   I can check within the staging system with arbitrary node versions (I already downloaded some random precompiled binaries from the Internet upon request from addaleax). For the production system I would have to check with my manager, not using the Debian Packages comes with quite a bit of operational complexity I'd like to avoid.
9. TimWolla commented on Jan 24, 2020

   

   TimWolla

   on Jan 24, 2020

   Author

   More actions

   It's 21:44:50 UTC now. The process is now running since roughly 8 hours without the crash by specifying the --noturbo_load_elimination option. Previously it would more or less reliably crash every 30 minutes to 1 hour. I guess it's safe to say that the option works around the issue. I don't expect a noticeable performance hit, so this might be the best solution for production.

   While researching the command line flag I also stumbled upon --trace-turbo-load-elimination. Would you consider it useful to enable this option for debugging? I suppose it would log whenever that specific optimization kicks in?
10. bnoordhuis commented on Jan 25, 2020

    

    bnoordhuis

    on Jan 25, 2020

    Member

    More actions

    As a quick sanity check, let's see if the regression test from v8/v8@b28637b triggers a crash.

    https://ci.nodejs.org/job/node-test-commit-v8-linux/2817/
11. bnoordhuis commented on Jan 25, 2020

    

    bnoordhuis

    on Jan 25, 2020

    Member

    More actions

    It does so I think we've found our culprit. I've opened #31507 to back-port the fix to v10.x.
12. TimWolla commented on Jan 25, 2020

    

    TimWolla

    on Jan 25, 2020

    Author

    More actions

    FWIW: The CI appears to be behind OAuth I did not attempt to authorize. Not sure whether it was meant for me or just for tracking it yourself.

    In any case I trust your judgement here. Thanks for the quick turn around time. If you'd like me to test some special build to verify I'm happy to do so if you give me instructions.
13. bnoordhuis commented on Jan 26, 2020

    

    bnoordhuis

    on Jan 26, 2020

    Member

    More actions

    For us both, really. :-)

    If you could build from source and test with the patch, that'd of course be great. Second-best is if you can try with the next v10.x release because I don't know how often Debian updates their node package.
14. TimWolla commented on Jan 26, 2020

    

    TimWolla

    on Jan 26, 2020

    Author

    More actions

    If you could build from source and test with the patch, that'd of course be great.

    I compiled my last node.js from source back in the 0.10.x days. I'll have a look.

    because I don't know how often Debian updates their node package.

    Not at all. Debian Buster never got an updated nodejs since 2019-04-17 (https://tracker.debian.org/pkg/nodejs). Not even for backported security fixes which is odd. I plan to file a bug with them to get the patch backported to Debian stable once it is verified fixed, though.
15. 3 remaining items

    Load more

16. bnoordhuis commented on Jan 27, 2020

    

    bnoordhuis

    on Jan 27, 2020

    Member

    More actions

    Hm, that's sad news. If you're up for it, a debug build (make -j8 -C out BUILDTYPE=Debug - binary at out/Debug/node) might help catch the bug earlier or produce more meaningful stack traces.

    I ran the regression test from the pull request with Debian's node, self compiled node w/o patch and self compiled w/ patch. None of those crashed.

    Yes, that's expected. Node does a lot of JS bootstrapping that d8, the V8 shell, doesn't - and perturbs the environment the test runs in. Many of V8's regression tests rely on a pristine environment.

    ./configure --enable-d8 && make -j8 builds d8 if you want to try it out.
17. TimWolla commented on Jan 27, 2020

    

    TimWolla

    on Jan 27, 2020

    Author

    More actions

    If you're up for it, a debug build (make -j8 -C out BUILDTYPE=Debug - binary at out/Debug/node) might help catch the bug earlier or produce more meaningful stack traces.

    After adding 3G of swap file to that 1G of memory VM it compiled without being killed due to an out of memory condition and spat out 4.7G of artifacts, filling up the disk to ~100%.

    That said: The application is now running on a debug binary … I hope.
18. TimWolla commented on Jan 27, 2020

    

    TimWolla

    on Jan 27, 2020

    Author

    More actions

    Okay, it took 34 minutes for the Debug binary to crash. Compiled as make -C out BUILDTYPE=Debug with g++, copied just the node binary into a different folder.

    Core was generated by `/my-node/node dist/server.js'.
    Program terminated with signal SIGSEGV, Segmentation fault.
    #0  0x0000564d776eb69a in v8::internal::compiler::Node::type (this=0x0) at ../deps/v8/src/compiler/node.h:267
    267	  Type type() const { return type_; }
    [Current thread is 1 (Thread 0x7ff812af3700 (LWP 1795))]
    (gdb) bt
    #0  0x0000564d776eb69a in v8::internal::compiler::Node::type (this=0x0) at ../deps/v8/src/compiler/node.h:267
    #1  0x0000564d776eb6fd in v8::internal::compiler::NodeProperties::IsTyped (node=0x0) at ../deps/v8/src/compiler/node-properties.h:186
    #2  0x0000564d776eb72a in v8::internal::compiler::NodeProperties::GetType (node=0x0) at ../deps/v8/src/compiler/node-properties.h:188
    #3  0x0000564d778027e5 in v8::internal::compiler::(anonymous namespace)::MayAlias (a=0x564d7a5977d0, b=0x0) at ../deps/v8/src/compiler/load-elimination.cc:39
    #4  0x0000564d77802874 in v8::internal::compiler::(anonymous namespace)::MayAlias (a=0x564d7a5977d0, b=0x564d7a25e9b0) at ../deps/v8/src/compiler/load-elimination.cc:56
    #5  0x0000564d778051ec in v8::internal::compiler::LoadElimination::AliasStateInfo::MayAlias (this=0x7ff812af2320, other=0x564d7a25e9b0) at ../deps/v8/src/compiler/load-elimination.cc:679
    #6  0x0000564d77803db9 in v8::internal::compiler::LoadElimination::AbstractField::Kill (this=0x7ff7fc089810, alias_info=..., name=..., zone=0x564d7a4da140) at ../deps/v8/src/compiler/load-elimination.cc:372
    #7  0x0000564d778050b0 in v8::internal::compiler::LoadElimination::AbstractState::KillFields (this=0x7ff7fc08a780, object=0x564d7a5977d0, name=..., zone=0x564d7a4da140)
        at ../deps/v8/src/compiler/load-elimination.cc:654
    #8  0x0000564d7780677a in v8::internal::compiler::LoadElimination::ReduceStoreField (this=0x7ff812af28f0, node=0x564d7a5fe5a0) at ../deps/v8/src/compiler/load-elimination.cc:979
    #9  0x0000564d77802d26 in v8::internal::compiler::LoadElimination::Reduce (this=0x7ff812af28f0, node=0x564d7a5fe5a0) at ../deps/v8/src/compiler/load-elimination.cc:131
    #10 0x0000564d7786bfcb in v8::internal::compiler::(anonymous namespace)::SourcePositionWrapper::Reduce (this=0x564d7a634648, node=0x564d7a5fe5a0) at ../deps/v8/src/compiler/pipeline.cc:651
    #11 0x0000564d77735c77 in v8::internal::compiler::GraphReducer::Reduce (this=0x7ff812af2b30, node=0x564d7a5fe5a0) at ../deps/v8/src/compiler/graph-reducer.cc:85
    #12 0x0000564d77736107 in v8::internal::compiler::GraphReducer::ReduceTop (this=0x7ff812af2b30) at ../deps/v8/src/compiler/graph-reducer.cc:152
    #13 0x0000564d777359fe in v8::internal::compiler::GraphReducer::ReduceNode (this=0x7ff812af2b30, node=0x564d7a3d3828) at ../deps/v8/src/compiler/graph-reducer.cc:56
    #14 0x0000564d77735bb6 in v8::internal::compiler::GraphReducer::ReduceGraph (this=0x7ff812af2b30) at ../deps/v8/src/compiler/graph-reducer.cc:78
    #15 0x0000564d7786fb2e in v8::internal::compiler::LoadEliminationPhase::Run (this=0x7ff812af2c6f, data=0x564d7a193cd8, temp_zone=0x564d7a4da140) at ../deps/v8/src/compiler/pipeline.cc:1457
    #16 0x0000564d7787473f in v8::internal::compiler::PipelineImpl::Run<v8::internal::compiler::LoadEliminationPhase> (this=0x564d7a193e58) at ../deps/v8/src/compiler/pipeline.cc:1037
    #17 0x0000564d778715e9 in v8::internal::compiler::PipelineImpl::OptimizeGraph (this=0x564d7a193e58, linkage=0x564d7a1453d0) at ../deps/v8/src/compiler/pipeline.cc:1912
    #18 0x0000564d7786ca87 in v8::internal::compiler::PipelineCompilationJob::ExecuteJobImpl (this=0x564d7a193b20) at ../deps/v8/src/compiler/pipeline.cc:847
    #19 0x0000564d77662c2c in v8::internal::OptimizedCompilationJob::ExecuteJob (this=0x564d7a193b20) at ../deps/v8/src/compiler.cc:223
    #20 0x0000564d7765882f in v8::internal::OptimizingCompileDispatcher::CompileNext (this=0x564d7a02e690, job=0x564d7a193b20) at ../deps/v8/src/compiler-dispatcher/optimizing-compile-dispatcher.cc:118
    #21 0x0000564d776584c6 in v8::internal::OptimizingCompileDispatcher::CompileTask::RunInternal (this=0x564d7a228c40) at ../deps/v8/src/compiler-dispatcher/optimizing-compile-dispatcher.cc:69
    #22 0x0000564d7758732f in v8::internal::CancelableTask::Run (this=0x564d7a228c40) at ../deps/v8/src/cancelable-task.h:148
    #23 0x0000564d771cd011 in node::BackgroundRunner (data=0x564d79fbc640) at ../src/node_platform.cc:42
    #24 0x00007ff8144c4fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
    #25 0x00007ff8143f34cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
    (gdb) info registers
    rax            0x0                 0
    rbx            0x7ff7fc08fa00      140703062096384
    rcx            0x0                 0
    rdx            0x0                 0
    rsi            0x0                 0
    rdi            0x0                 0
    rbp            0x7ff812af2140      0x7ff812af2140
    rsp            0x7ff812af2140      0x7ff812af2140
    r8             0x7ff7fc08f6a0      140703062095520
    r9             0x1                 1
    r10            0x7c                124
    r11            0x1                 1
    r12            0x564d7a4d5c60      94890764360800
    r13            0x7fff36c0f2df      140734112002783
    r14            0x7ff812af3700      140703442089728
    r15            0x0                 0
    rip            0x564d776eb69a      0x564d776eb69a <v8::internal::compiler::Node::type() const+12>
    eflags         0x10206             [ PF IF RF ]
    cs             0x33                51
    ss             0x2b                43
    ds             0x0                 0
    es             0x0                 0
    fs             0x0                 0
    gs             0x0                 0
    k0             0x0                 0
    k1             0x0                 0
    k2             0x0                 0
    k3             0x0                 0
    k4             0x0                 0
    k5             0x0                 0
    k6             0x0                 0
    k7             0x0                 0
    (gdb) disassemble
    Dump of assembler code for function v8::internal::compiler::Node::type() const:
       0x0000564d776eb68e <+0>:	push   %rbp
       0x0000564d776eb68f <+1>:	mov    %rsp,%rbp
       0x0000564d776eb692 <+4>:	mov    %rdi,-0x8(%rbp)
       0x0000564d776eb696 <+8>:	mov    -0x8(%rbp),%rax
    => 0x0000564d776eb69a <+12>:	mov    0x8(%rax),%rax
       0x0000564d776eb69e <+16>:	pop    %rbp
       0x0000564d776eb69f <+17>:	retq   
    End of assembler dump.
    (gdb) list
    262	
    263	  // Only NodeProperties should manipulate the op.
    264	  void set_op(const Operator* op) { op_ = op; }
    265	
    266	  // Only NodeProperties should manipulate the type.
    267	  Type type() const { return type_; }
    268	  void set_type(Type type) { type_ = type; }
    269	
    270	  // Only NodeMarkers should manipulate the marks on nodes.
    271	  Mark mark() const { return mark_; }
    
19. TimWolla commented on Jan 27, 2020

    

    TimWolla

    on Jan 27, 2020

    Author

    More actions

    Find attached the output of the Debug node binary running with --trace-turbo-load-elimination. It shows the last seconds of output before the crash. Any private log output by the application itself has been redacted and replaced with *snip*. I have the core dump on record (the stack matches my previous comment).

    turbofan_log.txt
20. bnoordhuis commented on Feb 1, 2020

    

    bnoordhuis

    on Feb 1, 2020

    Member

    More actions

    Thanks, I can see the general shape of the bug now.

    The fourth stack frame is a return MayAlias(a, b->InputAt(0)); that I'm reasonably sure ought to be guarded on if (!b->IsDead()). There are a couple of similar calls that I need to check.

    If you're up for it, I can write a patch for you to try out.
21. TimWolla commented on Feb 1, 2020

    

    TimWolla

    on Feb 1, 2020

    Author

    More actions

    If you're up for it, I can write a patch for you to try out.

    Sure, I'm happy to test the patches you throw at me. It's not too much effort for me to test them, most of the time is spent waiting to find whether it crashes or not.
22. bnoordhuis commented on Feb 1, 2020

    

    bnoordhuis

    on Feb 1, 2020

    Member

    More actions

    https://github.com/bnoordhuis/io.js/commit/473868ecc2.patch

    Passes make test locally. Let's see if it also passes V8's test suite: https://ci.nodejs.org/job/node-test-commit-v8-linux/2834/
23. TimWolla commented on Feb 1, 2020

    

    TimWolla

    on Feb 1, 2020

    Author

    More actions

    I've applied that patch and recompiled node.js. The process was started with the new binary at 14:31:34 UTC. Let's wait and see.
24. TimWolla commented on Feb 1, 2020

    

    TimWolla

    on Feb 1, 2020

    Author

    More actions

    It's 23:00:00 UTC now. The process is running fine since 8.5 hours. I guess it's safe to say that the patch indeed fixes my issue. Thanks.
25. bnoordhuis

    added a commit that references this issue on Feb 2, 2020

    v8: fix load elimination liveness checks

    ...

    2b4f214
26. bnoordhuis commented on Feb 2, 2020

    

    bnoordhuis

    on Feb 2, 2020

    Member

    More actions

    Thanks for testing. I've opened #31613.

    👍React with 👍1Reacted by Beth Griggs
27. BethGriggs

    added a commit that references this issue on Feb 26, 2020

    v8: fix load elimination liveness checks

    ...

    dc61e09
28. BethGriggs commented on Jun 25, 2020

    

    BethGriggs

    on Jun 25, 2020

    Member

    More actions

    @TimWolla, #31613 landed in v10.20.0 - so this issue is now hopefully resolved. I'm going to close this out, but please let us know if you're still seeing this issue and we can reopen.

    ❤️React with ❤️1Reacted by Tim Düsterhus
29. BethGriggs

    closed this as completedon Jun 25, 2020