Skip to content

Chromium database much smaller than it should be #8755

Closed
Closed
Chromium database much smaller than it should be#8755
Labels
C++bugSomething isn't working

Description

@Ludgershall
Ludgershall
opened on Apr 15, 2022

Hello,

I've created a CodeQL database of Chromium. I've used the official tutorial from Google (https://chromium.googlesource.com/chromium/src/+/main/docs/linux/build_instructions.md).
When it came to building Chromium, I've swapped the command autoninja -C out/Default chrome by codeql database create chromium-database --language=cpp --command="autoninja -C out/Default chrome" -J-Xmx26034M.
EDIT: Just to be clear, I've started compiling from scratch.

Everything built just fine but when I took a look at the final database, it looked a lot smaller that I think it should be (a folder of 268 items for 762MB).
I've opened the resulting DB in VSCode and sure enough plenty of files were missing.

I've included a picture of the available files/folders present when opening the DB in VSCode.
download

Can you point out a way to have a more complete database? Thanks

Configuration:
Ubuntu 20
CodeQL CLI v2.8.5

Activity

  1. Ludgershall
    added
    questionFurther information is requested
    on Apr 15, 2022

  2. aibaars commented on Apr 18, 2022

    @aibaars
    aibaars
    on Apr 18, 2022
    Contributor

    Thanks for the report @Ludgershall . To help debugging this issue, could you have a look in the log/build-tracer.log file for error messages that could explain what's going wrong? The file is probably too large to post as an attachment. It may also contain sensitive data (user name, paths) you might not want to share. Could you look for the phrases intercepted , /extractor and catastrophic error . This should give an idea of which processes were intercepted by CodeQL. For the analysis the work CodeQL needs to intercept all compiler calls (eg clang, gcc) and invoke the CodeQL C/C++ extractor from each of them to extract the information from the compiled source files and store them in the CodeQL database. The phrase catastrophic error usually indicates something going very wrong causing CodeQL to skip some source files. If you can post any error messages or other relevant information to this issue that would be greatly appreciated. Make sure to redact any sensitive information (in file paths) you don't want to share. Thanks!

  3. aibaars
    added
    bugSomething isn't working
    C++
    and removed
    questionFurther information is requested
    on Apr 18, 2022

  4. pooneh-nb commented on Apr 18, 2022

    @pooneh-nb
    pooneh-nb
    on Apr 18, 2022

    I can share the build-tracer.log here. It's uploaded to my github.

  5. aibaars commented on Apr 19, 2022

    @aibaars
    aibaars
    on Apr 19, 2022
    Contributor

    Thanks @pooneh-nb . I see there are over 35000 catastrophic error messages like language modes specified are incompatible. I'll pass the log file on to the team so they can have a closer look.

    @Ludgershall could you have a look at the build-tracer.log on your system as well and check if you're seeing a similar failure pattern?

  6. Ludgershall commented on Apr 19, 2022

    @Ludgershall
    Ludgershall
    on Apr 19, 2022
    Author

    I've got plenty of errors like the following

    1 catastrophic error detected in this compilation.
Compilation terminated.
Current directory: /home/micah/chromium/src/out/Default
Command: /home/micah/Downloads/codeql/cpp/tools/linux64/extractor --mimic '/home/micah/chromium/src/third_party/llvm-build/Release+Asserts/bin/clang' -MMD -MF 'obj/buildtools/third_party/libc++/libc++/vector.o.d' -DLIBCXX_BUILDING_LIBCXXABI -DDCHECK_ALWAYS_ON=1 -DUSE_UDEV -DUSE_AURA=1 -DUSE_GLIB=1 -DUSE_OZONE=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_GNU_SOURCE '-DCR_CLANG_REVISION="llvmorg-15-init-7040-g50de659a-3"' -DCOMPONENT_BUILD -D_LIBCPP_ABI_UNSTABLE -D_LIBCPP_ABI_VERSION=Cr -D_LIBCPP_ENABLE_NODISCARD -D_LIBCPP_DEBUG=0 -DCR_LIBCXX_REVISION=79a2e924d96e2fc1e4b937c42efd08898fa472d7 -DCR_SYSROOT_HASH=60354520bb7001d8d0288a2ab82cec7efec9fa3d -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -D_LIBCPP_BUILDING_LIBRARY -I../.. -Igen '-I../../buildtools/third_party/libc++' -fno-delete-null-pointer-checks -fno-ident -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -funwind-tables -fPIC -pthread -fcolor-diagnostics -fmerge-all-constants -fcrash-diagnostics-dir=../../tools/clang/crashreports -mllvm -instcombine-lower-dbg-declare=0 -ffp-contract=off -fcomplete-member-pointers -m64 -msse3 -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -ffile-compilation-dir=. -no-canonical-prefixes -ftrivial-auto-var-init=pattern -O0 -fno-omit-frame-pointer -gdwarf-4 -g2 -gdwarf-aranges -gsplit-dwarf -ggnu-pubnames -Xclang -fuse-ctor-homing -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-template-as-trivial-member -Xclang -plugin-arg-find-bad-constructs -Xclang use-classify-type -Xclang -plugin-arg-find-bad-constructs -Xclang check-ipc -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -fstrict-aliasing -fPIC -Werror -Wall -Wno-unused-variable '-Wno-c++11-narrowing' -Wno-unused-but-set-variable -Wno-misleading-indentation -Wno-missing-field-initializers -Wno-unused-parameter -Wloop-analysis -Wno-unneeded-internal-declaration -Wenum-compare-conditional -Wno-psabi -Wno-ignored-pragma-optimize -Wno-unqualified-std-cast-call -Wmax-tokens -fvisibility=default -Wno-undefined-bool-conversion -Wno-tautological-undefined-compare '-std=c++17' -Wno-trigraphs -fno-aligned-new '-nostdinc++' '-isystem../../buildtools/third_party/libc++/trunk/include' '-isystem../../buildtools/third_party/libc++abi/trunk/include' --sysroot=../../build/linux/debian_bullseye_amd64-sysroot '-std=c++20' -fexceptions -frtti -c '../../buildtools/third_party/libc++/trunk/src/vector.cpp' -o 'obj/buildtools/third_party/libc++/libc++/vector.o'
[T 11:50:24 92785] Extractor terminated with exit code 1.
[T 11:50:24 92795] Reading configuration file /home/micah/chromium/src/chromium-database/working/tracing/compiler-tracing12366888720025849700.spec...
[T 11:50:24 92795] Initialising tags...
[T 11:50:24 92795] ID set to 0000000000016A7B_0000000000000001 (parent 0000000000016731_0000000000000003)
[T 11:50:24 92795] ==== Candidate to intercept: /bin/sh (canonical: /usr/bin/dash) ====
[T 11:50:24 92796] Reading configuration file /home/micah/chromium/src/chromium-database/working/tracing/compiler-tracing12366888720025849700.spec...
[T 11:50:24 92796] Initialising tags...
[T 11:50:24 92796] ID set to 0000000000016A7C_0000000000000001 (parent 0000000000016A7B_0000000000000001)
[T 11:50:24 92796] ==== Candidate to intercept: ../../third_party/llvm-build/Release+Asserts/bin/clang++ (canonical: /home/micah/chromium/src/third_party/llvm-build/Release+Asserts/bin/clang) ====
[T 11:50:24 92796] === Intercepted call to /home/micah/chromium/src/third_party/llvm-build/Release+Asserts/bin/clang === 
[T 11:50:24 92798] Attempting to switch stdout/stderr to 3...
[T 11:50:24 92798] Reading configuration file /home/micah/chromium/src/chromium-database/working/tracing/compiler-tracing12366888720025849700.spec...
[T 11:50:24 92798] Initialising tags...
[T 11:50:24 92798] ID set to 0000000000016A7E_0000000000000001 (parent 0000000000016A7C_0000000000000001)
[E 11:50:24 92798] CodeQL C/C++ Extractor 2.7.5
[E 11:50:24 92798] Current directory: /home/micah/chromium/src/out/Default
[E 11:50:24 92798] Command: /home/micah/Downloads/codeql/cpp/tools/linux64/extractor --mimic '/home/micah/chromium/src/third_party/llvm-build/Release+Asserts/bin/clang' -MMD -MF 'obj/buildtools/third_party/libc++abi/libc++abi/cxa_aux_runtime.o.d' -DLIBCXXABI_SILENT_TERMINATE -D_LIBCPP_CONSTINIT=constinit -DDCHECK_ALWAYS_ON=1 -DUSE_UDEV -DUSE_AURA=1 -DUSE_GLIB=1 -DUSE_OZONE=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_GNU_SOURCE '-DCR_CLANG_REVISION="llvmorg-15-init-7040-g50de659a-3"' -DCOMPONENT_BUILD -D_LIBCPP_ABI_UNSTABLE -D_LIBCPP_ABI_VERSION=Cr -D_LIBCPP_ENABLE_NODISCARD -D_LIBCPP_DEBUG=0 -DCR_LIBCXX_REVISION=79a2e924d96e2fc1e4b937c42efd08898fa472d7 -DCR_SYSROOT_HASH=60354520bb7001d8d0288a2ab82cec7efec9fa3d -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -D_LIBCPP_BUILDING_LIBRARY '-I../../buildtools/third_party/libc++/trunk/src' -I../.. -Igen '-I../../buildtools/third_party/libc++' -fno-delete-null-pointer-checks -fno-ident -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -funwind-tables -fPIC -pthread -fcolor-diagnostics -fmerge-all-constants -fcrash-diagnostics-dir=../../tools/clang/crashreports -mllvm -instcombine-lower-dbg-declare=0 -ffp-contract=off -fcomplete-member-pointers -m64 -msse3 -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -ffile-compilation-dir=. -no-canonical-prefixes -ftrivial-auto-var-init=pattern -O0 -fno-omit-frame-pointer -gdwarf-4 -g2 -gdwarf-aranges -gsplit-dwarf -ggnu-pubnames -Xclang -fuse-ctor-homing -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-template-as-trivial-member -Xclang -plugin-arg-find-bad-constructs -Xclang use-classify-type -Xclang -plugin-arg-find-bad-constructs -Xclang check-ipc -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -Werror -Wall -Wno-unused-variable '-Wno-c++11-narrowing' -Wno-unused-but-set-variable -Wno-misleading-indentation -Wno-missing-field-initializers -Wno-unused-parameter -Wloop-analysis -Wno-unneeded-internal-declaration -Wenum-compare-conditional -Wno-psabi -Wno-ignored-pragma-optimize -Wno-unqualified-std-cast-call -Wmax-tokens -fstrict-aliasing -fPIC -fvisibility=default -Wno-undefined-bool-conversion -Wno-tautological-undefined-compare '-std=c++17' -Wno-trigraphs -fno-aligned-new '-nostdinc++' '-isystem../../buildtools/third_party/libc++/trunk/include' '-isystem../../buildtools/third_party/libc++abi/trunk/include' --sysroot=../../build/linux/debian_bullseye_amd64-sysroot -fexceptions -frtti '-std=c++20' -c '../../buildtools/third_party/libc++abi/trunk/src/cxa_aux_runtime.cpp' -o 'obj/buildtools/third_party/libc++abi/libc++abi/cxa_aux_runtime.o'
Warning: Unrecognised command line argument -mllvm
Warning: Unrecognised command line argument -instcombine-lower-dbg-declare=0
Warning: Unrecognised command line argument -mllvm
Warning: Unrecognised command line argument -instcombine-lower-dbg-declare=0
[E 11:50:24 92798] Checking whether C compilation already happened.
[E 11:50:24 92798] Checking for tag c-compilation-happened
[E 11:50:24 92798] Checking CODEQL_TRACER_DB_ID 0000000000016A7C_0000000000000001
[E 11:50:24 92798] Locking DB /home/micah/chromium/src/chromium-database/working/tags.db
[E 11:50:24 92798] Locked DB /home/micah/chromium/src/chromium-database/working/tags.db
[E 11:50:24 92798] Unlocking DB
[E 11:50:24 92798] Unlocked DB
[E 11:50:24 92798] Looks like C compilation didn't already happen.
[E 11:50:24 92798] Checking whether C compilation has been attempted.
[E 11:50:24 92798] Checking for tag c-compilation-attempted
[E 11:50:24 92798] Checking CODEQL_TRACER_DB_ID 0000000000016A7C_0000000000000001
[E 11:50:24 92798] Locking DB /home/micah/chromium/src/chromium-database/working/tags.db
[E 11:50:24 92798] Locked DB /home/micah/chromium/src/chromium-database/working/tags.db
[E 11:50:24 92798] Unlocking DB
[E 11:50:24 92798] Unlocked DB
[E 11:50:24 92798] Marking C compilation as attempted.
[E 11:50:24 92798] Setting tag c-compilation-attempted
[E 11:50:24 92798] Starting from CODEQL_TRACER_DB_ID 0000000000016A7C_0000000000000001
[E 11:50:24 92798] Locking DB /home/micah/chromium/src/chromium-database/working/tags.db
[E 11:50:24 92798] Locked DB /home/micah/chromium/src/chromium-database/working/tags.db
[E 11:50:24 92798] Set tag for0000000000016A7C_0000000000000001
[E 11:50:24 92798] Set tag for0000000000016A7B_0000000000000001
[E 11:50:24 92798] Set tag for0000000000016731_0000000000000003
[E 11:50:24 92798] Set tag for0000000000016731_0000000000000002
[E 11:50:24 92798] Set tag for0000000000016731_0000000000000001
[E 11:50:24 92798] Set tag for0000000000016728_0000000000000004
[E 11:50:24 92798] Set tag for0000000000016728_0000000000000003
[E 11:50:24 92798] Set tag for0000000000016728_0000000000000002
[E 11:50:24 92798] Set tag forroot
[E 11:50:24 92798] Unlocking DB
[E 11:50:24 92798] Unlocked DB
[E 11:50:24 92798] Processed command line: /home/micah/Downloads/codeql/cpp/tools/linux64/extractor --trapfolder /home/micah/chromium/src/chromium-database/trap/cpp --src_archive /home/micah/chromium/src/chromium-database/src --mimic_config /home/micah/chromium/src/chromium-database/working/compiler_mimic_cache/bc9026de7277 --object_filename 'obj/buildtools/third_party/libc++abi/libc++abi/cxa_aux_runtime.o' -w --error_limit 1000 --disable_system_macros --variadic_macros '--c++20' --gcc --gnu_version 40000 --clang -DLIBCXXABI_SILENT_TERMINATE -D_LIBCPP_CONSTINIT=constinit -DDCHECK_ALWAYS_ON=1 -DUSE_UDEV -DUSE_AURA=1 -DUSE_GLIB=1 -DUSE_OZONE=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_GNU_SOURCE '-DCR_CLANG_REVISION="llvmorg-15-init-7040-g50de659a-3"' -DCOMPONENT_BUILD -D_LIBCPP_ABI_UNSTABLE -D_LIBCPP_ABI_VERSION=Cr -D_LIBCPP_ENABLE_NODISCARD -D_LIBCPP_DEBUG=0 -DCR_LIBCXX_REVISION=79a2e924d96e2fc1e4b937c42efd08898fa472d7 -DCR_SYSROOT_HASH=60354520bb7001d8d0288a2ab82cec7efec9fa3d -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -D_LIBCPP_BUILDING_LIBRARY '-I../../buildtools/third_party/libc++/trunk/src' -I../.. -Igen '-I../../buildtools/third_party/libc++' -D__DATE__= -D__TIME__= -D__TIMESTAMP__= --isysroot ../../build/linux/debian_bullseye_amd64-sysroot --exceptions --rtti --sys_include '../../buildtools/third_party/libc++/trunk/include' --sys_include '../../buildtools/third_party/libc++abi/trunk/include' -- '../../buildtools/third_party/libc++abi/trunk/src/cxa_aux_runtime.cpp'
Command-line error: language modes specified are incompatible

  7. pooneh-nb commented on Apr 24, 2022

    @pooneh-nb
    pooneh-nb
    on Apr 24, 2022

    A quick update; I added a few arguments in the process of making ninja files using gn. So instead of running gn gen out/Default, I ran gn gen out/Default --args="is_component_build = false is_debug = false enable_nacl = false symbol_level = 0 blink_symbol_level = 0 v8_symbol_level = 0". Then created the codeql database codeql database create chromium-database --language=cpp --command="autoninja -C out/Default chrome”
    The result changes significantly. Now instead of 3k files, I have 85K files but it is still much less than the original number of files in the chromium source code. Here is the comparison:

      Original chormium Codeql src
    Total files in database 847.207 85.758
    Total .c, .cc, .cpp, .cxx files 109.927 35.468
    Total .h, .hxx, .hpp, .hh files 108.392 49.765

  8. ducphanduyagentp commented on Jul 23, 2022

    @ducphanduyagentp
    ducphanduyagentp
    on Jul 23, 2022

    I've recently experienced the same outcome. Is there any update on the situation? I can also share the error logs if it helps.

    Edit 1: I switched to CodeQL CLI 2.8.2 and add a few build arguments to chromium (is_debug=false is_component_build=false dcheck_always_on=false) and it seems to be generating the right amount of output, but I still experienced some catastrophic errors. I will try a few more combinations of CodeQL version and build configs to see what works and what doesn't. The first combination that didn't work was v2.10.1 (latest) with default build configuration from gn gen out/Default.

  9. xeroskim commented on Dec 30, 2022

    @xeroskim
    xeroskim
    on Dec 30, 2022

    Are there any updates on this problem? I'm facing the same issue.

  10. jketema commented on Jan 27, 2023

    @jketema
    jketema
    on Jan 27, 2023
    Contributor

    Hi all, the problem with language modes specified are incompatible should be fixed in CodeQL 2.12.2, which will be released in a few weeks time. I'm closing this issue, but feel free to re-open if you spot more language modes specified are incompatible problems. Please open new issues for any other problems related to missing files.

  11. jketema
    closed this as completedon Jan 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    C++bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions