-
Notifications
You must be signed in to change notification settings - Fork 1.9k
-
I am using CodeQL to analyse an open-source code base which uses./waf build for building at the root level. I created 3 databases for 3 different languages: C, C++, and Python, with some C and C++ packages, extra like MISRA, CERTC, etc . Also, I successfully created a DB for all of them. When I tried to analyse the codebase for the MISRA C package, it took me a lot of time (not analysed). All the queries were loaded, but all of them were queued, and my terminal stopped. My ram is 20gb and i5 processor.
So, to avoid this, I tried analysing the subdirectory present in the root, but couldn't do it
I tried
- Analyzing the existing database, but I couldn't find an option to select a subdirectory. If anything is there, please let me know
- Tried creating another DB forthe subdirectory, but there was no build command for the subdirectory to build
Also, if there are any other methods to try, please let me know
Beta Was this translation helpful? Give feedback.
All reactions
Replies: 1 comment 12 replies
-
I'm afraid you would indeed need to build a database out of some subset of your code, and the best way to achieve that is to run a build command that only builds part of your application. For example, if there's some large library that gets build (either your code or third-party but nontheless built locally) and then an application that uses it, you could build the library outside CodeQL and then build just the user application under CodeQL.
Alternatively you could make a database available to us so that we can analyse why the C/C++ queries are consuming too much memory. FWIW I would always expect CodeQL to report an out of memory situation rather than freeze entirely -- make sure your --ram setting is something like 75% of your available memory for a compromise between memory made available to the analysis and crashing with an out-of-memory message rather than exhaust system resources.
Beta Was this translation helpful? Give feedback.
All reactions
-
I tried to reproduce @Beeram12's work on a larger computer (56 cores, 2.5Ghz, 128 GB RAM). I configured with 50 threads and 32GB RAM. It ran overnight. top showed this for java when I killed it:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3324489 joel 20 0 40.7g 20.2g 240348 S 161.2 16.1 931:56.24 java
java is about 16 hours of CPU time there. The database is huge and at 236GB is too large to transfer in a reasonable length of time. Especially from hotel wifi.
$ du -s -h rtems-db/*
236G rtems-db/cpp
8.0K rtems-db/diagnostic
60M rtems-db/log
15M rtems-db/python
2.6M rtems-db/working
I have attached the nohup_out.txt in case that helps any.
I am on travel this week but will report on the smaller test case if @Beeram12 doesn't beat me to it.
Beta Was this translation helpful? Give feedback.
All reactions
-
@smowton
I ran queries for Python language it worked well, and I reported as well, but it's not the case for c/cpp packages
Beta Was this translation helpful? Give feedback.
All reactions
-
@Beeram12 thanks for sharing-- unfortunately I'm unable to download it from Google Drive. Could you please try--
- Prior to upload, delete the
logandresultsdirectories which seem to be very large, and runcodeql database cleanup --cache-cleanup=clear. - Please upload a zip file, as Google Drive struggles to zip up such a large object server-side.
Beta Was this translation helpful? Give feedback.
All reactions
-
Thanks for sharing the pared-down database; I was able to reproduce and make a candidate fix: github/codeql-coding-standards#907
Beta Was this translation helpful? Give feedback.
All reactions
-
That fix is queued to merge; the next release should resolve the issue and avoid needing to exclude that query.
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 1