Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Help: Oops! A fatal internal error occurred (ArrayIndexOutOfBoundsException) #16888

Answered by aschackmull
mindcrunch4u asked this question in Q&A
Discussion options

Hello!

I am running the cpp-security-extended.qls on a 20GB database

  • CodeQL-CLI: version 2.17.4
  • command:
    • codeql database analyze codeql_db path/to/1.0.0/codeql-suites/cpp-security-extended.qls --format=csv --output =20240629_result.csv --threads 32 --ram 102400
    • codeql database analyze codeql_db path/to/1.0.0/codeql-suites/cpp-security-extended.qls --format=csv --output =20240629_result.csv

Both commands (with or without threads and ram options) had the same result.

The output (without --threads option, and --ram set to 102400):

(omitted)
[83/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Best Practices/Likely Errors/CommaBeforeMisleadingIndentation.ql.
[84/91] Loaded /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Critical/MissingCheckScanf.qlx.
[85/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Critical/SizeCheck.ql.
[86/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Critical/SizeCheck2.ql.
[87/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Diagnostics/FailedExtractorInvocations.ql.
[88/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Diagnostics/ExtractionWarnings.ql.
[89/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Diagnostics/ExtractedFiles.ql.
[90/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Summary/LinesOfCode.ql.
[91/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Summary/LinesOfUserCode.ql.
Starting evaluation of codeql/cpp-queries/Critical/DoubleFree.ql.
Oops! A fatal internal error occurred. Details:
com.semmle.util.exception.CatastrophicError: An error occurred while evaluating DataFlowImplCommon::Cached::edgeOrder/1#6d956c05/2@43a39avp
java.lang.ArrayIndexOutOfBoundsException: arraycopy: last destination index 1506635235 out of bounds for long[224938403]
The RA to evaluate was:
 {2} r1 = AGGREGATE `_DataFlowPrivate::DataFlowCallable.totalorder/0#dispred#48dd4ff9__DataFlowImplCommon::Cached::TMkCal__#rank_range`, `_DataFlowPrivate::DataFlowCallable.totalorder/0#dispred#48dd4ff9__DataFlowImplCommon::Cached::TMkCal__#rank_term` ON In.5, In.6, In.7 WITH RANK<1 ASC2 ASC> OUTPUT , Agg.0, Agg.1
 return r1
(eventual cause: ArrayIndexOutOfBoundsException "arraycopy: last destination index 1506635235 out of bounds for long[224938403]")
 at com.semmle.inmemory.pipeline.PipelineInstance.wrapWithRaDump(PipelineInstance.java:168)
 at com.semmle.inmemory.pipeline.PipelineInstance.exceptionCaught(PipelineInstance.java:152)
 at com.semmle.inmemory.scheduler.execution.ThreadableWork.handleAndLog(ThreadableWork.java:593)
 at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:410)
 at com.semmle.inmemory.scheduler.IntensionalLayer$IntensionalWork.evaluate(IntensionalLayer.java:93)
 at com.semmle.inmemory.scheduler.SimpleLayerTask$SimpleLayerWork.doWork(SimpleLayerTask.java:69)
 at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:396)
 at com.semmle.inmemory.scheduler.execution.ExecutionScheduler.runnerMain(ExecutionScheduler.java:677)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ArrayIndexOutOfBoundsException: arraycopy: last destination index 1506635235 out of bounds for long[224938403]
 at java.base/java.lang.System.arraycopy(Native Method)
 at com.semmle.inmemory.eval.aggregate.TupleListList.addList(TupleListList.java:44)
 at com.semmle.inmemory.eval.aggregate.AggregateEvaluator.commitCurrentRun(AggregateEvaluator.java:463)
 at com.semmle.inmemory.eval.aggregate.AggregateEvaluator$GroupAndJoin.addTuple(AggregateEvaluator.java:512)
 at com.semmle.inmemory.eval.CancelCheckingSink.addTuple(CancelCheckingSink.java:18)
 at com.semmle.inmemory.relations.BaseIntArrayRelation.map(BaseIntArrayRelation.java:84)
 at com.semmle.inmemory.caching.PagedRelation.map(PagedRelation.java:158)
 at com.semmle.inmemory.relations.AbstractRelation.deduplicateMap(AbstractRelation.java:138)
 at com.semmle.inmemory.eval.aggregate.AggregateEvaluator.evaluate(AggregateEvaluator.java:256)
 at com.semmle.inmemory.pipeline.AggregateStep.generateTuples(AggregateStep.java:43)
 at com.semmle.inmemory.pipeline.SimpleHeadStep.lambda$forwardInitialize0ドル(SimpleHeadStep.java:32)
 at com.semmle.inmemory.pipeline.HeadEndDispatcher.headEndWork(HeadEndDispatcher.java:75)
 at com.semmle.inmemory.pipeline.PipelineState.doSomeWork(PipelineState.java:88)
 at com.semmle.inmemory.pipeline.PipelineInstance.doWork(PipelineInstance.java:117)
 at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:396)
 ... 7 more

Seems like many queries have been completed, but this one triggers an exception.

Any suggestion is appreciated!
You must be logged in to vote

We've implemented most of what's needed to avoid the totalorder predicate in future versions, and the rest of the work will happen as soon as the relevant people are back from summer vacation.

This work is now complete, and should be part of 2.18.1 once that releases (likely around July 26th).

Replies: 2 comments 6 replies

Comment options

Could you check if this is fixed with 2.17.5? There's a possibility the snag has already been fixed.
You must be logged in to vote
3 replies
Comment options

Hello @smowton !

Tried with 2.17.6 (since it's the latest version)

image

Now I get this error:

Starting evaluation of codeql/cpp-queries/Security/CWE/CWE-497/PotentiallyExposedSystemData.ql.
Starting evaluation of codeql/cpp-queries/Security/CWE/CWE-416/IteratorToExpiredContainer.ql.
Starting evaluation of codeql/cpp-queries/Security/CWE/CWE-732/UnsafeDaclSecurityDescriptor.ql.
Starting evaluation of codeql/cpp-queries/Security/CWE/CWE-497/ExposedSystemData.ql.
Starting evaluation of codeql/cpp-queries/Security/CWE/CWE-807/TaintedCondition.ql.
Oops! A fatal internal error occurred. Details:
com.semmle.util.exception.CatastrophicError: An error occurred while evaluating DataFlowImplCommon::Cached::edgeOrder/1#6d956c05/2@e09049d4
java.lang.IllegalArgumentException: Maximum array length exceeded: 2147483640
The RA to evaluate was:
 {2} r1 = AGGREGATE `_DataFlowPrivate::DataFlowCallable.totalorder/0#dispred#48dd4ff9__DataFlowImplCommon::Cached::TMkCal__#rank_range`, `_DataFlowPrivate::DataFlowCallable.totalorder/0#dispred#48dd4ff9__DataFlowImplCommon::Cached::TMkCal__#rank_term` ON In.5, In.6, In.7 WITH RANK<1 ASC2 ASC> OUTPUT , Agg.0, Agg.1
 return r1
(eventual cause: IllegalArgumentException "Maximum array length exceeded: 2147483640")
 at com.semmle.inmemory.pipeline.PipelineInstance.wrapWithRaDump(PipelineInstance.java:168)
 at com.semmle.inmemory.pipeline.PipelineInstance.exceptionCaught(PipelineInstance.java:152)
 at com.semmle.inmemory.scheduler.execution.ThreadableWork.handleAndLog(ThreadableWork.java:593)
 at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:410)
 at com.semmle.inmemory.scheduler.IntensionalLayer$IntensionalWork.evaluate(IntensionalLayer.java:93)
 at com.semmle.inmemory.scheduler.SimpleLayerTask$SimpleLayerWork.doWork(SimpleLayerTask.java:69)
 at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:396)
 at com.semmle.inmemory.scheduler.execution.ExecutionScheduler.runnerMain(ExecutionScheduler.java:677)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalArgumentException: Maximum array length exceeded: 2147483640
 at com.semmle.inmemory.eval.aggregate.TupleListList.checkSize(TupleListList.java:41)
 at com.semmle.inmemory.eval.aggregate.TupleListList.prepareForAdd(TupleListList.java:32)
 at com.semmle.inmemory.eval.aggregate.TupleListList.addList(TupleListList.java:59)
 at com.semmle.inmemory.eval.aggregate.AggregateEvaluator.commitCurrentRun(AggregateEvaluator.java:463)
 at com.semmle.inmemory.eval.aggregate.AggregateEvaluator$GroupAndJoin.addTuple(AggregateEvaluator.java:512)
 at com.semmle.inmemory.eval.CancelCheckingSink.addTuple(CancelCheckingSink.java:18)
 at com.semmle.inmemory.relations.BaseIntArrayRelation.map(BaseIntArrayRelation.java:84)
 at com.semmle.inmemory.caching.PagedRelation.map(PagedRelation.java:158)
 at com.semmle.inmemory.relations.AbstractRelation.deduplicateMap(AbstractRelation.java:138)
 at com.semmle.inmemory.eval.aggregate.AggregateEvaluator.evaluate(AggregateEvaluator.java:256)
 at com.semmle.inmemory.pipeline.AggregateStep.generateTuples(AggregateStep.java:43)
 at com.semmle.inmemory.pipeline.SimpleHeadStep.lambda$forwardInitialize0ドル(SimpleHeadStep.java:32)
 at com.semmle.inmemory.pipeline.HeadEndDispatcher.headEndWork(HeadEndDispatcher.java:75)
 at com.semmle.inmemory.pipeline.PipelineState.doSomeWork(PipelineState.java:88)
 at com.semmle.inmemory.pipeline.PipelineInstance.doWork(PipelineInstance.java:117)
 at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:396)
 ... 7 more

Any idea how should I proceed?
Comment options

I'm afraid at least for now you have simply exceeded our maximum problem size, specifically by making us try to compute a ranking larger than the maximum Java array size. For the time being I'm afraid the only solution would be to bound the scope of analysis more tightly, typically by considering more components of your software opaque external libraries (things that are only seen as binary objects or libraries during database creation), and less of them the source code that is subject to analysis (compiled from source during database creation).

As an example, codeql database create -l cpp -c "gcc test.c test2.c -o binary" might be reduced in scope like ... make test2.c into a library test2.a ...; codeql database create -l cpp -c "gcc -c test.c -ltest2 -o binary" -- so that test.c is analysed in full depth, while the functions in test2.c are considered opaque. The downside is that data-flow queries will only find results that do not originate, flow through or terminate inside test2.c, so this consideration should inform what subset of your source you consider important to build from source and therefore analyse in depth.
Comment options

@smowton thank you for the advice!
Comment options

I'm afraid you've run into scalability problems with the totalorder predicate that was introduced in 2.17.4. If you downgrade to 2.17.3 it should work. We've implemented most of what's needed to avoid the totalorder predicate in future versions, and the rest of the work will happen as soon as the relevant people are back from summer vacation.
You must be logged in to vote
3 replies
Comment options

@jbj thank you! will give 2.17.3 a try, and I wish the relevant people a nice summer vacation!
Comment options

We've implemented most of what's needed to avoid the totalorder predicate in future versions, and the rest of the work will happen as soon as the relevant people are back from summer vacation.

This work is now complete, and should be part of 2.18.1 once that releases (likely around July 26th).
Answer selected by mindcrunch4u
Comment options

@aschackmull thank you! will give 2.18.1 a try once it's out!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

AltStyle によって変換されたページ (->オリジナル) /