Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Can we disable multiprocessing in CodeQL for AWS Lambda compatibility? #17469

Answered by jketema
AmarBhosale asked this question in Q&A
Discussion options

Our internal security scanner orchestrator requires that each tool runs within its own AWS Lambda function. However, we're facing issues with running CodeQL, particularly for Python, because CodeQL's Python extractor relies on multiprocessing, which apparently isn't supported in AWS Lambda. Is there a way to disable multiprocessing in CodeQL to allow it to function properly in the AWS Lambda environment?

Additional information on the error we are getting:

Initializing database at /tmp/codeql-db.
Running build command: []
Running command in /tmp/security-ghas-eval/full-zip/security-ghas-eval/ed749002036ef5f3c199c8dac54d046a84589182: [/opt/codeql/python/tools/autobuild.sh]
[2024年09月13日 02:00:06] [build-stderr] /bin/sh: line 1: python2: command not found
[2024年09月13日 02:00:06] [build-stdout] No directories containing root identifiers were found. Returning working directory as root.
[2024年09月13日 02:00:06] [build-stdout] Will try to guess Python version, as it was not specified in `lgtm.yml`
[2024年09月13日 02:00:06] [build-stdout] Trying to guess Python version based on Trove classifiers in setup.py
[2024年09月13日 02:00:06] [build-stdout] Did not find setup.py (expected it to be at /tmp/security-ghas-eval/full-zip/security-ghas-eval/ed749002036ef5f3c199c8dac54d046a84589182/setup.py)
[2024年09月13日 02:00:06] [build-stdout] Trying to guess Python version based on travis file
[2024年09月13日 02:00:06] [build-stdout] Did not find any travis files (expected them at either ['/tmp/security-ghas-eval/full-zip/security-ghas-eval/ed749002036ef5f3c199c8dac54d046a84589182/.travis.yml', '/tmp/security-ghas-eval/full-zip/security-ghas-eval/ed749002036ef5f3c199c8dac54d046a84589182/travis.yml'])
[2024年09月13日 02:00:06] [build-stdout] Trying to guess Python version based on installed versions
[2024年09月13日 02:00:06] [build-stdout] Wanted to run Python 2, but it is not available. Using Python 3 instead
[2024年09月13日 02:00:06] [build-stdout] This script is running Python 3, but Python 2 is also available (as 'python3')
[2024年09月13日 02:00:06] [build-stdout] Could not guess Python version, will use default: Python 3
[2024年09月13日 02:00:06] [build-stdout] Calling python3 -S /opt/codeql/python/tools/python_tracer.py --verbosity 3 -z all -c /tmp/codeql-db/working/trap_cache -R /tmp/security-ghas-eval/full-zip/security-ghas-eval/ed749002036ef5f3c199c8dac54d046a84589182
[2024年09月13日 02:00:08] [build-stderr] Traceback (most recent call last):
[2024年09月13日 02:00:08] [build-stderr] File "/opt/codeql/python/tools/python_tracer.py", line 53, in <module>
[2024年09月13日 02:00:08] [build-stderr] semmle.populator.main(original_path)
[2024年09月13日 02:00:08] [build-stderr] File "/opt/codeql/python/tools/python3src.zip/semmle/populator.py", line 60, in main
[2024年09月13日 02:00:08] [build-stderr] File "/opt/codeql/python/tools/python3src.zip/semmle/logging.py", line 85, in __init__
[2024年09月13日 02:00:08] [build-stderr] File "/usr/lib64/python3.9/multiprocessing/context.py", line 103, in Queue
[2024年09月13日 02:00:08] [build-stderr] return Queue(maxsize, ctx=self.get_context())
[2024年09月13日 02:00:08] [build-stderr] File "/usr/lib64/python3.9/multiprocessing/queues.py", line 43, in __init__
[2024年09月13日 02:00:08] [build-stderr] self._rlock = ctx.Lock()
[2024年09月13日 02:00:08] [build-stderr] File "/usr/lib64/python3.9/multiprocessing/context.py", line 68, in Lock
[2024年09月13日 02:00:08] [build-stderr] return Lock(ctx=self.get_context())
[2024年09月13日 02:00:08] [build-stderr] File "/usr/lib64/python3.9/multiprocessing/synchronize.py", line 162, in __init__
[2024年09月13日 02:00:08] [build-stderr] SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
[2024年09月13日 02:00:08] [build-stderr] File "/usr/lib64/python3.9/multiprocessing/synchronize.py", line 57, in __init__
[2024年09月13日 02:00:08] [build-stderr] sl = self._semlock = _multiprocessing.SemLock(
[2024年09月13日 02:00:08] [build-stderr] FileNotFoundError: [Errno 2] No such file or directory
[2024年09月13日 02:00:08] [build-stderr] Traceback (most recent call last):
[2024年09月13日 02:00:08] [build-stderr] File "/opt/codeql/python/tools/index.py", line 28, in <module>
[2024年09月13日 02:00:08] [build-stderr] buildtools.index.main()
[2024年09月13日 02:00:08] [build-stderr] File "/opt/codeql/python/tools/python3src.zip/buildtools/index.py", line 409, in main
[2024年09月13日 02:00:08] [build-stderr] File "/usr/lib64/python3.9/subprocess.py", line 373, in check_call
[2024年09月13日 02:00:08] [build-stderr] raise CalledProcessError(retcode, cmd)
[2024年09月13日 02:00:08] [build-stderr] subprocess.CalledProcessError: Command '['python3', '-S', '/opt/codeql/python/tools/python_tracer.py', '--verbosity', '3', '-z', 'all', '-c', '/tmp/codeql-db/working/trap_cache', '-R', '/tmp/security-ghas-eval/full-zip/security-ghas-eval/ed749002036ef5f3c199c8dac54d046a84589182']' returned non-zero exit status 1.
[2024年09月13日 02:00:08] [ERROR] Spawned process exited abnormally (code 1; tried to run: [/opt/codeql/python/tools/autobuild.sh])
A fatal error occurred: Exit status 1 from command: [/opt/codeql/python/tools/autobuild.sh]
You must be logged in to vote

Hi @AmarBhosale,

Unfortunately it is not possible to disable multiprocessing.

Replies: 1 comment

Comment options

Hi @AmarBhosale,

Unfortunately it is not possible to disable multiprocessing.

You must be logged in to vote
0 replies
Answer selected by AmarBhosale
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet

AltStyle によって変換されたページ (->オリジナル) /