-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Can we disable multiprocessing in CodeQL for AWS Lambda compatibility? #17469
Answered
by
jketema
AmarBhosale
asked this question in
Q&A
-
Our internal security scanner orchestrator requires that each tool runs within its own AWS Lambda function. However, we're facing issues with running CodeQL, particularly for Python, because CodeQL's Python extractor relies on multiprocessing, which apparently isn't supported in AWS Lambda. Is there a way to disable multiprocessing in CodeQL to allow it to function properly in the AWS Lambda environment?
Additional information on the error we are getting:
Initializing database at /tmp/codeql-db.
Running build command: []
Running command in /tmp/security-ghas-eval/full-zip/security-ghas-eval/ed749002036ef5f3c199c8dac54d046a84589182: [/opt/codeql/python/tools/autobuild.sh]
[2024年09月13日 02:00:06] [build-stderr] /bin/sh: line 1: python2: command not found
[2024年09月13日 02:00:06] [build-stdout] No directories containing root identifiers were found. Returning working directory as root.
[2024年09月13日 02:00:06] [build-stdout] Will try to guess Python version, as it was not specified in `lgtm.yml`
[2024年09月13日 02:00:06] [build-stdout] Trying to guess Python version based on Trove classifiers in setup.py
[2024年09月13日 02:00:06] [build-stdout] Did not find setup.py (expected it to be at /tmp/security-ghas-eval/full-zip/security-ghas-eval/ed749002036ef5f3c199c8dac54d046a84589182/setup.py)
[2024年09月13日 02:00:06] [build-stdout] Trying to guess Python version based on travis file
[2024年09月13日 02:00:06] [build-stdout] Did not find any travis files (expected them at either ['/tmp/security-ghas-eval/full-zip/security-ghas-eval/ed749002036ef5f3c199c8dac54d046a84589182/.travis.yml', '/tmp/security-ghas-eval/full-zip/security-ghas-eval/ed749002036ef5f3c199c8dac54d046a84589182/travis.yml'])
[2024年09月13日 02:00:06] [build-stdout] Trying to guess Python version based on installed versions
[2024年09月13日 02:00:06] [build-stdout] Wanted to run Python 2, but it is not available. Using Python 3 instead
[2024年09月13日 02:00:06] [build-stdout] This script is running Python 3, but Python 2 is also available (as 'python3')
[2024年09月13日 02:00:06] [build-stdout] Could not guess Python version, will use default: Python 3
[2024年09月13日 02:00:06] [build-stdout] Calling python3 -S /opt/codeql/python/tools/python_tracer.py --verbosity 3 -z all -c /tmp/codeql-db/working/trap_cache -R /tmp/security-ghas-eval/full-zip/security-ghas-eval/ed749002036ef5f3c199c8dac54d046a84589182
[2024年09月13日 02:00:08] [build-stderr] Traceback (most recent call last):
[2024年09月13日 02:00:08] [build-stderr] File "/opt/codeql/python/tools/python_tracer.py", line 53, in <module>
[2024年09月13日 02:00:08] [build-stderr] semmle.populator.main(original_path)
[2024年09月13日 02:00:08] [build-stderr] File "/opt/codeql/python/tools/python3src.zip/semmle/populator.py", line 60, in main
[2024年09月13日 02:00:08] [build-stderr] File "/opt/codeql/python/tools/python3src.zip/semmle/logging.py", line 85, in __init__
[2024年09月13日 02:00:08] [build-stderr] File "/usr/lib64/python3.9/multiprocessing/context.py", line 103, in Queue
[2024年09月13日 02:00:08] [build-stderr] return Queue(maxsize, ctx=self.get_context())
[2024年09月13日 02:00:08] [build-stderr] File "/usr/lib64/python3.9/multiprocessing/queues.py", line 43, in __init__
[2024年09月13日 02:00:08] [build-stderr] self._rlock = ctx.Lock()
[2024年09月13日 02:00:08] [build-stderr] File "/usr/lib64/python3.9/multiprocessing/context.py", line 68, in Lock
[2024年09月13日 02:00:08] [build-stderr] return Lock(ctx=self.get_context())
[2024年09月13日 02:00:08] [build-stderr] File "/usr/lib64/python3.9/multiprocessing/synchronize.py", line 162, in __init__
[2024年09月13日 02:00:08] [build-stderr] SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
[2024年09月13日 02:00:08] [build-stderr] File "/usr/lib64/python3.9/multiprocessing/synchronize.py", line 57, in __init__
[2024年09月13日 02:00:08] [build-stderr] sl = self._semlock = _multiprocessing.SemLock(
[2024年09月13日 02:00:08] [build-stderr] FileNotFoundError: [Errno 2] No such file or directory
[2024年09月13日 02:00:08] [build-stderr] Traceback (most recent call last):
[2024年09月13日 02:00:08] [build-stderr] File "/opt/codeql/python/tools/index.py", line 28, in <module>
[2024年09月13日 02:00:08] [build-stderr] buildtools.index.main()
[2024年09月13日 02:00:08] [build-stderr] File "/opt/codeql/python/tools/python3src.zip/buildtools/index.py", line 409, in main
[2024年09月13日 02:00:08] [build-stderr] File "/usr/lib64/python3.9/subprocess.py", line 373, in check_call
[2024年09月13日 02:00:08] [build-stderr] raise CalledProcessError(retcode, cmd)
[2024年09月13日 02:00:08] [build-stderr] subprocess.CalledProcessError: Command '['python3', '-S', '/opt/codeql/python/tools/python_tracer.py', '--verbosity', '3', '-z', 'all', '-c', '/tmp/codeql-db/working/trap_cache', '-R', '/tmp/security-ghas-eval/full-zip/security-ghas-eval/ed749002036ef5f3c199c8dac54d046a84589182']' returned non-zero exit status 1.
[2024年09月13日 02:00:08] [ERROR] Spawned process exited abnormally (code 1; tried to run: [/opt/codeql/python/tools/autobuild.sh])
A fatal error occurred: Exit status 1 from command: [/opt/codeql/python/tools/autobuild.sh]
Beta Was this translation helpful? Give feedback.
All reactions
Answered by
jketema
Sep 16, 2024
Hi @AmarBhosale,
Unfortunately it is not possible to disable multiprocessing.
Replies: 1 comment
-
Hi @AmarBhosale,
Unfortunately it is not possible to disable multiprocessing.
Beta Was this translation helpful? Give feedback.
All reactions
0 replies
Answer selected by
AmarBhosale
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment