importing python package from AWS Lambda /tmp directory

Question

I am trying to upload a zip file, which contains mostly python packages, of size 300MB to AWS Lambda. I clearly understand that this exceeds the limit for the zip that can be uploaded to Lambda if we uploaded directly using AWS SDK. Therefore, this will not work.

In order to overcome this, i decided to download the packages in the /tmp directory and import them to the main file (reference here). I compressed the required packages as pkgs.zip and upload it to AWS S3. Then I download them using requests extract them to /tmp/.

def get_pkgs(url):
 import requests
 import io
 import zipfile
 print("Getting Packages...")
 re = requests.get(url)
 z = zipfile.ZipFile(io.BytesIO(re.content))
 print("Extracting Packages...")
 z.extractall("/tmp/")
 print("Packages are downloaded and extracted.")
def attempt_import():
 print("="*50)
 print("ATTEMPT TO IMPORT DEPENDENCIES...")
 print("="*50)
 import numpy
 import scipy
 import six
 print("IMPORTING DONE.")
def main():
 URL = "https://s3-ap-southeast-1.amazonaws.com/BUCKET_NAME/pkgs.zip"
 get_pkgs(URL)
 attempt_import()
def lambda_handler(event, context):
 main()
 return "Hello Lambda"

However, when i test the lambda function, it returns an error saying that numpy cannot be found

Import Error: No module named numpy

My question is, How do I import the required packages from the /tmp/ diretory?

Thanks in advance.

• John Rotenstein: Sounds similar to: How to import a module given the full path?

Answer 1

Before you can import any package that you have downloaded to the /tmp folder (for example to the /tmp/requirements folder) you have to tell the system to look for the dependencies over there. In the beginning of the code, just at these lines:

import sys
sys.path.insert(0, '/tmp/requirements/') # Or any path you desire