AWS Lambda and python numpy module

Question

I am trying to import a python deployment package in aws lambda. The python code uses numpy. I followed the deployment package instructions for virtual env but it still gave Missing required dependencies ['numpy']. I followed the instruction given on stack overflow (skipped step 4 for shared libraries, could not find any shared libraries) but no luck. Any suggestions to make it work?

• alanwill: Can you provide the code for your lambda function?
• joarleymoraes: whats your project structure look like ? it could be just how you are zipping your package. also have you pre-compiled 'numpy' ?
• user1484793: The project folder contains *.py files, all the files of site-packages and files and folders from dist-package. I am not sure what pre-compilation of numpy means. I guess I am not zipping it correctly but not sure what I am doing wrong. Should I only include dist-packages and not site-packages? Should I include all the folders with files within or copy all the files from different folders at the same place?
• Yuchen: @user1484793 hey, have you solved your problem yet?

Answer 1

The easiest way is to use AWS Cloud9, there is no need to start EC2 instances and prepare deployment packages.

Step 1: start up Cloud9 IDE

• Go to the AWS console and select the Cloud9 service.
• Create environment
• enter Name
• Environment settings (consider using t2.small instance type, with the default I had sometimes problems to restart the environment)
• Review
• click Create environment

Step 2: create Lambda function

• Create Lambda Function (bottom of screen)
• enter Function name and Application name
• Select runtime (Python 3.6) and blueprint (empty-python)
• Function trigger (none)
• Create serverless application (keep defaults)
• Finish
• wait couple of seconds for the IDE to open

Step 3: install Numpy

• at the bottom of the screen there is a command prompt

• go to the Application folder, for me this is

  cd App
  

• install the package (we have to use pip from the virtual environment, because the default pip points to /usr/bin/pip and this is python 2.7)

  venv/bin/pip install numpy -t .
  

Step4: test installation

• you can test the installation by editing the lambda_function.py file:

  import numpy as np
  def lambda_handler(event, context):
   return np.sin(1.0)
  

• save the changes and click the green Run button on the top of the screen

• a new tab should appear, now click the green Run button inside the tab

• after the Lambda executes I get:

  Response
  0.8414709848078965
  

Step 5: deploy Lambda function

• on the right hand side of the screen select your Lambda function
• click the upwards pointing arrow to deploy
• go to the AWS Lambda service tab

• the Lambda function should be visible, the name has the format

  cloud9-ApplicationName-FunctionName-RandomString
  

Answer 2

Using Numpy is a real pain.

Numpy needs to be properly compiled on the same OS as it runs. This means that you need to install/compile Numpy on an AMI image in order for it erun properly in Lambda.

The easiest way to do this is to start a small EC2 instance and install it there. Then copy the compiled files (from /usr/lib/python/site-packages/numpy). These are the files you need to include in your Lambda package.

I believe you can also use the serverless tool to achieve this.

• 1737973: This is what I do, and it works. But don't forget to extract the linear algebra shared objects out of the EC2 instance and include them in a lib/ dir in your Lambda package ZIP contents. I.e. AWS Lambda de facto includes $LAMBDA_ROOT/lib/ in $LD_LIBRARY_PATH.

Answer 3

NumPy must be compiled on the platform that it will be run on. Easiest way to do this is to use Docker. Docker has a lambda container. Compile NumPy locally in Docker with the lambda container, then push to AWS lambda.

The serverless framework handles all this for you if you want an easy solution. See https://stackoverflow.com/a/50027031/1085343

Answer 4

I was having this same issue and pip install wasn't working for me. Eventually, I found some obscure posts about this issue here and here. They suggested going to pypi, downloading the .whl file (select py version and manylinux1_x86_64), uploading, and unzipping. This is what ultimately worked for me. If nothing else is working for you, I would suggest trying this.

Answer 5

For Python 3.6, you can use a Docker image as someone already suggested. Full instructions for that here: https://medium.com/i-like-big-data-and-i-cannot-lie/how-to-create-an-aws-lambda-python-3-6-deployment-package-using-docker-d0e847207dd6

The key piece is:

docker run -it dacut/amazon-linux-python-3.6