Using numpy in AWS Lambda

Question

I'm looking for a work around to use numpy in AWS lambda. I am not using EC2 just lambda for this so if anyone has a suggestion that'd be appreciated. Currently getting the error:

cannot import name 'multiarray'

Using grunt lambda to create the zip file and upload the function code. All the modules that I use are installed into a folder called python_modules inside the root of the lambda function which includes numpy using pip install and a requirements.txt file.

• Nikolay Grishchenko: You should probably have your modules installed or symlinked directly to the root of your Lambda function package. And make sure that all the dependencies also go there.
• Sebastian Łącki: I know what you're getting at but all the modules work fine as the env variable points to them.
• JavaCake: Did you solve this issue by any chance? I have the exact issue on Python 3.6?
• Sebastian Łącki: I haven't gotten back to it, but i did find this link althought i haven't had a chance to look back into it! Hopefully you get something out of this and if you do let me know! github.com/vitolimandibhrata/aws-lambda-numpy . From my understanding it takes complete versions of numpy packages and you can drop them into your project.
• DaveRGP: As of 2018 it's best and easiest to use layers: aws.amazon.com/blogs/aws/… AWS even have premade one for numpy

Answer 1

An easy way to make your lambda function support the numpy library for python 3.7:

1. Go to your lambda function page
2. Find the Layers section at the bottom of the page.
3. Click on Add a layer.
4. Choose AWS layers as layer source.
5. Select AWSLambda-Python37-Scipy1x as AWS layers.
6. Select 37 for version.
7. And finally click on Add.

Now your lambda function is ready to support numpy.

• DaveP: Numpy can be added with the layer "AWSDataWrangler-Python39" (I don't see the layer mentioned above. )
• user_5: @DaveP Maybe your Python version was different from mine.

Answer 2

Updated to include the solution here, rather than a link:

After much effort, I found that I had to create my deployment package from within a python3.6 virtualenv, rather than directly from the host machine. I did the following within a Ubuntu 16.04 docker image. This assumes that you have python3.6, virtualenv and awscli already installed/configured, and that your lambda function code is in the ~/lambda_code directory:

1) cd ~ (We'll build the virtualenv in the home directory)

2) virtualenv venv --python=python3.6 (Create the virtual environment)

3) source venv/bin/activate (Activate the virtual environment)

4) pip install numpy

5) cp -r ~/venv/lib/python3.6/site-packages/* ~/lambda_code (Copy all installed packages into root level of lambda_code directory. This will include a few unnecessary files, but you can remove those yourself if needed)

6) cd ~/lambda_code

7) zip -r9 ~/package.zip . (Zip up the lambda package)

8) aws lambda update-function-code --function-name my_lambda_function --zip-file fileb://~/package.zip (Upload to AWS)

Your lambda function should now be able to import numpy with no problems.

If you want a more out-of-the-box solution, you could consider using serverless to deploy your lambda function. Before I found the above solution, I followed the guide here and was able to run numpy successfully in a python3.6 lambda function.

• GWed: you can save yourself some time by using Docker and/or Serverless framework stackoverflow.com/questions/43859497/…

Answer 3

As of 2018 it's best to just use the inbuilt layers functionality.

AWS have actually released a pre-made one with numpy in it: https://aws.amazon.com/blogs/aws/new-for-aws-lambda-use-any-programming-language-and-share-common-components/

Answer 4

Add numpy layer in this way:

• Go on your lambda function

• select add a new layer

• add it using this arn: arn:aws:lambda:eu-central-1:770693421928:layer:Klayers-p39-numpy:7

(change your zone if you are not in eu-central-1)

Let me know if it will work

Answer 5

I was unable to find a good solution using serverless plugins, but I did find a good way with layers. See Serverless - Numpy - Unable to find good bind path format

Answer 6

1.) Do a Pip install of numpy to a folder on your local machine.

2.) once complete, zip the entire folder and create a zip file.

3.) Go to AWS lambda console, create a layer and upload zip file created in step 2 there and save the layer.

4.) After you create your lambda function, click add layer and add the layer you created. That's it, import numpy will start working.