Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Commit c7f931f

Browse files
first commit
0 parents commit c7f931f

File tree

3 files changed

+200
-0
lines changed

3 files changed

+200
-0
lines changed

‎README.md

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Build AWS Lambda Layer zip file for Python Dependancies
2+
3+
Creates an AWS Lambda Layers **optimized** zip file using the [Lambda Layer directory structure](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-path), ensures compiled libraries are compatible with Lambda environment, and optimized to reduce file size.
4+
5+
This function was created to address these issues:
6+
7+
- Many methods of creating Lambda zip files for Python functions don't work for Lambda Layers because Lambda Layers require specific library paths within the zip
8+
- Some dependancies required compiled components, which requires that the zip is created in an environment that matches the Lambda runtime environment
9+
- Minimize the zip size by removing unnecessary items
10+
11+
**Note: This script requires Docker and uses a container to mimic the Lambda environment.**
12+
13+
## Features
14+
15+
- Builds zip file containing Python dependancies and places the libraries into the proper directory structure for lambda layers
16+
- Ensures compiled libraries are compatible with Lambda environment by using [docker container](https://hub.docker.com/r/lambci/lambda) that mimics the lambda runtime environment
17+
- Optimized the zip size by removing `.pyc` files and unnecessary libraries
18+
- allows specifying lambda supported python versions: 2.7, 3.6 and 3.7
19+
- Automatically searches for requirements.txt file in several locations:
20+
- same directory as script
21+
- parent directory or script (useful when used as submodule)
22+
- function sub-directory of the parent directory
23+
24+
## Installation
25+
26+
This function can be **cloned** for standalone use, into a parent repo or added as a **submodule**.
27+
28+
Clone for standalone use or within a repo:
29+
30+
``` bash
31+
# If installing into an exisiting repo, navigate to repo dir
32+
git clone --depth 1 https://github.com/robertpeteuil/build_py_lambda_layer _build_layer
33+
```
34+
35+
Alternatively, add as a submodule:
36+
37+
``` bash
38+
# PUBLIC USE HTTPS
39+
cd {repo root}
40+
# eventual public repo will use http
41+
git submodule add https://github.com/robertpeteuil/build_py_lambda_layer _build_layer
42+
# Update submodule
43+
git submodule update --init --recursive --remote
44+
```
45+
46+
## Use
47+
48+
- Run the builder with the command `./build_layer.sh`
49+
- Optionally specify Python Version
50+
- `-p PYTHON_VER` - specifies the Python version: 2.7, 3.6, 3.7 (default 3.6)
51+
- It uses the first requirements.txt file found in these locations (in order):
52+
- same directory as script
53+
- parent directory of script (useful when used as submodule)
54+
- function sub-directory of the parent directory (useful when used as submodule)
55+
56+
## Reference - remove submodule
57+
58+
If installed as submodule and want to remove
59+
60+
``` bash
61+
# Remove the submodule entry from .git/config
62+
git submodule deinit -f $submodulepath
63+
# Remove the submodule directory from the superproject's .git/modules directory
64+
rm -rf .git/modules/$submodulepath
65+
# Remove the entry in .gitmodules and remove the submodule directory located at path/to/submodule
66+
git rm -f $submodulepath
67+
# remove entry in submodules file
68+
git config -f .git/config --remove-section submodule.$submodulepath
69+
```

‎_make_zip.sh

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
#!/usr/bin/env bash
2+
3+
set -e
4+
5+
# AWS Lambda Layer Zip Builder for Python Libraries
6+
# This script is executed inside a docker container by the "build_layer.sh" script
7+
# It builds the zip file with files in lambda layers dir structure
8+
# /python/lib/pythonX.X/site-packages
9+
10+
scriptname=$(basename "0ドル")
11+
scriptbuildnum="1.0.0"
12+
scriptbuilddate="2019年03月30日"
13+
14+
### VARS
15+
CURRENT_DIR=$(reldir=$(dirname -- "0ドル"; echo x); reldir=${reldir%?x}; cd -- "$reldir" && pwd && echo x); CURRENT_DIR=${CURRENT_DIR%?x}
16+
17+
PYTHON="python${PYTHON_VER}"
18+
ZIP_FILE="base_${PYTHON}.zip"
19+
20+
echo "BUILDING ZIP: ${ZIP_FILE} for ${PYTHON}"
21+
22+
# Create build dir
23+
mkdir /tmp/build
24+
25+
# Create virtual environment and activate it
26+
virtualenv -p $PYTHON /tmp/build
27+
source /tmp/build/bin/activate
28+
29+
# Install requirements
30+
pip install -r /temp/build/requirements.txt --no-cache-dir
31+
32+
# Create staging area in dir structure req for lambda layers
33+
mkdir -p "/tmp/base/python/lib/${PYTHON}"
34+
35+
# Move dependancies to staging area
36+
mv "/tmp/build/lib/${PYTHON}/site-packages" "/tmp/base/python/lib/${PYTHON}"
37+
38+
# remove unused libraries
39+
cd "/tmp/base/python/lib/${PYTHON}/site-packages"
40+
rm -rf easy-install*
41+
rm -rf wheel*
42+
rm -rf setuptools*
43+
rm -rf virtualenv*
44+
rm -rf pip*
45+
46+
# Delete .pyc files from staging area
47+
cd "/tmp/base/python/lib/${PYTHON}"
48+
find . -name '*.pyc' -delete
49+
50+
# Add files from staging area to zip
51+
cd /tmp/base
52+
zip -r "${CURRENT_DIR}/${ZIP_FILE}" .
53+
54+
echo -e "\nBASE ZIP CREATION FINISHED"

‎build_layer.sh

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
#!/usr/bin/env bash
2+
3+
set -e
4+
5+
# AWS Lambda Layer Zip Builder for Python Libraries
6+
# requires: docker, _make.zip.sh, build_layer.sh (this script)
7+
# Launches docker container from lambci/lambda:build-pythonX.X image
8+
# where X.X is the python version (2.7, 3.6, 3.7) - defaults to 3.6
9+
# Executes build script "_make.zip.sh" within container to create zip
10+
# with libs specified in requirements.txt
11+
# Zip filename includes python version used in its creation
12+
13+
scriptname=$(basename "0ドル")
14+
scriptbuildnum="1.0.0"
15+
scriptbuilddate="2019年03月30日"
16+
17+
# used to set destination of zip
18+
SUBDIR_MODE=""
19+
20+
displayVer() {
21+
echo -e "${scriptname} ver ${scriptbuildnum} - ${scriptbuilddate}"
22+
}
23+
24+
usage() {
25+
[[ "1ドル" ]] && echo -e "AWS Lambda Layer Zip Builder for Python Libraries\n"
26+
echo -e "usage: ${scriptname} [-p PYTHON_VER] [-s] [-r REQUIREMENTS-DIR] [-h] [-v]"
27+
echo -e " -p PYTHON_VER\t: Python version to use: 2.7, 3.6, 3.7 (default 3.6)"
28+
echo -e " -h\t\t\t: help"
29+
echo -e " -v\t\t\t: display ${scriptname} version"
30+
}
31+
32+
while getopts ":p:hv" arg; do
33+
case "${arg}" in
34+
p) PYTHON_VER=${OPTARG};;
35+
h) usage; exit;;
36+
v) displayVer; exit;;
37+
\?) echo -e "Error - Invalid option: $OPTARG"; usage; exit;;
38+
:) echo "Error - $OPTARG requires an argument"; usage; exit 1;;
39+
esac
40+
done
41+
shift $((OPTIND-1))
42+
43+
# default Python to 3.6 if not set by CLI params
44+
PYTHON_VER="${PYTHON_VER:-3.6}"
45+
46+
CURRENT_DIR=$(reldir=$(dirname -- "0ドル"; echo x); reldir=${reldir%?x}; cd -- "$reldir" && pwd && echo x); CURRENT_DIR=${CURRENT_DIR%?x}
47+
BASE_DIR=$(basename $CURRENT_DIR)
48+
PARENT_DIR=${CURRENT_DIR%"${BASE_DIR}"}
49+
50+
# find location of requirements.txt
51+
if [[ -f "${CURRENT_DIR}/requirements.txt" ]]; then
52+
REQ_PATH="${CURRENT_DIR}/requirements.txt"
53+
echo "reqs in base dir"
54+
elif [[ -f "${PARENT_DIR}/requirements.txt" ]]; then
55+
REQ_PATH="${PARENT_DIR}/requirements.txt"
56+
SUBDIR_MODE="True"
57+
echo "reqs in parent"
58+
elif [[ -f "${PARENT_DIR}/function/requirements.txt" ]]; then
59+
REQ_PATH="${PARENT_DIR}/function/requirements.txt"
60+
SUBDIR_MODE="True"
61+
echo "reqs in parent/function"
62+
else
63+
echo "Unable to find requirements.txt"
64+
exit 1
65+
fi
66+
67+
docker run --rm -e PYTHON_VER="$PYTHON_VER" -v "$CURRENT_DIR":/var/task -v "$REQ_PATH":/temp/build/requirements.txt "lambci/lambda:build-python${PYTHON_VER}" bash /var/task/_make_zip.sh
68+
69+
# Move ZIP to parent dir if SUBDIR_MODE set
70+
if [[ "$SUBDIR_MODE" ]]; then
71+
ZIP_FILE="base_python${PYTHON_VER}.zip"
72+
# Make backup of zip if exists in parent dir
73+
if [[ -f "${PARENT_DIR}/${ZIP_FILE}" ]]; then
74+
mv "${PARENT_DIR}/${ZIP_FILE}" "${PARENT_DIR}/${ZIP_FILE}.bak"
75+
fi
76+
mv "${CURRENT_DIR}/${ZIP_FILE}" "${PARENT_DIR}"
77+
fi

0 commit comments

Comments
(0)

AltStyle によって変換されたページ (->オリジナル) /