Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Commit fc2f92c

Browse files
committed
Initial version 0.1.0
1 parent 36b3a2b commit fc2f92c

File tree

5 files changed

+291
-1
lines changed

5 files changed

+291
-1
lines changed

‎LICENSE‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
MIT License
22

3-
Copyright (c) 2020 Tobi
3+
Copyright (c) 2020 TobiLG <tobilg@gmail.com>
44

55
Permission is hereby granted, free of charge, to any person obtaining a copy
66
of this software and associated documentation files (the "Software"), to deal

‎README.md‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# python-lambda-layer-builder
2+
3+
Creates an AWS Lambda Layers structure that is **optimized** for: [Lambda Layer directory structure](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-path), compiled library compatibility, and minimal file size.
4+
5+
This repo was created to address these issues:
6+
7+
- Many methods of creating Lambda zip files for Python functions don't work for Lambda Layers
8+
- This is due to the fact Lambda Layers require specific library paths within the zip, where regular Lambda zips don't
9+
- Compiled dependencies must be created in an environment that matches the Lambda runtime
10+
- Reduce size of the layer by removing unnecessary libraries and files
11+
12+
**Note: This script requires Docker and uses a container to mimic the Lambda environment.**
13+
14+
## Features
15+
16+
- Builds either a zip file or a raw directory strucutre (e.g. if you want to use frameworks like Serverless for packaging) containing Python dependencies and places the libraries into the proper directory structure for lambda layers
17+
- Ensures compiled libraries are compatible with Lambda environment by using the [lambci/lambda](https://hub.docker.com/r/lambci/lambda) Docker container that mimics the lambda runtime environment
18+
- Optimized the zip size by removing `.pyc` files and unnecessary libraries
19+
- allows specifying lambda supported python versions: 2.7, 3.6 and 3.7
20+
- Automatically searches for requirements.txt file in several locations:
21+
- same directory as script
22+
- parent directory or script (useful when used as submodule)
23+
- function sub-directory of the parent directory
24+
25+
## Installation
26+
27+
This function can be **cloned** for standalone use, into a parent repo or added as a **submodule**.
28+
29+
Clone for standalone use or within a repo:
30+
31+
``` bash
32+
# If installing into an exisiting repo, navigate to repo dir
33+
git clone --depth 1 https://github.com/tobilg/python-lambda-layer-builder _build_layer
34+
```
35+
36+
Alternatively, add as a submodule:
37+
38+
``` bash
39+
cd {repo root}
40+
git submodule add https://github.com/tobilg/python-lambda-layer-builder _build_layer
41+
# Update submodule
42+
git submodule update --init --recursive --remote
43+
```
44+
45+
## Usage
46+
47+
```text
48+
$ ./build.sh -h
49+
AWS Lambda Layer Builder for Python Libraries
50+
51+
Usage: build.sh [-p PYTHON_VER] [-n NAME] [-r] [-h] [-v]
52+
-p PYTHON_VER : Python version to use: 2.7, 3.6, 3.7 (default 3.7)
53+
-n NAME : Name of the layer
54+
-r : Raw mode, don't zip layer contents
55+
-h : Help
56+
-v : Display build.sh version
57+
```
58+
59+
- Run the builder with the command `./build.sh`
60+
- or `_build_layer/build.sh` if installed in sub-dir
61+
- It uses the first requirements.txt file found in these locations (in order):
62+
- Same directory as script
63+
- Parent directory of script (useful when used as submodule)
64+
- Function sub-directory of the parent directory (useful when used as submodule)
65+
- Optionally specify Python Version
66+
- `-p PYTHON_VER` - specifies the Python version: 2.7, 3.6, 3.7 (default 3.6)
67+
68+
### Custom cleaning logic
69+
70+
You can edit the `_clean.sh` file if you want to add custom cleaning logic for the build of the Lambda layer. The above part of the file must stay intact:
71+
72+
```bash
73+
#!/usr/bin/env bash
74+
# Change to working directory
75+
cd 1ドル
76+
# ----- DON'T CHANGE THE ABOVE -----
77+
78+
# Cleaning statements
79+
# ----- CHANGE HERE -----
80+
rm test.xt
81+
```
82+
83+
The `_make.sh` script will then execute the commands after the Python packages have been installed.
84+
85+
## Deinstallation
86+
87+
If installed as submodule and need to remove
88+
89+
```bash
90+
# Remove the submodule entry from .git/config
91+
git submodule deinit -f $submodulepath
92+
# Remove the submodule directory from the superproject's .git/modules directory
93+
rm -rf .git/modules/$submodulepath
94+
# Remove the entry in .gitmodules and remove the submodule directory located at path/to/submodule
95+
git rm -f $submodulepath
96+
# remove entry in submodules file
97+
git config -f .git/config --remove-section submodule.$submodulepath
98+
```

‎_clean.sh‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#!/usr/bin/env bash
2+
# Change to working directory
3+
cd 1ドル
4+
# ----- DON'T CHANGE THE ABOVE -----
5+
6+
# Cleaning statements
7+
# ----- CHANGE HERE -----

‎_make.sh‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
#!/usr/bin/env bash
2+
3+
set -e
4+
5+
# Credits for initial version: https://github.com/robertpeteuil/build-lambda-layer-python
6+
7+
# AWS Lambda Layer Zip Builder for Python Libraries
8+
# This script is executed inside a docker container by the "build_layer.sh" script
9+
# It builds the zip file with files in lambda layers dir structure
10+
# /python/lib/pythonX.X/site-packages
11+
12+
scriptname=$(basename "0ドル")
13+
scriptbuildnum="1.0.0"
14+
scriptbuilddate="2020年03月29日"
15+
16+
### Variables
17+
CURRENT_DIR=$(reldir=$(dirname -- "0ドル"; echo x); reldir=${reldir%?x}; cd -- "$reldir" && pwd && echo x); CURRENT_DIR=${CURRENT_DIR%?x}
18+
PYTHON="python${PYTHON_VER}"
19+
ZIP_FILE="${NAME}_${PYTHON}.zip"
20+
21+
echo "Building layer: ${NAME} for ${PYTHON}"
22+
23+
# Delete build dir
24+
rm -rf /tmp/build
25+
26+
# Create build dir
27+
mkdir -p /tmp/build
28+
29+
# Create virtual environment and activate it
30+
virtualenv -p $PYTHON /tmp/build
31+
source /tmp/build/bin/activate
32+
33+
# Install requirements
34+
pip install -r /temp/build/requirements.txt --no-cache-dir
35+
36+
# Create staging area in dir structure req for lambda layers
37+
mkdir -p "/tmp/base/python/lib/${PYTHON}"
38+
39+
# Move dependancies to staging area
40+
mv "/tmp/build/lib/${PYTHON}/site-packages" "/tmp/base/python/lib/${PYTHON}"
41+
42+
# Remove unused stuff
43+
cd "/tmp/base/python/lib/${PYTHON}/site-packages"
44+
echo "Original layer size: $(du -sh . | cut -f1)"
45+
rm -rf easy-install*
46+
rm -rf wheel*
47+
rm -rf setuptools*
48+
rm -rf virtualenv*
49+
rm -rf pip*
50+
find . -type d -name "tests" -exec rm -rf {} +
51+
find . -type d -name "test" -exec rm -rf {} +
52+
find . -type d -name "__pycache__" -exec rm -rf {} +
53+
find -name "*.so" -not -path "*/PIL/*" | xargs strip
54+
find -name "*.so.*" -not -path "*/PIL/*" | xargs strip
55+
find . -name '*.pyc' -delete
56+
if [[ -f "/temp/build/_clean.sh" ]]; then
57+
echo "Running custom cleaning script"
58+
source /temp/build/_clean.sh $PWD
59+
fi
60+
echo "Final layer size: $(du -sh . | cut -f1)"
61+
62+
# Delete .pyc files from staging area
63+
cd "/tmp/base/python/lib/${PYTHON}"
64+
find . -name '*.pyc' -delete
65+
66+
# Produce output
67+
if [[ "$RAW_MODE" = true ]]; then
68+
# Copy raw files to layer directory
69+
echo "${CURRENT_DIR}/layer"
70+
mkdir -p "${CURRENT_DIR}/layer"
71+
cp -R /tmp/base/. "${CURRENT_DIR}/layer"
72+
echo "Raw layer contents have been copied to the 'layer' subdirectory"
73+
else
74+
# Add files from staging area to zip
75+
cd /tmp/base
76+
zip -q -r "${CURRENT_DIR}/${ZIP_FILE}" .
77+
echo "Zipped layer size: $(ls -s --block-size=1048576 ${CURRENT_DIR}/${ZIP_FILE} | cut -d' ' -f1)M"
78+
fi
79+
80+
echo -e "\n${NAME} layer creation finished"

‎build.sh‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
#!/usr/bin/env bash
2+
3+
set -e
4+
5+
# Credits for initial version: https://github.com/robertpeteuil/build-lambda-layer-python
6+
7+
# AWS Lambda Layer Zip Builder for Python Libraries
8+
# requires: docker, _make.zip.sh, build_layer.sh (this script)
9+
# Launches docker container from lambci/lambda:build-pythonX.X image
10+
# where X.X is the python version (2.7, 3.6, 3.7) - defaults to 3.7
11+
# Executes build script "_make.zip.sh" within container to create zip
12+
# with libs specified in requirements.txt
13+
# Zip filename includes python version used in its creation
14+
15+
scriptname=$(basename "0ドル")
16+
scriptbuildnum="1.0.0"
17+
scriptbuilddate="2020年03月29日"
18+
19+
# Used to set destination of zip
20+
SUBDIR_MODE=""
21+
22+
# Display version
23+
displayVer() {
24+
echo -e "${scriptname} v${scriptbuildnum} (${scriptbuilddate})"
25+
}
26+
27+
# Display usage
28+
usage() {
29+
echo -e "AWS Lambda Layer Builder for Python Libraries\n"
30+
echo -e "Usage: ${scriptname} [-p PYTHON_VER] [-n NAME] [-r] [-h] [-v]"
31+
echo -e " -p PYTHON_VER\t: Python version to use: 2.7, 3.6, 3.7 (default 3.7)"
32+
echo -e " -n NAME\t: Name of the layer"
33+
echo -e " -r\t\t: Raw mode, don't zip layer contents"
34+
echo -e " -h\t\t: Help"
35+
echo -e " -v\t\t: Display ${scriptname} version"
36+
}
37+
38+
# Handle configuration
39+
while getopts ":p:n:rhv" arg; do
40+
case "${arg}" in
41+
p) PYTHON_VER=${OPTARG};;
42+
n) NAME=${OPTARG};;
43+
r) RAW_MODE=true;;
44+
h) usage; exit;;
45+
v) displayVer; exit;;
46+
\?) echo -e "Error - Invalid option: $OPTARG"; usage; exit;;
47+
:) echo "Error - $OPTARG requires an argument"; usage; exit 1;;
48+
esac
49+
done
50+
shift $((OPTIND-1))
51+
52+
# Default Python to 3.7 if not set by CLI params
53+
PYTHON_VER="${PYTHON_VER:-3.7}"
54+
NAME="${NAME:-base}"
55+
CURRENT_DIR=$(reldir=$(dirname -- "0ドル"; echo x); reldir=${reldir%?x}; cd -- "$reldir" && pwd && echo x); CURRENT_DIR=${CURRENT_DIR%?x}
56+
BASE_DIR=$(basename $CURRENT_DIR)
57+
PARENT_DIR=${CURRENT_DIR%"${BASE_DIR}"}
58+
RAW_MODE="${RAW_MODE:-false}"
59+
60+
# Find location of requirements.txt
61+
if [[ -f "${CURRENT_DIR}/requirements.txt" ]]; then
62+
REQ_PATH="${CURRENT_DIR}/requirements.txt"
63+
echo "Using requirements.txt from script dir"
64+
elif [[ -f "${PARENT_DIR}/requirements.txt" ]]; then
65+
REQ_PATH="${PARENT_DIR}/requirements.txt"
66+
SUBDIR_MODE="True"
67+
echo "Using requirements.txt from ../"
68+
elif [[ -f "${PARENT_DIR}/function/requirements.txt" ]]; then
69+
REQ_PATH="${PARENT_DIR}/function/requirements.txt"
70+
SUBDIR_MODE="True"
71+
echo "Using requirements.txt from ../function"
72+
else
73+
echo "Unable to find requirements.txt"
74+
exit 1
75+
fi
76+
77+
# Find location of _clean.sh
78+
if [[ -f "${CURRENT_DIR}/_clean.sh" ]]; then
79+
CLEAN_PATH="${CURRENT_DIR}/_clean.sh"
80+
echo "Using clean.sh from script dir"
81+
elif [[ -f "${PARENT_DIR}/_clean.sh" ]]; then
82+
CLEAN_PATH="${PARENT_DIR}/_clean.sh"
83+
echo "Using clean.sh from ../"
84+
else
85+
echo "Using default cleaning step"
86+
fi
87+
88+
if [[ "$RAW_MODE" = true ]]; then
89+
echo "Using RAW mode"
90+
else
91+
echo "Using ZIP mode"
92+
fi
93+
94+
# Run build
95+
docker run --rm -e PYTHON_VER="$PYTHON_VER" -e NAME="$NAME" -e RAW_MODE="$RAW_MODE" -v "$CURRENT_DIR":/var/task -v "$REQ_PATH":/temp/build/requirements.txt -v "$CLEAN_PATH":/temp/build/_clean.sh "lambci/lambda:build-python${PYTHON_VER}" bash /var/task/_make.sh
96+
97+
# Move ZIP to parent dir if SUBDIR_MODE set
98+
if [[ "$SUBDIR_MODE" ]]; then
99+
ZIP_FILE="${NAME}_python${PYTHON_VER}.zip"
100+
# Make backup of zip if exists in parent dir
101+
if [[ -f "${PARENT_DIR}/${ZIP_FILE}" ]]; then
102+
mv "${PARENT_DIR}/${ZIP_FILE}" "${PARENT_DIR}/${ZIP_FILE}.bak"
103+
fi
104+
mv "${CURRENT_DIR}/${ZIP_FILE}" "${PARENT_DIR}"
105+
fi

0 commit comments

Comments
(0)

AltStyle によって変換されたページ (->オリジナル) /