Install Python script requirements

Question

Sometimes I download a Python script (from a trusted source), and can't run it because of missing dependencies, which I have to install one by one (no requirements.txt is provided). I'm looking for a way to automate this process. It's similar to this question, but I'd like to infer the requirements automatically. I.e. given the script, install all the packages it's importing.

If myscript.py looks like:

import pandas
import tensorflow as tf
from keras.models import Sequential

Then after running the script (e.g. bash extract_requirements.sh myscript.py) pandas, tensorflow, and keras will be installed.

Of course, a script can call another script, which has its own dependencies, package names don't always match to import names etc. So it's not a perfect solution, but better than nothing.

What I'm using now is a command-line script:

#!/env bash
set -u
usage() {
 echo "0ドル" '[-d] <MY_SCRIPT>'
 cat <<-EOF
 options:
 -d Dry-run. Print package names without installing.
EOF
 exit 1;
}
(( $# == 0 )) && usage
dry=0
if [[ "1ドル" == '-d' ]]; then
 dry=1
 shift
fi
(( $# != 1 )) && usage
script="1ドル"
imports1=$(egrep '^\s*from\s+[a-zA-Z_.-]+\simport' "$script" | cut -f2 -d' ' | cut -f1 -d'.')
imports2=$(egrep '^\s*import\s+[a-zA-Z_-]+' "$script" | cut -f2 -d' ')
all=$(printf "$imports1\n$imports2" | sort -u)
if (( dry )); then
 echo "$all" | xargs -L 1 echo
else
 echo "$all" | xargs -L 1 sudo pip install
fi
set +u

I was wondering if I'm missing some import cases, and more importantly, whether there's a simpler, more python-ic, solution.

• Daniel: 'Pythonic' applies more to pieces of Python code than to the language. Are you suggesting you might rewrite this in Python, or are you looking for tools provided by the Python Software Foundation that do this more cleanly?
• dimid: @Coal_ Yes, I was hoping there's a standard way to do it in python, rather than parsing python code in bash.

Answer 1

!!! Use this at your own risk. Fiddling with user's packages is scary business !!!

You can monkey-patch the builtin import statement and upon encountering ModuleNotFoundError install whatever dependency is missing. Here is a small example for a script.py. You can of course extend this to feature multiple files or dynamically use the file given as input.

import builtins
import pip
old_import = __import__
def my_import(name, globals=None, locals=None, fromlist=(), level=0):
 if globals["__name__"] == "script":
 try:
 return_args = old_import(name, globals=globals, locals=locals, 
 fromlist=fromlist, level=level)
 except ModuleNotFoundError:
 builtins.__import__, temp = old_import, builtins.__import__
 pipcode = pip.main(['install', name])
 if pipcode != 0:
 builtins.__import__ = temp
 raise
 else:
 return_args = old_import(name, globals=globals, locals=locals, 
 fromlist=fromlist, level=level)
 builtins.__import__ = temp
 else:
 return_args = old_import(name, globals=globals, locals=locals, 
 fromlist=fromlist, level=level)
 return return_args
builtins.__import__ = my_import
import script

and the script.py is your custom script:

import os
from time import time
import argh # this should be installed
print(time())

Note: People sometimes do things like:

try:
 import optional_module
except ImportError:
 pass # optional all cool

Above code patches import itself. If you remove the if that limits install powers to script you allow recursive installation for everything that is imported. Ever. Even if it is an optional dependency for a 4th level dependency test case of script pip will try to install it.

• dimid: Thanks, a nice pythonic solution, but I rather not to hardcode the name of the script.
• FirefoxMetzger: @dimid This is just a small example to demonstrate the idea. You can extend this, e.g. globbing a folder for filenames which are okay to install or passing in the name of the script as input argument.