Safely have two versions of Python on Cloudera virtual machine without Python installation

Question

Presently we have a big-data cluster built using Cloudera-Virtual machines. By default the Python version on the VM is 2.7.

For one of my programs I need Python 3.6. My team is very skeptical about 2 installations and afraid of breaking existing cluster/VM. I was planning to follow this article and install 2 versions https://www.digitalocean.com/community/tutorials/how-to-set-up-python-2-7-6-and-3-3-3-on-centos-6-4

Is there a way "I can package Python 3.6" version in my project, and set the Python home path to my project folder, so that there is no installation that needs to be done on the existing Virtual machine?

Since we have to download python and build source for the Unix version, I want to skip this part on VM, and instead ship the folder which has Python 3.6

Answer 1

It seems that miniconda is what you need. using it you can manage multiple python environments with different versions of python.

to install miniconda3 just run:

# this will download & install miniconda3 on your home dir
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod +x Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh -b -p ~/miniconda3

then, create new python3.6 env:

conda create -y -n myproject 'python>3.6'

now, enter the new python3.6 env

source activate myproject
python3

miniconda can also install python packages, including pip packages and compiled packages. you can also copy envs from one machine to another. I encourage you to take a deeper look into it.

• M80: The point I'm keen is , can I package Pyhton3.6 in my project and set the path for Pyhton in my project-dir during the program launch. So that I don't have to do any installation on my server.

Answer 2

ShmulikA's suggestion is pretty good.

Here I'd like to add another one - I use Python 2.7.x, but for few prototypes, I had to go with Python 3.x. For this I used the pyenv utility.

Once installed, all you have to do is:

pyenv install 3.x.x

Can list all the available Python variants:

pyenv versions

To use the specific version, while at the project root, execute the following:

pyenv local 3.x.x

It'll create a file .python-version at the project root, having the version as it's content:

[nahmed@localhost ~]$ cat some-project/.python-version 
3.5.2

Example:

[nahmed@localhost ~]$ pyenv versions
* system (set by /home/nahmed/.pyenv/version)
 3.5.2
 3.5.2/envs/venv_scrapy
 venv_scrapy
[nahmed@localhost ~]$ pyenv local 3.5.2
[nahmed@localhost ~]$ pyenv versions
 system
* 3.5.2 (set by /home/nahmed/.python-version)
 3.5.2/envs/venv_scrapy
 venv_scrapy

---

I found it very simple to use. Here's a post regarding the installation and basic usage (blog post by me).

---

For the part:

Since we have to download python and build source for the Unix version, I want to skip this part on VM, and instead ship the folder which has Python 3.6

You might look into ways to embed Python interpreter with your Python application:

And for both Windows and Linux, there's bbfreeze or also pyinstaller

from - SOAnswer.