Hi,
Working on integrating IP with Python in order to call a random forest model to do some classifications of images.
I start out in IP9 load image and calculate the gray level co-occurance matrix values, imageGLCM, and want to pass them to a python based random forest model.
I am on a Mac.
I have a python script file, RF_model.py, with the following
#!/usr/bin/env python
# coding: utf-8
# In[1]:
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
import joblib
# In[6]:
rf_classifier=joblib.load('/Users/andy.hegedus/Desktop/Image Ground Truth/IP2Py2IP/rf_model.joblib')
# In[7]:
val_data=pd.read_csv('/Users/andy.hegedus/Desktop/Image Ground Truth/IP2Py2IP/haralick.csv')
Xval=val_data.drop(['cluster','x','y'], axis=1)
yval_pred = rf_classifier.predict(Xval)
val_data['cluster']=yval_pred
val_data.to_csv('/Users/andy.hegedus/Desktop/Image Ground Truth/IP2Py2IP/cluster.csv',index=False)
And then I have a shell script, run_python.sh:
#!/bin/zsh
python3 '/Users/andy.hegedus/Desktop/Image Ground Truth/IP2Py2IP/RF_model.py'
If I prepopulate the necessary files and run the shell script it works as expected and creates an output file that I am going to load back into IP9.
Note I have done a chmod 755 on the shell script to be executable.
So all is working.
Within IP9.
I create a string variable with
string Run_Python = "do shell script \"'/Users/andy.hegedus/Desktop/Image Ground Truth/IP2Py2IP/run_python.sh'\""
And then do
executeScriptText run_python
I receive an error and the S_Value
Traceback (most recent call last):
File "/Users/andy.hegedus/Desktop/Image Ground Truth/IP2Py2IP/RF_model.py", line 7, in <module>
import pandas as pd
ModuleNotFoundError: No module named 'pandas'
The shell script and its associated python script work as intended from terminal, but from within IP9 it does not work, but it finds the shell script file and the associated python script.
What am I missing?
Andy
On macOS, do you have to preface the call to pandas to define it as being found within the bash shell rather than within zsh? Or vice-versa? And what might happen if you comment out the pandas module and code lines to only include numpy? Same problem, but now with numpy???
This section of the ExecuteScriptText command help may be relevant:
Prior to Igor Pro 8.00, the current directory was always / and the PATH environment variable was the system default, not the user default (usually PATH=/usr/bin:/bin:/usr/sbin:/sbin.). Now neither of these is guaranteed to be the case. You should not make any assumptions about the current directory or the value of any environment variable. Use full paths to files and programs whenever you need to provide a path.
When you call ExecuteScriptText it is possible that a different python is being used than when you call the command on Terminal, due to the PATH possibly being different.
You may need to alter your run_python.sh script to set environment variables that are appropriate for your conditions.
Also, in the other thread, you mentioned that eventually you need to get this running on Windows for your client. You are dealing with an area that is very platform specific, so you will likely need to tweak almost all of this to get it working on Windows. You might wish to drop trying to do it on macOS and just skip to Windows.
In reply to This section of the... by aclight
hegedusI do understand it is very platform specific, but this step is a small portion of the overall work and I am much more comfortable doing the bulk of the rest of the work on my Mac. I will need help from the client to tweak that part for Windows.
Any guidance on how I define the path to installation I have from Anaconda?
One thing I tried without success is to include
#!/usr/bin/env python3
in the shell script which is supposed to direct it to the current user python installation that I am using from terminal, which works as intended. I am very much out of depth here.
Andy
I've never tried running Python from Igor so I don't have any solid advice.
However if I were in your position I think I'd try creating a python virtual environment that has the packages you need installed in it. Then, in the script you call from Igor, you would change to the virtual environment directory and then call "source ./bin/activate" to activate the virtual environment.
Hi,
After asking chatGPT, I think I have a solution/workaround.
In terminal issued the command
echo $(which python3)
To capture the full path of the exact python installation being used. I modified the the run_python.sh script
#!/bin/zsh
/Users/andy.hegedus/anaconda3/bin/python3 '/Users/andy.hegedus/Desktop/Image Ground Truth/IP2Py2IP/RF_model.py'
to include the full path instead of the previous python3 command.
Andy
I would have suggested that you import using the full path to the modules rather than just the module names. Perhaps your revised approach with the full path to the command properly sets the paths to the modules as well.
Would it be better to "source .bash_profile" or whatever configuration file sets up your environment in zsh now, when you create new terminal? That seems to be solution on Linux when we try to open terminal session from other scripts. This way you will have everything setup same as in terminal session where you likely test everything. And may be change to known location, in case some files stay after run, so they can be easily found. Might be more robust in case of future changes, like python upgrade etc.
In reply to Would it be better to ... by ilavsky
aclightilavsky wrote:Would it be better to "source .bash_profile" or whatever configuration file sets up your environment in zsh now, when you create new terminal?
If you want to make your process easily portable to a different machine and/or a different operating system, or even a different shell on the same machine, you want your code to have as much control over the environment as possible. Sourcing .bash_profile is almost the opposite of that, because everyone's .bash_profile will be different (if they even have one, which typically isn't the case when running on Windows). If you'll never run your process on your other machine, a colleague's machine, a HPC cloud instance, a different shell, or anywhere else then sourcing your .bash_profile might be easier, though even then I would avoid the tactic because you might change .bash_profile for some other purpose and that could break whatever you are doing in Igor.
If you need to set an environment variable, you can use SetEnvironmentVariable from Igor. If you later call ExecuteScriptText that process will inherit the value of the variable. Your ExecuteScriptText command should have the full path to whatever script you are calling. That script can then do whatever it wants to set up the environment properly--that could be activating a python virtual environment, changing the directory, etc.
If you use a Python virtual environment and pip, you can create a venv, install the packages you need, then execute "pip freeze > requirements.txt". In another virtual environment (a different machine, on the same machine, etc.) you can then call "pip install -r requirements.txt" to get essentially the same setup (assuming you used the same version of python to create the virtual environments and you copied requirements.txt from one place to the other). This also makes it much easier to test new versions of packages and/or new Python versions. Just create the new venv, install packages, and test your process. If there are problems you can always just use the old virtual environment while you figure things out.