Can't modify environment variable(assigned in Shell Script) using Python

Question

I'm trying to create a loop inside a Shell Script and I want to break out of the loop and finish the shell script execution when i find an integer different than 0 in a specific string(using Python).The problem is even after the first occurrence of an integer different than 0 in that specific string the shell script keeps executing.I tried to debug it by echoing the value of GET_OUT_OF_LOOP but it just keeps echoing 0 even after finding the kind of integer I was looking for. I already looked on the web for a way to do this but I still didn't figure it out... Here's my shell script:

#!/bin/sh 
export GET_OUT_OF_LOOP=0 
while [ $GET_OUT_OF_LOOP -ne 1 ]; do 
 python3 provas.py provas.txt 
 ./provas < provas.txt >> data.txt 
 python3 test.py data.txt 
 sh clear_data.sh 
done

And here is my Python code(test.py) where I'm trying to change the value of the GET_OUT_OF_LOOP variable using os.environ:

#!usr/env/bin python3 
import sys 
import os 
import re 
script, filename = sys.argv 
os.environ['GET_OUT_OF_LOOP'] = '0' 
fin = open("data.txt", 'r') 
for line in fin: 
 if "A percentagem de aprovação foi de" in line: 
 if int(re.search(r'\d+', line).group()) != 0: 
 print(line) 
 os.environ['GET_OUT_OF_LOOP'] = '1'

Answer 1

The python process is a subprocess of the shell process, and it can not modify environment vars of its parent process.

For your case, you can use the exit code to pass the message; i.e.

shell script:

python3 test.py data.txt || GET_OUT_OF_LOOP=1

python:

#!usr/env/bin python3 
import sys 
import os 
import re 
script, filename = sys.argv
fin = open("data.txt", 'r') 
for line in fin: 
 if "A percentagem de aprovação foi de" in line: 
 if int(re.search(r'\d+', line).group()) != 0: 
 print(line) 
 sys.exit(1)
sys.exit(0)

• Andre Korol: Thank you very much! It's now working as expected! I'm trying to upvote your answer but I can't do it because of my reputation... But I'll mark your answer as correct. Thanks again

Answer 2

That is just the way environment variables work: you can't in a sub-process change variables in the environment of the process which called it. (And in shell script, almost all lines of code, but for control structures, are external sub-processes)

What you can have is a simple unsigned byte return value of your sub-process that can be read in the shell script as the implicit $? variable.

In Python's case, you terminate the program with this return value by calling sys.exit()

So, in your shell script you can do this to assign the variable:

python3 test.py data.txt 
GET_OUT_OF_LOOP=$?

And the Python in the Python script change:

os.environ['GET_OUT_OF_LOOP'] = '1'

for

sys.exit(1)

Of course, it would be much more sane and maintainable if yu just use Python all the way from the top - the shutils module in the stdlib makes it easy to copy files around, and you, above all, get a consistent syntax across all lines of your script, much easier to use comparison operators and variables.

• Andre Korol: Thank you very much @jsbueno! I'll try to use more Python from now on as you said.
• Andre Korol: I don't know if I'm able to mark two answers as correct... And I just can't upvote your answer because of my reputation constraints...
• jsbueno: You can't mark two as correct :-) Juts mark one, and that is fine. If you still remember this when you've gt the reputation, come back and upvote it.

Answer 3

Here are two similar stackoverflow questions that might explain yours:

how-do-i-make-environment-variable-changes-stick-in-python

environment-variables-in-python-on-linux

So the real reason causing this issue is that when we run a process, the environment variables being changed by the process are only available during the process runtime, it won't change the external variables, here is a simplified script of yours to prove it:

#test.py
import os
os.environ['test_env_var'] = '1'
#test.sh
export test_env_var=0 
while [ $test_env_var -ne 1 ]; do 
 python test.py
 echo $test_env_var
done

As you might have already seen what's coming, the loop will echo $tev to be 0 forever. Hence the solution to solve this problem to my understanding, would be to out-source the change into the external system files, if it's necessary. Append changes to the configuration files of the regarding systems, for instance of this example, you can append "export test_env_var=1" into ~/.bashrc, if you are a linux bash user.