Dynamic Variables from JSON Parameter File

Question

I want to assign Python variables imported from a JSON file. This question had an interesting answer using a classmethod, but I couldn't get it to work and I'm not allowed to comment...

So, let's consider a really simple example: I want to evaluate z = x^2+y^2 but I want to be able to define x and y in a JSON file. My json file (params.json) might look like:

{
 "x":3,
 "y":2
}

Then I could load an load the file and generate dynamic variables:

with open("params.json", "r") as read_file:
 params = json.load(read_file)
for k, v in params.items():
 vars()[k] = v
 
z = x^2+y^2

This works, but it seems dangerous to dynamically generate variables. Is there a standard/smarter way to do this?

• Doc Brown: My Python is a little bit rusty, but why don't you just use x=params['x'], y=params['y'] (with some error checking if 'x' and 'y' exist as keys in params)?
• Liz Livingston: It's actually quite a long list of variables defined in the parameters file, I just simplified it for my question. I was hoping for a safer approach than dynamic that might be a bit less code than listing them all out.
• Liz Livingston: I made an active effort to make the question concise and clear. Also hopefully generic enough for others with similar yet different goals... Thought it was rather obvious I was importing just two variables...
• Doc Brown: Well, your question gave an example with two variables, your comment says you have a very long list of variables, and another comment says "I do have functions that I want to add to the JSON parameter file as well" which is not obvious from the example above. So it seems the example is oversimplifying.

Answer 1

This depends a lot on your security goals, and on what kind of user interface you want to offer to the author of these expressions.

Loading variables into the local scope does work, since Python is a very dynamic language. There's a risk though that the variables might re-define existing objects, thus breaking your code – what if there's a variable called len, for example?

Therefore, it's usually safer to avoid running the user input in a Python context. Instead:

• define a simple programming language for these expressions
• write an interpreter that executes the expressions

Python does have tools to help here. We can parse strings as Python code via the ast module. This returns a data structure that represents the syntax, and doesn't execute anything (though the parser isn't necessarily safe against malicious inputs). We can take the data structure, walk it, and execute it according to the rules we define – such as by resolving variables only from a dictionary. Example code for Python 3.10:

import ast
def interpret(code: str, variables: dict) -> dict:
 module: ast.Module = ast.parse(code, mode='exec')
 for statement in module.body:
 _interpret_statement(statement, variables)
 return variables
def _interpret_statement(statement: ast.stmt, variables: dict) -> None:
 match statement:
 case ast.Assign(targets=[ast.Name(id=name)], value=value):
 variables[name] = _interpret_expr(value, variables)
 return
 case other:
 raise InterpreterError("Syntax not supported", other)
def _interpret_expr(expr: ast.expr, variables: dict) -> Any:
 match expr:
 case ast.BinOp(left=left_ast, op=op, right=right_ast):
 left = _interpret_expr(left_ast, variables)
 right = _interpret_expr(right_ast, variables)
 return _interpret_binop(left, op, right)
 case ast.Name(id=name):
 return variables[name]
 case ast.Constant(value=(int(value) | float(value))):
 return value
 case other:
 raise InterpreterError("Syntax not supported", other)
 
def _interpret_binop(left: Any, op: ast.operator, right: Any) -> Any:
 match op:
 case ast.Add(): return left + right
 case ast.Sub(): return left - right
 case ast.Mult(): return left * right
 case ast.Div(): return left / right
 case ast.Pow(): return left**right
 case other:
 raise InterpreterError(
 "Operator not supported",
 ast.BinOp(ast.Name("_"), other, ast.Name("_")))
class InterpreterError(Exception):
 def __init__(self, msg: str, code: Optional[ast.AST] = None) -> None:
 super().__init__(msg, code)
 self._msg = msg
 self._code = code
 def __str__(self):
 if self._code:
 return f"{self._msg}: {ast.unparse(self._code)}"
 return self._msg

This can then be used to interpret commands, returning a dictionary with all the variables:

>>> interpret("z = x**2+y**2", {"x": 3, "y": 2})
{'x': 3, 'y': 2, 'z': 13}

While this allows you to interpret the Python code however you want (you control the semantics), you are still limited to Python's syntax. For example, you should use the ** operator for exponentiation, not Python's ^ xor-operator.

If you want your own syntax, then you'll probably have to write your own parser. There are a variety of parsing algorithms and parser generators, but I'm partial to hand-written "recursive descent". This generally involves writing recursive functions of the form parse(Position) -> Optional[tuple[Position, Value]] that gradually consume the input. I have written an example parser and interpreter using that strategy, and have previously contrasted different parsing approaches in an answer about implementing query languages in a Python program.

• Doc Brown: This is definitely way-too-complicated for getting the values of two variables x and y from the given JSON file.
• amon: @DocBrown Oh right, the question does only ask about the input data, whereas this answer would apply if the formula were also an input. Please consider turning your comment in an actual answer! I nevertheless had fun writing the code for this answer, the calculator language problem is essentially a code kata for me at this point.
• Doc Brown: Well, I will wait for a reply from the OP - if that's what they really meant, it seems to be way-too-trivial.
• Liz Livingston: I actually do have functions that I want to add to the JSON parameter file as well, so this answer is very helpful, thank you!