Python with alternative keywords

Question

A while ago, a user on Programming Puzzles and Code Golf had an idea:

How about a language where all of the core commands are PPCG usernames? -- Helka Homba

A list was made of username/(keyword|builtin function) pairs, but nobody really wanted to actually make the code, so it kinda went on hold.

Recently (read: yesterday) another user suggested using the tokenize module, so I did. With his help, I finished some py2 code for replacing the names.

Here is my code:

import tokenize
import ast
import sys
def handle_token(type, token, (srow, scol), (erow, ecol), line):
 # Return the info about the tokens, if it's a NAME token then replace it
 if tokenize.tok_name[type] == "NAME":
 token = token_names.get(token, token)
 return (type, token, (srow, scol), (erow, ecol), line)
def run(assignments="assignments.txt",open_from="peoples.txt"):
 with open(assignments, "r") as f:
 # Read the replacements into token_names
 global token_names
 token_names = ast.literal_eval(f.read())
 with open(open_from) as source:
 # Get the tokenized version of the input, replace it, and untokenize into pretty output
 tokens = tokenize.generate_tokens(source.readline)
 handled_tokens = (handle_token(*token) for token in tokens)
 output = tokenize.untokenize(handled_tokens)
 with open(open_from[:-4]+"-output.txt",'w') as outfile:
 # Write to the output file
 outfile.write(output)
 return output
if __name__ == "__main__":
 if len(sys.argv) > 1:
 if len(sys.argv) > 2:
 try:exec run(assignments=sys.argv[1],open_from=sys.argv[2])
 except:pass
 else:
 try:exec run(assignments=sys.argv[1])
 except:pass
 else:
 try:exec run()
 except:pass

This is the content of main.py. It uses assignments.txt, which is a file containing a python dict of the replacement pairs.

{"Martin":"False",
"Geobits":"None",
"Dennis":"True",
"adnan":"and",
"rainbolt":"as",
"buttner":"assert",
"flawr":"break",
"aditsu":"class",
"katenkyo":"continue",
"quill":"def",
"nathan":"del",
"hobbies":"elif",
"helkahomba":"else",
"irk":"except",
"ender":"finally",
"peter":"for",
"conor":"from",
"gnibbler":"global",
"calvins":"if",
"obrien":"import",
"taylor":"in",
"fryamtheeggman":"is",
"starman":"lambda",
"sp3000":"nonlocal",
"phinotpi":"not",
"xnor":"or",
"maltysen":"pass",
"mego":"raise",
"alex":"return",
"easterly":"try",
"molarmanful":"while",
"minxomat":"with",
"optimizer":"yield",
"mbomb007":"abs",
"digital":"all",
"trauma":"any",
"asciionly":"ascii",
"zyabin":"bin",
"bkul":"bool",
"chris":"chr",
"jesteryoung":"classmethod",
"elendia":"enumerate",
"gcampbell":"eval",
"fatalize":"filter",
"sandbox":"help",
"zgarb":"id",
"phase":"input",
"loovjo":"int",
"minibits":"issubclass",
"lynn":"len",
"doorknob":"map",
"upgoat":"max",
"briantompsett":"memoryview",
"downgoat":"min",
"jimmy23013":"open",
"destructiblewatermelon":"ord",
"ninjabearmonkey":"pow",
"you":"print",
"djmcmayhem":"range",
"qwerpderp":"round",
"orlp":"sorted",
"timmyd":"staticmethod",
"muddyfish":"sum",
"balint":"super",
"trichoplax":"tuple",
"quartata":"zip"}

An example peoples.txt:

peter i taylor djmcmayhem(10):
 you(list(set(i)))

Should output:

for i in range(10):
 print(list(set(i)))

What does the code do? It takes in an input file (by default peoples.txt in the same dir as main.py) and translates it from "People's Python" to standard python, and executes the result. It also writes the compiled "normal" code into <input_file_name>-output.txt, by default peoples-output.txt

Run as python main.py custom/assignments.txt path/to/inputfile.txt (inputfile.txt is the People's Python code and assignments.txt is the dict of assignments you're using)

In terms of style, how can I improve this?

The things I notice:

• Probably could some more comments.

• Replace ast.literal_eval with another method that doesn't involve using any eval. Not sure if this is even possible though. Would best practice say to just include the massive dictionary in main.py?

• Is using blank except:s with no special error (i.e. except SyntaxError:) okay in this situation?

• Downgoat: I found typo. djmcmayhem should be spelt drhamjam
• Pavel: @Downgoat how do you figure that?

Answer 1

Some Musings about Exceptions

(1) NEVER pass on every single exception!

You're basically absorbing every possible exception without usefulness. Normally, to prevent runtime crashes, I do this with my excepts when I want to catch all:

try:
 # some code here...
except Exception as e:
 print "An exception has occurred:\n%s" % str(e)

... which will put a nicer message. Of course, you can expand this by defining different types to capture in the except blocks. There may be cases for this, but since this isn't a code golf challenge, you can do this here instead.

(2) "Blank" except blocks

There's a rule of thumb that I abide by: "In most cases, you should always try and catch the most narrow exception that you expect to end up seeing in a try/except block, with an ultimate "catch-all" block later to handle any unexpected exceptions." That basically means that if I'm trying to catch an "Invalid JSON" error when running json.load (you'll see why I mention this later), I'd do something like this:

with open('jsondata.json', 'r') as f:
 try:
 data = json.load(f)
 except ValueError:
 print "Could not properly parse JSON data, is the file 'jsondata.json' comprised of actual JSON data?"

In effect, we do something different with a ValueError (we could pass, or we could simply quit, or we can print a nice message like this does), but for all other Exceptions will not capture them and will raise whatever exception it was trying to raise. Let's say, though, this was in a run() function, and I call the run function in a manner like you do in your code:

if __name__ == "__main__":
 try:
 run()
 except Exception as e:
 print "An unhandled exception has occurred:\n%s" % str(e)

Since most errors inherit from Exception this captures all other errors, but not warnings, this will allow for us to capture unhandled exceptions and 'handle' them as a "final option" type of try/except block, for when they aren't handled in other blocks below run directly.

There are always rare cases where you want to do nothing when an exception happens, so you can use pass in those cases, but in 99% of all cases, you should not be simply 'allowing exceptions to pass on without raising some notice'.

---

Replace ast with json instead

JSON data is basically a structured dict. Since you're storing a dict and it meets the most basic JSON formatting, we can just parse your assignments file as JSON instead, and remove the literal_eval and import ast. This does require you to import json instead of import ast but this is a more sane approach:

with open(assignments, "r") as f:
 # Read the replacements into token_names
 global token_names
 token_names = json.load(f)

This way, we don't have to worry about ast and having a literal evaluation that could cause evil in the future. This also permits us to error out proper with an exception if we don't have a valid assignments file. (And we can handle it with customized error messages if we wish).

As for storing the 'massive dict' in main.py, if it's likely to expand in the future leave it in its own file, and continue to load it as a JSON object as my suggestion here has.

---

This isn't code golf! Whitespace is a good thing!

Your code is hard to read when your try/except blocks and such are all golfed to remove whitespace. Add whitespace for readability.

---

You don't need that many try/accept blocks around your 'run' calls, and you don't need exec either!

You have four separate try/except blocks, yet all they do is silently let exceptions go by. Taking into account I don't suggest using 'pass' on exceptions that could be important, we can just simply take a page from my book, and change your code to have one try/except block wrapped around all the run calls, and handle exceptions whenever they come up, without having individual try/except blocks just for each call.

You also don't need the exec calls you have in here.

So, ultimately, you save some code:

if __name__ == "__main__":
 try:
 if len(sys.argv) > 1:
 if len(sys.argv) > 2:
 run(assignments=sys.argv[1], open_from=sys.argv[2])
 else:
 run(assignments=sys.argv[1])
 else:
 run()
 except Exception as e:
 print "An exception has occurred:\n%s" % str(e)

---

Nitpicking Section

I'mma nitpick a little here with some of your code, and bring more suggestions that are minor/aesthetic in nature, rather than code-critical. These suggestions are reflected below though.

type is actually a builtin, use a different name in handle_token

This one's fairly obvious, but type is actually a built-in. Shadowing built-ins is bad if you eventually could use a builtin, so let's replace type with type_ in handle_token, to get rid of the "Shadows built-in 'type'" problems.

Unnecessary parentheses on 'return' in handle_token

You don't need the parentheses around the 'return' object. It'll return all the objects you specify as a tuple anyways, and operate as we expect it to.

She-bang line: python2 and python are identical

NOTE: When this answer was written, the code had not yet been updated to reflect OP's removal of the shebang line. The shebang line works, but it needs tweaked as per this part of the review; it's added back in for this code, accordingly, because it's part of the original code, but needed a tweak. Refer to the revision history for the original question for more.

In most cases, the current guidelines that python is Python 2 and python3 is Python 3 from Python upstream are still in play. Consider any Linux system - python3 is the Python 3 interpreter, but a call to python when both Python 2 and 3 are installed is a call to Python 2.7. Because python2 may not exist in the env either, it's sane to call python instead of python2. Alternatively, remove the shebang line so you can't call the program as a script directly and have to invoke the file via a call to python with python main.py [args].

Use of a global but not defining it at the Module level

This is more or less me griping about PyCharm (my IDE) being annoying. It reports that your use of global token_names is not defined at the module level. While ultimately this has no real effect on the execution of your program, you can easily avoid this notice by just putting token_names = None in the if __name__ == "__main__": block right before the try/except block. And I did this, just to suppress the IDE alert. (It doesn't affect execution apparently, and still works as it should)

---

---

This is your code with all my suggestions from above:

#!/usr/bin/env python
import tokenize
import sys
import json
def handle_token(type_, token, (srow, scol), (erow, ecol), line):
 # Return the info about the tokens, if it's a NAME token then replace it
 if tokenize.tok_name[type_] == "NAME":
 token = token_names.get(token, token)
 return type_, token, (srow, scol), (erow, ecol), line
def run(assignments="assignments.txt", open_from="peoples.txt"):
 with open(assignments, "r") as f:
 # Read the replacements into token_names
 global token_names
 token_names = json.load(f)
 with open(open_from) as source:
 # Get the tokenized version of the input, replace it, and untokenize into pretty output
 tokens = tokenize.generate_tokens(source.readline)
 handled_tokens = (handle_token(*token) for token in tokens)
 output = tokenize.untokenize(handled_tokens)
 with open(open_from[:-4]+"-output.txt", 'w') as outfile:
 # Write to the output file
 outfile.write(output)
 return output
if __name__ == "__main__":
 token_names = None
 try:
 if len(sys.argv) > 1:
 if len(sys.argv) > 2:
 exec run(assignments=sys.argv[1], open_from=sys.argv[2])
 else:
 run(assignments=sys.argv[1])
 else:
 run()
 except Exception as e:
 print "An exception has occurred:\n%s" % str(e)

• user95591: And my assignments.txt is already formatted a JSON object, correct? Thanks the advice!
• Thomas Ward: @Riker Correct - a basic dict like the one you have is also a compatible JSON object - so we just load it as if it is a JSON object and all works as we expect it to. No more literal_eval :)
• Thomas Ward: I have reread your quesiton, you may actually want to use exec here, but I have a better approach to it. If you repost for a supplementary CR I may be able to give some suggestions.