for key in val_2.keys()
-- The above loop occurs as the inner-most loop of both loops 1 and 2. With each iteration you check that key isn't equivalent to 7 other possible key values. 6 of these key values occur in the data you've presented and the 7th (condition) doesn't. It should be more efficient to replace:
for key in val_2.keys():
if(key != "block" and key != "username" and key != "setid"
and key != "setid_hash" and key != "predecessor"
and key != "time_string" and key != "condition"):
val[key]=val_2[key]
with:
# put this at the top of the test function
x_keys = set(['block', 'username', 'setid', 'setid_hash', 'predecessor', 'time_string', 'condition'])
# ...
for key in set(val_2.keys()) - x_keys:
val[key] = val_2[key]
for key in val_2.keys()
The above loop occurs as the inner-most loop of both loops 1 and 2. With each iteration you check that key isn't equivalent to 7 other possible key values. 6 of these key values occur in the data you've presented and the 7th (condition) doesn't. It should be more efficient to replace:
for key in val_2.keys():
if(key != "block" and key != "username" and key != "setid"
and key != "setid_hash" and key != "predecessor"
and key != "time_string" and key != "condition"):
val[key]=val_2[key]
with:
# put this at the top of the test function
x_keys = set(['block', 'username', 'setid', 'setid_hash', 'predecessor', 'time_string', 'condition'])
# ...
for key in set(val_2.keys()) - x_keys:
val[key] = val_2[key]
ast.literal_eval(...)
If we can remove your calls to ast.literal_eval(...) we should see a nice reduction in the run time of your loops. But, why can we remove this you ask? Consider:
m = '[0, 1, 2, ... , 9,999]' # a str representation of list w/ 10k elements, 0-9999
n = '[0, 1, 2]'
x = ast.literal.eval(m)
y = ast.literal.eval(n)
x == range(10000) # true
As you can see from the snippet above, ast.literal_eval(...) will parse and evaluate whatever string you pass it, and return a literal representation of that string (assuming of course that the string represents a valid literal). Clearly, it is more efficient to compare m and n than it is to compare x and y. Also, it doesn't appear that you are concerned with whether or not val or val_2 is a valid python literal because under the scenario that ast.literal_eval(...)throws an exception, you default to just comparing the strings returned by getter(val) and getter(val_2). Long story short you can remove the try: / except: and just use the statements you have under the except clause.
`for key in val_2.keys()` -- The above loop occurs as the inner-most loop of both loops `1` and `2`. With each iteration you check that `key` isn't equivalent to 7 other possible key values. 6 of these key values occur in the data you've presented and the 7th (`condition`) doesn't. It should be more efficient to replace: for key in val_2.keys(): if(key != "block" and key != "username" and key != "setid" and key != "setid_hash" and key != "predecessor" and key != "time_string" and key != "condition"): val[key]=val_2[key] with: # put this at the top of the test function x_keys = set(['block', 'username', 'setid', 'setid_hash', 'predecessor', 'time_string', 'condition']) # ... for key in set(val_2.keys()) - x_keys: val[key] = val_2[key]