Finding items of related dependencies in Factorio

Question

Factorio is a game about building factories and about automation. Items can be crafted (just like Minecraft) yourself, or crafted automatically by assembling machines. To help me expand my factories I created a little script in Python 3 that shows me what other items that I can produce once I create factory of an item.

Given a list of Factorio item recipes in CSV format extracted from the game by this script (the sample input file is also there) and a "tolerance" which is a natural number that describes how many other dependent items I have to add, I produce a list of related recipes.

import csv
recipes = {}
tolerance = 0
def append_ignore_none(dictionary, key, value):
 if key in dictionary:
 dictionary[key].add(value)
 else:
 dictionary[key] = {value}
with open('rec.csv') as csvfile:
 reader = csv.reader(csvfile)
 for row in reader:
 append_ignore_none(recipes, row[0], row[1])
dependents = {}
for kleft, vleft in recipes.items():
 for kright, vright in recipes.items():
 if kleft != kright and vleft.issuperset(vright):
 needed_items = vleft.symmetric_difference(vright)
 if len(needed_items) <= tolerance:
 if len(needed_items) != 0:
 print("Factory of {0} can also produce {1}, just add {2}".format(kleft, kright, ",".join(needed_items)))
 else:
 print("Factory of {0} can also produce {1}".format(kleft, kright))

Sample output (tolerance = 0):

Factory of logistic-chest-storage can also produce logistic-chest-requester
Factory of logistic-chest-storage can also produce logistic-chest-active-provider
Factory of logistic-chest-storage can also produce logistic-chest-passive-provider
Factory of empty-barrel can also produce steel-chest
Factory of piercing-shotgun-shell can also produce big-electric-pole
Factory of piercing-shotgun-shell can also produce piercing-bullet-magazine
Factory of piercing-shotgun-shell can also produce heavy-armor
Factory of piercing-shotgun-shell can also produce medium-electric-pole
Factory of logistic-chest-requester can also produce logistic-chest-storage
Factory of logistic-chest-requester can also produce logistic-chest-active-provider
Factory of logistic-chest-requester can also produce logistic-chest-passive-provider
Factory of productivity-module can also produce effectivity-module
Factory of productivity-module can also produce speed-module
Factory of iron-gear-wheel can also produce pipe
Factory of iron-gear-wheel can also produce iron-chest
Factory of iron-gear-wheel can also produce iron-stick
Factory of iron-gear-wheel can also produce basic-bullet-magazine
Factory of iron-gear-wheel can also produce steel-plate
Factory of iron-gear-wheel can also produce basic-armor
Factory of big-electric-pole can also produce piercing-shotgun-shell
Factory of big-electric-pole can also produce piercing-bullet-magazine
Factory of big-electric-pole can also produce heavy-armor
Factory of big-electric-pole can also produce medium-electric-pole
Factory of chemical-plant can also produce pumpjack
Factory of player-port can also produce basic-inserter
Factory of player-port can also produce rocket-launcher
Factory of player-port can also produce radar
Factory of player-port can also produce basic-mining-drill
Factory of player-port can also produce assembling-machine-1
Factory of pipe can also produce iron-gear-wheel
Factory of pipe can also produce iron-chest
Factory of pipe can also produce iron-stick
Factory of pipe can also produce basic-bullet-magazine
Factory of pipe can also produce steel-plate
Factory of pipe can also produce basic-armor
Factory of basic-inserter can also produce player-port
Factory of basic-inserter can also produce rocket-launcher
Factory of basic-inserter can also produce radar
Factory of basic-inserter can also produce basic-mining-drill
Factory of basic-inserter can also produce assembling-machine-1
Factory of pistol can also produce shotgun-shell
Factory of energy-shield-equipment can also produce night-vision-equipment
Factory of basic-electric-discharge-defense-equipment can also produce basic-laser-defense-equipment
Factory of lubricant can also produce solid-fuel-from-heavy-oil
Factory of logistic-chest-active-provider can also produce logistic-chest-storage
Factory of logistic-chest-active-provider can also produce logistic-chest-requester
Factory of logistic-chest-active-provider can also produce logistic-chest-passive-provider
Factory of blueprint can also produce deconstruction-planner
Factory of iron-chest can also produce iron-gear-wheel
Factory of iron-chest can also produce pipe
Factory of iron-chest can also produce iron-stick
Factory of iron-chest can also produce basic-bullet-magazine
Factory of iron-chest can also produce steel-plate
Factory of iron-chest can also produce basic-armor
Factory of poison-capsule can also produce slowdown-capsule
Factory of piercing-bullet-magazine can also produce piercing-shotgun-shell
Factory of piercing-bullet-magazine can also produce big-electric-pole
Factory of piercing-bullet-magazine can also produce heavy-armor
Factory of piercing-bullet-magazine can also produce medium-electric-pole
Factory of steel-chest can also produce empty-barrel
Factory of stone-furnace can also produce stone-brick
Factory of heavy-armor can also produce piercing-shotgun-shell
Factory of heavy-armor can also produce big-electric-pole
Factory of heavy-armor can also produce piercing-bullet-magazine
Factory of heavy-armor can also produce medium-electric-pole
Factory of rocket-launcher can also produce player-port
Factory of rocket-launcher can also produce basic-inserter
Factory of rocket-launcher can also produce radar
Factory of rocket-launcher can also produce basic-mining-drill
Factory of rocket-launcher can also produce assembling-machine-1
Factory of pumpjack can also produce chemical-plant
Factory of medium-electric-pole can also produce piercing-shotgun-shell
Factory of medium-electric-pole can also produce big-electric-pole
Factory of medium-electric-pole can also produce piercing-bullet-magazine
Factory of medium-electric-pole can also produce heavy-armor
Factory of curved-rail can also produce straight-rail
Factory of iron-stick can also produce iron-gear-wheel
Factory of iron-stick can also produce pipe
Factory of iron-stick can also produce iron-chest
Factory of iron-stick can also produce basic-bullet-magazine
Factory of iron-stick can also produce steel-plate
Factory of iron-stick can also produce basic-armor
Factory of night-vision-equipment can also produce energy-shield-equipment
Factory of radar can also produce player-port
Factory of radar can also produce basic-inserter
Factory of radar can also produce rocket-launcher
Factory of radar can also produce basic-mining-drill
Factory of radar can also produce assembling-machine-1
Factory of gun-turret can also produce submachine-gun
Factory of green-wire can also produce red-wire
Factory of basic-bullet-magazine can also produce iron-gear-wheel
Factory of basic-bullet-magazine can also produce pipe
Factory of basic-bullet-magazine can also produce iron-chest
Factory of basic-bullet-magazine can also produce iron-stick
Factory of basic-bullet-magazine can also produce steel-plate
Factory of basic-bullet-magazine can also produce basic-armor
Factory of shotgun-shell can also produce pistol
Factory of basic-mining-drill can also produce player-port
Factory of basic-mining-drill can also produce basic-inserter
Factory of basic-mining-drill can also produce rocket-launcher
Factory of basic-mining-drill can also produce radar
Factory of basic-mining-drill can also produce assembling-machine-1
Factory of basic-transport-belt can also produce burner-inserter
Factory of slowdown-capsule can also produce poison-capsule
Factory of deconstruction-planner can also produce blueprint
Factory of burner-inserter can also produce basic-transport-belt
Factory of effectivity-module can also produce productivity-module
Factory of effectivity-module can also produce speed-module
Factory of steel-plate can also produce iron-gear-wheel
Factory of steel-plate can also produce pipe
Factory of steel-plate can also produce iron-chest
Factory of steel-plate can also produce iron-stick
Factory of steel-plate can also produce basic-bullet-magazine
Factory of steel-plate can also produce basic-armor
Factory of speed-module can also produce productivity-module
Factory of speed-module can also produce effectivity-module
Factory of stone-brick can also produce stone-furnace
Factory of basic-laser-defense-equipment can also produce basic-electric-discharge-defense-equipment
Factory of assembling-machine-1 can also produce player-port
Factory of assembling-machine-1 can also produce basic-inserter
Factory of assembling-machine-1 can also produce rocket-launcher
Factory of assembling-machine-1 can also produce radar
Factory of assembling-machine-1 can also produce basic-mining-drill
Factory of submachine-gun can also produce gun-turret
Factory of red-wire can also produce green-wire
Factory of logistic-chest-passive-provider can also produce logistic-chest-storage
Factory of logistic-chest-passive-provider can also produce logistic-chest-requester
Factory of logistic-chest-passive-provider can also produce logistic-chest-active-provider
Factory of basic-armor can also produce iron-gear-wheel
Factory of basic-armor can also produce pipe
Factory of basic-armor can also produce iron-chest
Factory of basic-armor can also produce iron-stick
Factory of basic-armor can also produce basic-bullet-magazine
Factory of basic-armor can also produce steel-plate
Factory of solid-fuel-from-heavy-oil can also produce lubricant
Factory of straight-rail can also produce curved-rail

How can I improve my script (Python best practices and maintainability)?

Answer 1

def append_ignore_none(dictionary, key, value):
 if key in dictionary:
 dictionary[key].add(value)
 else:
 dictionary[key] = {value}

Dictionaries already have a method to do that, it's called setdefault:

dictionary.setdefault(key, set()).add(value)

But since that's kind of clunky, and you do it every time you update the dictionary, use make recipes a defaultdict(set), and it's just recipes[key].add(value).

for kleft, vleft in recipes.items():
 for kright, vright in recipes.items():

Iterate over the cartesian product instead using itertools:

for kleft, vleft, kright, vright in it.product(recipes.items(), repeat=2):
 ...

Sets have easy-to-read operator forms for a lot of operations, so:

if kleft != kright and vleft.issuperset(vright):
 needed_items = vleft.symmetric_difference(vright)

Can be:

if kleft != kright and vleft >= vright:
 needed_items = vleft ^ vright

Are you sure that this does the right thing, by the way? I would have expected either needed_items should be the asymmetric difference (vleft - vright), or the condition to be vleft.is_disjoint(vright).

You define dependents = {}, but you don't seem to use it. I'm guessing that you'll be expanding the script later and you plan to use it then, but it's better to leave it out until you are ready to write that code.

Instead of doing:

if len(needed_items) <= tolerance:
 if len(needed_items) != 0:
 print("Factory of {0} can also produce {1}, just add {2}".format(kleft, kright, ",".join(needed_items)))
 else:
 print("Factory of {0} can also produce {1}".format(kleft, kright))

You can flatten this and save a level of indentation by reversing the order of the conditions and using elif. While we're at it, the preferred way of testing if a set (or any other collection) is empty is just if needed_items:

if not needed_items:
 print("Factory of {0} can also produce {1}".format(kleft, kright))
elif len(needed_items) <= tolerance:
 print("Factory of {0} can also produce {1}, just add {2}"
 .format(kleft, kright, ",".join(needed_items)))

I've also line wrapped the second print, since it was very long.

Finally, you can omit the numbers inside the braces in the format strings, like this:

"Factory of {} can also produce {}".format(kleft, kright)

• milleniumbug: Wow, that defaultdict really nails it! I like it. About the symmetric/asymmetric difference, yes, in fact this does the wrong thing, as I noticed soon after the first answer was posted, but I didn't want to invalidate the core reviews after this.

Answer 2

This part is an if in an if in an if in a for in a for. Not only is that ugly, it's not idiomatic in most languages. The moment you go deeper than 3 levels you should be asking yourself if this is really the way you want to go.

for kleft, vleft in recipes.items():
 for kright, vright in recipes.items():
 if kleft != kright and vleft.issuperset(vright):
 needed_items = vleft.symmetric_difference(vright)
 if len(needed_items) <= tolerance:
 if len(needed_items) != 0:
 print("Factory of {0} can also produce {1}, just add {2}".format(kleft, kright, ",".join(needed_items)))
 else:
 print("Factory of {0} can also produce {1}".format(kleft, kright))

Those for loops iterate over the same items! You're also calling the same function for each loop, while once would suffice. Even if you'd the value of that function twice, you can store it in a variable and look up the value of that variable instead of calling the function. This is more efficient.

I suspect (but I will leave the implementation and thinking about the consequences as homework for the OP) the following construct may be of use in shortening the if statements:

minimum < value <= maximum

The above is valid for all values above minimum and up to and including maximum. However, with the current data flow it's not possible to just lump the latest two if statements together. There's a bit more work involved.

Since the length of your print statements is quite long, you could define functions for that instead. It would definitely increase the readability of your code. You could even make it generic enough so the function covers both cases. You could even pass the len(needed_items) as an argument to that function and let it figure out what to print.

• milleniumbug: The two loops are iterating over every pair of recipes. (this reminded me of the itertools.product function). About the duplicated len statement, I'm trying to enable i18n for the string (the reason .format is used), but I have no idea how to remove the duplication of format strings. Thanks for the input!
• wilx: @milleniumbug: If you are thinking of L10N then keep the formatting strings separate and whole.

Answer 3

Personally, I dislike heavy nesting, so I would rewrite the main loop to use continue to continue early:

for kleft, vleft in recipes.items():
 for kright, vright in recipes.items():
 if not (kleft != kright and vleft.issuperset(vright)):
 continue
 needed_items = vleft.symmetric_difference(vright)
 if len(needed_items) > tolerance:
 continue
 if len(needed_items) != 0:
 print("Factory of {0} can also produce {1}, just add {2}".format(kleft, kright, ",".join(needed_items)))
 else:
 print("Factory of {0} can also produce {1}".format(kleft, kright))

• Maarten Fabré: you can decrease the nesting even further by using itertools.product

Answer 4

As per Mast's, wilx's and lvc's suggestions, I revised my code as follows:

import collections, csv, itertools
recipes = collections.defaultdict(set)
tolerance = 0
with open('rec.csv') as csvfile:
 reader = csv.reader(csvfile)
 for row in reader:
 recipes[row[0]].add(row[1])
def get_text(base_factory_item_name, related_factory_item_name, needed_items_list):
 default_text = "Factory of {} can also produce {}, just add {}"
 noadd_text = "Factory of {} can also produce {}"
 if needed_items_list:
 return default_text.format(base_factory_item_name,
 related_factory_item_name,
 ",".join(needed_items_list))
 else:
 return noadd_text.format(base_factory_item_name, related_factory_item_name)
for left, right in itertools.product(recipes, repeat=2):
 if left != right and recipes[right].issuperset(recipes[left]):
 needed_items_list = recipes[right] - recipes[left]
 if len(needed_items_list) <= tolerance:
 print(get_text(left, right, needed_items_list))

The kleft, vleft, kright and vright variable names aren't descriptive enough so instead of iterating over dict.items() and unpacking the tuple I iterate over dict and use operator [] of the dict.

The itertools.product is also used instead of two nested loops.

The localization and producing the text is too important and warrants to be put in separate function. (single responsibility principle)

The code was also buggy because of using the symmetric difference. The difference is calculated with operator -, but I find >= to be less readable than issuperset, so I didn't use it.

The one thing I can improve yet is to provide a better name for the "get the localized text" function.

Answer 5

Factory of pipe can also produce steel-plate

This might be correct from the problem statement as you've presented, but it's not how it works in "real" factorio. You need a smelter to make steel plates. You need an assembler to make pipes.

You thus might want to revise your input script to either filter such recipes, or add an extra column for factory types.

• milleniumbug: Yeah, that's a simplification I'm willing to live with (my use case was "if I have belts that deliver ingredients that can produce X, what else can I produce by extending the belts and placing new producers"). The extended version would be useful for playing with mods though, where there are more different factory types.