Eliminate Duplicates of a List (Python)

Question

I have a list of lists and need to eliminate all lists that have the same 3rd element, and save 1.

For example:

x=[[1,2,3],[1,3,3],[5,6,3],[2,4,6],[8,5,9],[10,5,9]]

could turn into:

x=[[1,2,3],[2,4,6],[8,5,9]]

I've attempted a solution that eliminates all lists that have the same 2nd index using lambda, but I don't know how to save one of each element like set() does--I just get:

x=[[2,4,6]]

• inspectorG4dget: Please post your attempt so that we can tell you where it falls short
• roippi: where did [1,3,3] and [5,6,3] go?
• user2357112: @roippi: Presumably, since the element at index 2 in both of those lists is 3, they disappeared because [1, 2, 3] was kept. It's not clear whether the OP means index 2 counting from 0, or second element counting from 1.
• roippi: @user2357112 the "second element" is the element at index 1.
• Noob Coder: I meant index 2 counting from 0, sorry.

Answer 1

Python sets should still work for you:

>>> seen = set()
>>> result = []
>>> for s in x:
 e = s[2]
 if e not in seen:
 result.append(s)
 seen.add(e)
>>> result
[[1, 2, 3], [2, 4, 6], [8, 5, 9]]

• Noob Coder: I kept getting set() errors like "unhashable type: list", had no idea that you can just define a variable to be an empty set like that. TY!
• rdodev: Bah, with Raymond in play is not even fair :)

Answer 2

You could also use a dict here, which could be a bit more flexible later:

data =[[1,2,3],[1,3,3],[5,6,3],[2,4,6],[8,5,9],[10,5,9]]
one_of = {el[2]:el for el in data}.values()
# [[10, 5, 9], [5, 6, 3], [2, 4, 6]]

• thefourtheye: Jon, we ll get only the last duplicate value but he wants only the first duplicate value.
• dansalmo: Just reverse the data if you want the first values.
• Jon Clements: @thefourtheye well... the first occurrence is a reasonable assumption, however, it's not actually specified that's the case :) need to eliminate all lists that have the same 3rd element, and save 1

Answer 3

You can use set if you define a helper class

>>> L = [[1,2,3],[1,3,3],[5,6,3],[2,4,6],[8,5,9],[10,5,9]]
>>> class Element(list):__hash__=lambda s:hash(s[2]);__eq__=lambda s,o:s[2]==o[2]
... 
>>> set(map(Element, L))
set([[8, 5, 9], [1, 2, 3], [2, 4, 6]])

Answer 4

You can do this with list comprehension and helper set:

>>> t = set()
>>> y = [l for l in x if l[2] not in t and not t.add(l[2])]
>>> y
[[1, 2, 3], [2, 4, 6], [8, 5, 9]]

Answer 5

The itertools docs come with a nice set of recipes, one of which, unique_everseen does exactly what you want: "List unique elements, preserving order. Remember all elements ever seen."

You can copy it paste it from the recipes, or you can pip install more-itertools and just use it as more_itertools.unique_everseen.

Or you can look at the code, and see that it's essentially just a slightly more flexible and slightly optimized version of exactly what Raymond Hettinger's answer does.*

And that's exactly what the recipes are there for—when you want to do something to an iterable, there's often a recipe for it, which not only does what you want, but is so simple that if you take a couple minutes to work out how it works, you'll know how to do it yourself in the future.

---

* This shouldn't surprise anyone who knows anything about Python, or just looks at the source to itertools...

• abarnert: @RomanPekar: In that case, I should have added a link to it. (Which I've now done.)

Answer 6

>>> L = [[1,2,3],[1,3,3],[5,6,3],[2,4,6],[8,5,9],[10,5,9]]
>>> s = set(i[2] for i in L) 
>>> [s.remove(i[2]) or i for i in L if i[2] in s]
[[1, 2, 3], [2, 4, 6], [8, 5, 9]]