How to remove duplicates from list in python

Question

I have a simple code here:

a=[['bcn09','113','shift1'],['bcn09','113','shift1'],['bps01','132','shift2']]
b=[]
for i in range (len(a)):
 if a[i] not in b:
 b.append([a[i]])
print (b)

The output i get is

b=[['bcn09','113','shift1'],['bcn09','113','shift1'],['bps01','132','shift2']]

i.e. the same as a The output i need is

b=[['bcn09','113','shift1'],['bps01','132','shift2']]

What am i doing wrong?

Thanks in advance

• SyntaxVoid: Remove the outer brackets from [a[i]]. You're appending a list of the single element to b which I don't think is what you want to do. Your line should read b.append(a[i])
• Gareth Latty: As general advice, you can just iterate directly over a here (i.e: for item in a: ...), no need to use range() and indexers, which obfuscated your issue. It is also worth noting that membership tests on a list (i.e: item not in b) get slower the more items there are in the list, so deduplicating this way won't scale well.

Answer 1

Very close! You could try this approach, hopefully it makes sense:

a=[['bcn09','113','shift1'],['bcn09','113','shift1'],['bps01','132','shift2']]
noDups = []
for i in a:
 if i not in noDups:
 noDups.append(i)
print(noDups)

Output:

>>>[['bcn09', '113', 'shift1'], ['bps01', '132', 'shift2']]

Answer 2

There is no need to create a new list when appending to 'b'.

Modify the below line from -

b.append([a[i]])

to

b.append(a[i])

The new output is (which you want) -

[['bcn09', '113', 'shift1'], ['bps01', '132', 'shift2']]

This explains more about the 'in' membership test operator.

Answer 3

For the most complete answer to getting only unique values from an iterable, itertools gives a recipe (which can also be found in the more-itertools package on PyPI) for this:

def unique_everseen(iterable, key=None):
 "List unique elements, preserving order. Remember all elements ever seen."
 # unique_everseen('AAAABBBCCDAABBB') --> A B C D
 # unique_everseen('ABBCcAD', str.lower) --> A B C D
 seen = set()
 seen_add = seen.add
 if key is None:
 for element in filterfalse(seen.__contains__, iterable):
 seen_add(element)
 yield element
 else:
 for element in iterable:
 k = key(element)
 if k not in seen:
 seen_add(k)
 yield element

This implementation is tested, optimised, maintains order, is a generator so it will work on infinite iterables or cases where you need to be lazy, and allows you to use a key function.

>>> a=[['bcn09','113','shift1'],['bcn09','113','shift1'],['bps01','132','shift2']]
>>> list(unique_everseen(tuple(item) for item in a))
[('bcn09', '113', 'shift1'), ('bps01', '132', 'shift2')]

Note the change to tuples so the elements are hashable and can be added to the set. You can of course reverse this at the end if needed, although in most cases I can imagine tuples will probably be fine. (In the same way, I am creating a list from the generator to show the output, but most of the time you should be able to just work with the generator directly).

Answer 4

you can achive by using itertools

import itertools
a = [['bcn09','113','shift1'],['bcn09','113','shift1'],['bps01','132','shift2']]
a.sort()
new_num = list(a for a,_ in itertools.groupby(a))
print("New List", new_num)

• Gareth Latty: This doesn't seem to do anything close to what the question asks, unless I'm missing something.
• Jay Kakadiya: author changed the requirement I also changed the code based on that. thanks for comment on the post and letting me know.
• Gareth Latty: It is worth noting that this answer now loses the original order of the items in the list, which may or may not be an issue for an individual case.