Remove duplicate values in list of list python

Question

Trying to remove duplicates in list of list and storing it.

Original List:

list1 = [[a, b, a, b], [b, c, d, c], [a, c, c]]

Looking for output:

list1 = [[a, b], [b,c,d], [a,c]]

My code:

unique_list = []
for sublist in list1:
 element = [elem for elem in sublist if elem not in unique_list]
 if len(element):
 unique_testset.append(element)

My code appends a sublist multiple times and doesn't get rid of the duplicates.

• OmegaOdie: stackoverflow.com/questions/2213923/…
• Stef: Does this answer your question? stackoverflow.com/questions/7961363/… without_duplicates = [list(set(sublist)) for sublist in list1]
• Stef: @OmegaOdie There is a confusion here; I believe the OP is trying to remove duplicate elements in each sublist; the question you linked is trying to remove duplicate sublists.
• Muhammad Rizwan Munawar: Here your answer stackoverflow.com/questions/55349986/…

Answer 1

Removing duplicates the simple way

A classic, efficient way to remove duplicates from a list in python is to build a set from the list: removing duplicates in lists

list_with_dups = [1, 1, 2, 3, 2]
list_without_dups = list(set(list_with_dups))

You can apply this method repeatedly using a list comprehension:

list1 = [['a', 'b', 'a', 'b'], ['b', 'c', 'd', 'c'], ['a', 'c', 'c']]
without_duplicates = [list(set(sublist)) for sublist in list1]
# = [['b', 'a'], ['d', 'b', 'c'], ['c', 'a']]

Removing duplicates whilst conserving order

Applying How do you remove duplicates whilst conserving order? to a list of lists:

def f7(seq):
 seen = set()
 seen_add = seen.add
 return [x for x in seq if not (x in seen or seen_add(x))]
list1 = [['a', 'b', 'a', 'b'], ['b', 'c', 'd', 'c'], ['a', 'c', 'c']]
without_duplicates = [f7(sublist) for sublist in list1]
# = [['a', 'b'], ['b', 'c', 'd'], ['a', 'c']]

• Stef: Dear @MuhammadRizwanMunawar, thank you for trying to improve my answer by suggesting an edit. Your edit suggestion to my answer is actually just deleting parts of my answer rather than improving it. This is doubly annoying because: 1) I cannot reject your edit. 2) I wanted to edit my answer to make it clearer, but I cannot, because your suggested edit prevents me from editing further. Please revert your edit. I do not understand why you decided to delete parts of my answer and this is very annoying.
• Miztory: Thanks! This works for me but it changes the order of the sublist. Is there any way to preserve the order?
• Stef: @Miztory Yes, set structures in python do not conserve order. Here is a stackoverflow question about removing duplicates whilst conserving order: stackoverflow.com/questions/480214/…
• Miztory: @Stef I think I'll be able to continue with my work even with a different order.Thanks for the help. :)
• Stef: @Miztory I've updated the answer to conserve order.

Answer 2

try this one...

 list1 = [[a, b, a, b], [b, c, d, c], [a, c, c]]
 
 import itertools
 list1.sort()
 list(list1 for list1,_ in itertools.groupby(list1))

Answer 3

One way you can achieve this is using:

list1 = [[a, b, a, b], [b, c, d, c], [a, c, c]]

for sublist in list1:
 l = list(set(sublist))
 if l:
 unique_list.append(l)

set removes all the duplicate values and we have unique values

• Muhammad Rizwan Munawar: try to implement with respect to question not in jorunal. Thanks
• AnsFourtyTwo: if len will always evaluate to True. You could just leave it (even if you use it correctly).

Answer 4

Use This code it'll help you

 import numpy as np
 lis=[['a','b','a'], ['b','c','d'], ['c','a','c']]
 uni=np.unique(lis)
 print(uni)