JavaScript Remove duplicates from text/array

Question

I want to remove lines in text based on duplicate keywords. The text goes something like this.

text1,abc,text2
text3,cde,text4
text5,abc,text6
text7,gfh,text8
text9,cde,text10

I want to make it into.

text1,abc,text2
text3,cde,text4
text7,gfh,text8

The idea is to take the text, split it based on lines and put it through 2 loops. Then comparing the two arrays it would remove duplicates from it. How can I do this?

• soktinpk: Possible duplicate of stackoverflow.com/questions/9229645/…
• David Thomas: The idea sounds good, so what went wrong? Where did you get stuck?
• edinvnode: I wrote some code but didn't work. It splits the text, starts comparing but doesn't do a good job.
• edinvnode: And this is not a duplicate from that question.It's different.
• Tom W: You haven't specified on what criteria a line would be considered a duplicate - I had to guess. Also, you haven't shown what you've done so far and why the answer @DavidThomas linked wasn't adequate.

Answer 1

Here's a jsFiddle that will remove duplicates in the middle column of your data array: http://jsfiddle.net/bonneville/xv90ypgf/

var sortedAr = ar.sort(sortOnMiddleText);
var noDupsAr = removeDups(sortedAr);

Firstly is sorts on the middle column, and then it removes the duplicates. The code is pure javascript but uses jQuery to display the results. (I would prefer to use javaScript forEach instead of the for loop but it is not supported on all browsers so you would need a polyfill.)

• edinvnode: Good but can be better. I need the text sorted in lines and lines with duplicates removed, not all lines joined.
• Bonneville: @macroscripts - the text IS sorted with duplicates removed. The result IS an array with duplicates removed (I only joined them for display purposes).

Answer 2

The best way to resolve it is to use a key_value array (key-value pair), and note that the key is unique, so you will get automatically a new table with unique values. Hope this will help you.

function RemoveDuplicate(table_data) {
 var listFinale = [], key_value = {};
 for (var i = 0; i < table_data.length; i++) {
 key_value[table_data[i]] = i;
 }
 for (var key in key_value) {
 listFinale.push(key);
 }
 return listFinale.join(",");
 }