Remove duplicate values in an attribute using regex and fme

Question

I have an FME process that is duplicating some values in an attribute. When I try to use regex in FME to clean it up it is deleting the separator (see screengrab below). Any ideas on how I can change the regex or process in FME to just get the required values.

So from

164511|119958|164512|164511|119958|164512|164511|119958|164512
161634|161635|161636|161634|161635|161636|161634|161635|161636|161634|161635|161636|161634|161635|161636|161634|161635|161636
1114|1114|1114|1114|1114

I just want

164511|119958|164512
161634|161635|161636
1114

The value in AttributeManager from FME is @ReplaceRegEx(@Value(Name_Ids),"(?:^|\G)(\b\w+\b),?(?=.*1円)\|",1円)

I have tried to replace \b),?(? with \b)\|(?to use the | seperator but that doesn't work either. This is based on https://superuser.com/a/1343524/639219

The regex tester is https://regex101.com/r/1YhC59/1

enter image description here

Answer 1

You are almost there, instead of

@ReplaceRegEx(@Value(Name_Ids),"(?:^|\G)(\b\w+\b),?(?=.*1円)\|",1円)

you need

@ReplaceRegEx(@Value(Name_Ids),"(?:^|\G)(\b\w+\b),?(?=.*1円)\|","")

• GeorgeC: Thanks but for my future knowledge can you let me know what the ` 1円` at the end does as against ""? That's the only difference I noticed.
• nielsgerrits: I don’t know, fiddled around and it just worked :)

Answer 2

Without regex you could always use:

• An AttributeSplitter transformer to turn the attribute into a list
• A ListDuplicateRemover transformer to remove duplicate list entries
• A ListConcatenator transformer to turn the list back into an attribute

But also, could we stop the duplication happening from the very start? Can you clarify how that string is being created?

• christoph: Wow, 3 transformers for a simple task ;-) ... would be nice if Attribute Manager could support Python!
• christoph: Or maybe we could ask Don and Dale for an expression engine with custom Python functions like QGIS offers us... only thinking loud.

Answer 3

You could use a PythonCaller as well to remove your duplicate values:

def processFeature(feature):
 ids = feature.getAttribute('Name_Ids')
 feature.setAttribute('Name_Ids', '|'.join(list(dict.fromkeys(ids.split('|')))))

enter image description here