Combining regex

Question

How can this be minimized?

// remove accent
byte[] bytes = System.Text.Encoding.UTF8.GetBytes(input);
input = System.Text.Encoding.UTF8.GetString(bytes);
// make it all lower case
input = input.ToLower();
// remove stop words
input = System.Text.RegularExpressions.Regex.Replace(input, "\\b" + string.Join("\\b|\\b", ENGLISH_STOP_WORDS) + "\\b", "");
// remove entities
input = System.Text.RegularExpressions.Regex.Replace(input, @"&\w+;", "");
// remove anything that is not letters, numbers, dash, or space
input = System.Text.RegularExpressions.Regex.Replace(input, @"[^a-z0-9\-\s]", "");
// replace spaces
input = input.Replace(' ', '-');
// collapse dashes
input = System.Text.RegularExpressions.Regex.Replace(input, @"-{2,}", "-");
// collapse spaces
input = System.Text.RegularExpressions.Regex.Replace(input, @"\s+", " ").Trim();
// Trim dashes and spaces
input = input.Trim(' ').Trim('-').Trim(' '); // double trim the spaces incase dashes were covering them
return input;

• Quentin Pradet: What is the final intent of this snippet?
• Rafael Herscovici: i think its quite obvious, including the comments... it should generate a UrlFriendly Slug.

Answer 1

// remove accent

Actually, no. The following code is just a lossless conversion to and from UTF-8 which doesn’t change the text.

In the following, I’d coalesce the regular expressions – if nothing else, this is way more efficient. I’d also import the namespace to get rid of this overlong explicit namespace qualification. The "collapse spaces" phase makes no sense since you’ve already removed spaces.

Finally, you can also coalesce the Trim statements.

Ignoring for now that the accent removal doesn’t work, this leaves us with:

input = input.ToLower();
// remove stop words, entities and anything that is not letters, numbers, dash, or space
string stopWords = string.Format("\\b{0}\\b", string.Join("\\b|\\b", ENGLISH_STOP_WORDS));
input = Regex.Replace(input, stopWords + @"|&\w+;|[^a-z0-9\-\s]", "");
// replace spaces
input = input.Replace(' ', '-');
// collapse dashes
input = Regex.Replace(input, @"-{2,}", "-");
// Trim dashes and spaces
input = input.Trim(' ', '-');

Finally, to remove accents, you need to normalize the Unicode string so that accented characters are decomposed into diacritics and remove combining diacritic marks:

static string RemoveDiacritics(string stIn) {
 string stFormD = stIn.Normalize(NormalizationForm.FormD);
 StringBuilder sb = new StringBuilder();
 for(int ich = 0; ich < stFormD.Length; ich++) {
 UnicodeCategory uc = CharUnicodeInfo.GetUnicodeCategory(stFormD[ich]);
 if(uc != UnicodeCategory.NonSpacingMark) {
 sb.Append(stFormD[ich]);
 }
 }
 return sb.ToString();
}

• Rafael Herscovici: nice, tested and works like a charm, interesting to see what others come up with.
• Jeff Vanzella: Why not make the regex a compiled regex: Regex theRegex = new Regex(stopWords + @"|&\w+;|[^a-z0-9\-\s]", RegexOptions.Compiled); then theRegex.Replace(...) This should make the calls a little quicker, and use less resources.
• Konrad Rudolph: @Jeff True, it’s potentially very useful. But this should never be the default option because it creates a memory leak: there is no way of unloading the compiled regex from memory until the end of the application domain. As such, only unchanging, frequently used regexes should be compiled. But this is probably the case here anyway.