Merging and Normalising

Merge multiple files into one

cat [file1] [file2] | sort -u > [combined]

Normalise filter

cat [combined] | tr '[:upper:]' '[:lower:]' | tr -d '\r' | grep -P '^[a-z0-9][a-z0-9._-]{4,}$' | sort -u > [combined_clean]
Converts uppercase to lowercase
Strips windows carriage returns
Removes noise strings, only: that start with an alphanumeric character, then allow letters, digits, dots, underscores or dashes, and are at least five characters long.