Blind test finds domain classifier misses many operator-aligned names
in plain words
Some computer tools sort website names into groups based on how they look. A new test checked one such tool against labels made by people without seeing its answers. The tool was often right when it picked a group, but it missed many names that belonged there.
Full summary
Mohd Hashim, founder of DomainGemsAI, has published a blind validation of a classification method used in his earlier CircleID An online publishing platform on internet infrastructure and domain policy, launched in 2002. Full definition of CircleID articles on domain namespace The set of all names in one naming system, such as the DNS. Full definition of namespace. The method sorts registration Obtaining the right to use a domain name for a set period. Full definition of registration into three categories: Operator-aligned, Investor-pattern and Neutral. These categories describe patterns visible in the domain string itself, not verified registrant The person or organization that holds a domain name registration. Full definition of registrant intent, ownership or actual use.
The classifier was frozen and tested against a 340-domain sample drawn from one namespace. The sample was built to cover different predicted categories and naming patterns, along with a random subset, and was designed to examine behavior across name types rather than estimate accuracy for the whole namespace. Reference labels were produced separately, without access to the classifier’s predictions, then reviewed and adjudicated. The labeling was AI-assisted and human-reviewed, not produced by independent domain experts, and the author says blindness to the classifier does not make the labels an independently established ground truth.
For the Investor-pattern category, precision and recall against the reference labels were both above 90%. For Operator-aligned names, precision was 91.5%, meaning most names the classifier placed there agreed with the reference labels, while recall was 58.1%, meaning the classifier identified only about 58% of names the reference review placed in that category.
Some names the classifier put in the Neutral category were assigned a more specific category during reference review. The author says this test does not establish the correct size of that group, but shows that the composition figures in the earlier articles should not be treated as settled. The missed cases were spread across several recurring naming patterns. The author says the current validation does not show which changes would improve performance without adding new errors.
The composition figures in the earlier pieces came from an earlier version of the method. The author says this later validation identified limitations that could also apply to that version, and regards the previously published percentages as provisional.