数据范儿_范老师-AI时代科研统计新范式-计量科研AI导航

I. Introduction: Two Kinds of "Significance," Two Entirely Different Worlds

If you have ever sat in on an economics seminar, you have surely seen this scene—

The presenter shows regression results, with three stars (***) next to the coefficient in parentheses. A professor in the audience raises a hand: "This coefficient is statistically significant, but at a magnitude of 0.002—is it economically significant? An extra year of schooling raises wages by 0.2%. Do you think anyone would attend university for such a small increase?"

In another seminar, the presenter's key coefficient is 0.45, which looks sizable, but the standard error in parentheses is 0.38. Again someone in the audience raises a hand: "The direction of this effect is interesting, but with a p-value of 0.24—you do not have enough statistical power to distinguish this 0.45 from zero. All you can say is that your data do not yet allow you to be confident that this effect exists."

These two scenes point to the same core issue: statistical significance and economic significance are two entirely different dimensions of judgment. Confuse them, and you will mistake noise for discovery, or treat a real signal as a null result. Most introductory econometrics textbooks teach you the t-test and p-values, but they do not systematically tell you: after p < 0.05, what is the second question you should ask?

Key message: Statistical significance answers "is this effect truly nonzero (rather than an artifact of sampling error)?" Economic significance answers "does this effect matter in the real world?" There is no necessary correspondence between the two—large samples can make a trivially small effect statistically significant, and small samples can make a large effect statistically insignificant.

II. Statistical Significance: An Over-Worshipped Threshold

2.1 What Does Statistical Significance Actually Say?

In a regression output, β̂ = 0.08, standard error = 0.02, p = 0.0001.

The precise meaning of the p-value: If the true β = 0 (i.e., education has no effect on wages at all), then with our sample size, the probability of observing |β̂| ≥ 0.08 purely due to sampling error is 0.0001, or one in ten thousand.

Because this probability is extremely low, we reject the null hypothesis that "β = 0" and say "this effect is statistically significant."

2.2 Three Factors That Affect Statistical Significance

$t = \frac{\hat{\beta}}{\text{se}(\hat{\beta})}$

The t-statistic depends on two things: the magnitude of the coefficient itself, $\hat{\beta}$ , and the size of the standard error.

The standard error in turn depends mainly on:

Sample size n: n ↑ → se ↓ → t ↑ → p ↓ (holding other things constant). This is why in large samples almost any variable can become significant—the standard error converges to zero, and even if $\hat{\beta}$ itself is negligibly small, the t-statistic will inflate.
The degree of variation in X: $\text{se}(\hat{\beta}) \propto \frac{1}{\sqrt{\sum (X_i - \bar{X})^2}}$ . The more variation in X → the smaller the standard error → the easier it is to achieve significance. If X has almost no variation, even a large β will struggle to be significant.
The noise level of the residual, $\sigma^2$ : The smaller the noise → the smaller the standard error → the easier it is to achieve significance.

Core lesson: Statistical significance does not directly tell you how large the effect is. In large samples, even $\hat{\beta} = 0.0003$ can achieve three stars—because the standard error has already shrunk to 0.00005. Statistical significance only tells you that "the data are consistent with the direction of the effect you observe," but it never promises that "the magnitude of the effect in that direction is worth caring about."

III. Economic Significance: Does the Effect Matter in the Real World?

3.1 Economic Significance Is a Matter of Human Judgment

Statistical significance has a universally recognized yardstick: p < 0.05. Although this 0.05 threshold is itself a convention (casually proposed by Fisher in 1925), at least it is a yardstick.

Economic significance has no such unified threshold. To judge economic significance, you need to answer the question:

"Is the magnitude of this coefficient—placed in the real world—'large enough to be worth caring about'?"

And the criteria depend entirely on the research context, the meaning of the variables, the baseline level, the policy implications, and the shared understanding of readers and referees.

3.2 Four Common Methods for Judging Economic Significance

Method 1: Look at the magnitude of the coefficient relative to the typical level of the dependent variable.

You estimate that "an additional year of schooling raises wages by 8%." With an average annual wage baseline of 50,000 yuan, 8% = 4,000 yuan. One more year of schooling, earning 4,000 yuan more per year—this effect is economically significant.

If you estimate that "an additional year of schooling raises wages by 0.3%." At the same baseline, 0.3% = 150 yuan. One more year of schooling earns only 150 yuan more per year—so why would anyone spend four years on a bachelor's degree? This effect is hard to call economically significant, even if its p-value may be less than 0.001.

Core operation: Whenever you report a percentage or a log value, also give the "implication for a typical individual." For example: $\hat{\beta} = 0.08$ means that for an individual at the median annual income of $50,000, an additional year of education corresponds to an increase in annual income of$ 4,000. This number is easy to read and easy to judge.

Method 2: Benchmark against other known effects.

You estimate that "a new teaching method improves test scores by 0.05 standard deviations." Is this effect large?

You need a frame of reference. In the education intervention literature, a typical practice is to compare with existing classic studies—for example, the Tennessee STAR experiment found that small class sizes improved achievement by about 0.2 standard deviations. 0.05 is roughly one-quarter of that classic effect—on the "education intervention" scale, this is a small but nonzero effect.

Method 3: Look at the cumulative impact of policy implications.

A β₁ = 0.02 may seem small. But if X is the "tax rate," 0.02 means "a 1 percentage point reduction in the tax rate increases investment by 2%." If a country's corporate income tax rate falls from 25% to 20% (a 5 percentage point reduction), that implies investment grows by about 10%—which at the national aggregate level could amount to hundreds of billions. A single coefficient may look small, but when amplified at the scale of policy action, the impact can be very large.

Method 4: Compare against the implicit costs of real-world behavior.

You estimate that "participating in vocational training increases the probability of employment by 2 percentage points." Is this 0.02 worth spending 5,000 yuan per person on training? This requires a cost-benefit analysis—is spending 5,000 yuan per person to raise employment by 2% better than spending that money on other interventions? This kind of judgment goes beyond econometrics itself and enters the domain of policy analysis.

3.3 A Formulaic Summary of the Relationship Between the Two

From the perspective of confidence intervals—

Statistical significance cares about: Does the confidence interval contain zero? If not → statistically significant.

Economic significance cares about: Is the magnitude of the effect represented by the lower and upper bounds of the confidence interval economically meaningful?

Here is a key distinction: the lower bound of the confidence interval is sufficiently large (does not contain zero) → statistically significant. The entire range of the confidence interval consists of economically meaningful magnitudes → economically significant.

IV. Four "Statistical × Economic" Combinations—Which One Is Your Regression Result?

	Statistically Significant	Statistically Insignificant
Economically Significant	✅ Ideal: You are confident both that the effect exists and that it is large enough to care about.	⚠️ Worth reporting, but underpowered: The effect is in the right direction and not small, but you cannot rule out sampling error. More data are needed.
Economically Insignificant	⚠️ "Over-sensitivity" in large samples: The effect truly exists, but it is too small to care about.	❌ The effect may not exist, and even if it does, it is too small to care about.

Combination 1: Statistically Significant + Economically Significant → Ideal

β̂ = 0.08, se = 0.02 → 95% CI = [0.04, 0.12]. The lower bound also represents a 4% increase—this is significant both statistically and economically.

Combination 2: Statistically Significant + Economically Insignificant → "Over-sensitivity" in Large Samples

β̂ = 0.002, se = 0.0003 → p < 0.001, 95% CI = [0.0014, 0.0026].

The effect truly exists; there is no doubt about that. But if this is the return to education—0.2% means one more year of schooling earns only 0.2% more. With an average annual income of 50,000, one more year of schooling earns 100 yuan more—investing four years of time and tuition, the return is almost negligible.

This kind of result is the truth detected by an "over-sensitive" large sample—but the truth itself is so small that it does not merit policy attention.

Combination 3: Statistically Insignificant + Economically Significant → Directional Evidence Worth Reporting

β̂ = 0.50, se = 0.38 → p = 0.19, 95% CI = [-0.26, 1.26].

The confidence interval is wide and includes zero. But it also includes 0.50 and 1.00—both economically sizable effects. Your problem is not that "the effect does not exist," but rather "the effect exists, but your data do not have enough power to confirm it. The sample size is insufficient or the variation in X is inadequate."

In this case, you cannot say "we found an effect of 0.50"—your data do not permit you to say that, because you cannot rule out that this number arose from random fluctuation. But you can discuss: based on the current data, the best estimate of the effect is 0.50, and the standard error reflects substantial uncertainty. If an effect of this magnitude truly exists, it would have significant policy implications. Future research with larger samples or data with more variation in X would help reduce the standard error.

Combination 4: Neither Significant → Abandon or Redesign

The effect may not exist, and even if it does, it is too small. In this case, unless the "null finding" itself is meaningful (e.g., finding that the overall effect of the minimum wage on employment is close to zero, which is valuable information in policy debates), it is difficult to extract useful conclusions from such a result.

V. If Statistically Insignificant, Can You Still Discuss Economic Significance?

This is the most delicate question in this article. The answer is: Yes, you can—but with strict boundary conditions.

5.1 Cases Where You Can Discuss It

Case 1: The confidence interval is narrow, does not contain zero, but straddles both sides of the "economically meaningful" threshold.

Sometimes your 95% confidence interval has a lower bound < 0 < upper bound (statistically insignificant), but the lower bound is also not large enough to be of "decisive magnitude" economically. You can say: "Our data provide ambiguous evidence. The best estimate is β̂ = 0.35. We cannot rule out zero, nor can we rule out effects above 0.50. A larger sample is needed to obtain a more precise estimate."

Case 2: The purpose of the study is an initial exploration of the direction of a policy effect.

If your study is the "first empirical paper" in a new field, and the goal is not to provide a "quantitative conclusion" but to determine whether "the direction and magnitude are large enough to justify investing resources in further research"—then even if the p-value is between 0.10 and 0.20, discussing the "magnitude of suggestive evidence" is valuable. However, you must use cautious language such as "suggestive" or "directional," rather than confirmatory language like "this study finds..."

5.2 Cases Where You Cannot Discuss It

❌ Statistically insignificant, but you report "β̂ = 0.45" as a definitive conclusion ("we found an effect of 0.45").
❌ Statistically insignificant, but you selectively report "economic significance" while deliberately omitting the fact of statistical insignificance.
❌ Statistically insignificant, and the direction of the effect is inconsistent with your theoretical prediction (the sign is reversed), yet you still claim "the data support our hypothesis."

5.3 The Golden Rule

When statistically insignificant, your standard wording framework:

"The point estimate is [X], with an economic meaning of [Y]. However, we cannot reject the null hypothesis at conventional significance levels (p = [Z]). [Possible reasons: small sample size/n, insufficient variation in X, large residual noise]. The uncertainty in this estimator means we cannot currently draw definitive conclusions. Future research with larger samples or greater variation in X would enable more precise estimation."

VI. A Practical Framework: When You Face a Regression Coefficient

Whether you are a reader or an author, when you see a regression coefficient, think through the following sequence:

(1) Look at the magnitude

Do not look at the stars first; look at the coefficient itself. How large is β̂? On the usual scale between -1 and 1, what is its "objective size"? Multiply it by a typical range of variation in X (e.g., education from one standard deviation to two, or a policy change from the current 25% to 20%) to obtain a predicted change in Y. Relative to the typical magnitude of Y, is this change large or small?

(2) Look at the confidence interval, not the p-value

p < 0.05 tells you very little. But a 95% CI of [0.04, 0.12] tells you two things: the effect is almost certainly between 4% and 12%, and the direction is certain. In contrast, a 95% CI of [-0.01, 0.17] tells you: the effect could be positive, negative, or zero—you cannot be certain of anything.

(3) Look at economic significance

Building on (1) and (2), judge whether the magnitude of the effect—at both the lower and upper bounds of the confidence interval—is economically meaningful. If even the lower bound represents an "economically meaningful" magnitude, then you have evidence that is both economically and statistically significant. If the upper bound is "meaningful" but the lower bound is not, you have mixed evidence—the direction is right, but the magnitude is uncertain.

(4) Check comparability with known benchmarks

Is your effect in the same ballpark as commonly estimated magnitudes for "the same pair of variables" in the existing literature? If the return to education in the literature is typically 0.07–0.10, and you obtain 0.42—then the question you should be asking is "is there an identification problem?" rather than "have I made a major discovery?"

VII. Summary: Four Principles

Statistical significance ≠ importance. It only tells you whether "the noise is small enough relative to the effect." Large samples can make any tiny effect statistically significant.
Statistical insignificance ≠ zero effect. It only tells you that "the data are not precise enough to rule out zero." Perhaps the effect is zero, or perhaps the effect is large but your sample is too small or the variation in X is insufficient.
Economic significance requires human judgment. There is no standard threshold. Judgment depends on baseline comparisons, references to the existing literature, the policy implications of cumulative effects, and cost-benefit thinking.
When statistically insignificant, you can still discuss economic significance—but with an explicit caveat of "uncertainty." "The best estimate is in the expected direction, the magnitude may have policy implications, but at the current level of data precision, no definitive conclusions can be drawn."

One-sentence conclusion:

"Statistical significance is your tool, not your master. After p < 0.05, the real work has just begun—you need to ask yourself: does this effect, in the real world, deserve to be cared about?"

VIII. Presentation Suggestions for Bilibili/WeChat Official Account

Bilibili video: For the four combinations, use a 2×2 matrix chart—statistical significance on the horizontal axis, economic significance on the vertical axis, with the four quadrants filled with four different colored scenarios. Pair with four concrete stories (tiny effects in large samples, ambiguous large effects in small samples, etc.) so viewers can identify their own situation.
WeChat article: The four combinations are well suited to a table infographic. For the comparison between confidence intervals "containing zero vs. not containing zero" and economic meaning, use an "interval comparison chart"—draw three intervals, placing them across the four scenarios, for a visual display.
Recommended titles:
- Main title: 《Statistical Significance ≠ Importance: What Exactly Is Economic Significance?》
- Alternative title: 《p < 0.01 but Coefficient 0.002—Is This a Discovery? Four Combinations of Statistical and Economic Significance》
Key quote:

"Statistical significance tells you whether an effect is a coincidence; economic significance tells you whether an effect is worth caring about. Large samples can make any tiny effect statistically significant—but they can never make a tiny effect important."