Outliers in Panel Data—How to Define Them? What Is the Relationship Between Winsorizing and Trimming? How to Choose? How to Check for Adverse Effects?

1. Introduction: What Does a Person Earning 50 Million a Year Do to Your Regression Coefficient?

You study the effect of education on individual annual income. Your sample has 10,000 people—9,999 of whom earn between 0 and 1 million per year. But one person earns 50 million.

You run an OLS regression on everyone. The coefficient on education changes from 0.08 to 0.03—one data point changes your core coefficient by 60%. You check—this person earning 50 million happens to have no college education (he is a college dropout who became a wealthy entrepreneur), but his income is extremely high. At a very low value of X, he is pulling the regression line for Y upward.

Is this person an outlier? Should you delete him? Or "compress" his value? How do you judge whether your choice is right or wrong?

Core message: Defining an outlier is not purely a technical issue—it requires judgment based on both data characteristics and domain knowledge. Winsorizing and Truncation/Trimming are the two mainstream methods for handling extreme values—Winsorizing "pulls extreme values within bounds but does not delete them," while Trimming "directly deletes extreme observations." The relationship between the two is: Winsorizing is more conservative (retains observations, retains sample size, but artificially modifies values); Trimming is more aggressive (discards observations, loses information, but does not modify any values). The choice depends on your research objectives, the nature of the outliers (data errors vs. genuine extreme values), and the trade-off you are willing to make between "bias" and "variance." To determine whether the treatment method introduces adverse effects, you need to run a series of sensitivity checks—varying thresholds, comparing coefficients before and after treatment, and confirming that core conclusions do not depend on a specific treatment decision.

2. Defining Outliers—It Is Not Just About "Greater Than 3 Standard Deviations"

2.1 Univariate vs. Multivariate—Outliers Have Two Levels

Univariate Outlier: The value of a single variable deviates extremely from the sample mean. In Stata, for each variable:

sum x, detail

Look at min, max, p1, p99, skewness, and kurtosis. If the ratio of max/p99 is greater than 10 times, or skewness > 2 → extreme values are likely present.

Multivariate Outlier / Regression Outlier: An observation may not be extreme on any single variable, but it is anomalous in the combination of X and Y—that is, it deviates greatly from the regression line (large residual). This needs to be detected using studentized residuals or Cook's Distance (see the residuals article).

Panel data adds one more dimension: outliers in the time dimension.

A firm's R&D expenditure has been between 1 million and 5 million each year for the past 10 years—but in one year it suddenly jumps to 50 million. In the cross-section of the entire sample, it is not extreme (50 million may not be rare in firm R&D data), but relative to its own history, this is a huge jump.

Detection method: Calculate the within-individual rate of change $\Delta X_{it} / X_{i,t-1}$ for each period and check for abnormal jumps.

bysort id (year): gen dx = (x - x[_n-1]) / x[_n-1]
sum dx, detail

If the rate of change for an individual in a given period is 5,000%, it is either a data error or an event requiring special treatment (such as a merger, acquisition, or restructuring).

2.2 Three Sources of Outliers—Judgment Matters More Than Detection

Source 1: Data entry/measurement errors

Income has an extra zero added (originally 50,000, recorded as 500,000). A firm's number of employees is reported as a mix of "in persons" and "in thousands of persons." These outliers should be corrected or deleted—they do not represent any real economic information.

Source 2: Genuine extreme individuals

Apple and Amazon are real in the "U.S. listed companies" sample—they are extreme values, but not errors. Deleting them is equivalent to deleting the two most important companies in the U.S. economy. Genuine extreme individuals contain important economic information and should not be deleted—Winsorizing may be the more appropriate choice.

Source 3: Heterogeneity in the data-generating process

You study "firm export behavior." 90% of firms do not export (export value = 0), and 10% of firms have large export values. The jump between zero and nonzero is not an "outlier"—it is two different data-generating processes (exporters and non-exporters). In this case, "outliers" should not be Winsorized or trimmed—they require a different model (such as the Heckman two-step method or Tobit).

2.3 A Judgment Framework for Panel Data

When judging whether an observation is an outlier in panel data, ask three questions:

1. Is this value reasonable in the cross-section? (Is it extreme among all individuals in the same period?) → Cross-sectional outlier
2. Is this value reasonable in the time series? (Does it suddenly jump across different periods for the same individual?) → Time-series outlier
3. What is the economic meaning of this value? (Data error? Genuine extreme? Structural break?) → Determines how to handle it

3. Winsorizing and Trimming—A Pair of Often-Confused Brothers

3.1 Winsorizing—"Pull Back, but Keep"

Operation: Replace all values below a certain percentile (e.g., 1%) with that percentile value, and replace all values above a certain percentile (e.g., 99%) with that percentile value.

Raw data: [-50, 2, 3, 5, 8, 12, 15, 20, 200, 5000]
After 1%/99% Winsorizing: [-50→2, 2, 3, 5, 8, 12, 15, 20, 200→20, 5000→20]

In Stata:

winsor2 x, replace cuts(1 99)   // two-sided 1%/99% Winsorizing
winsor2 x, replace cuts(1 99) by(groupvar)  // group-wise Winsorizing

Key point: Winsorizing retains all observations—no one is deleted. But values outside the bounds are "compressed" to the boundary values. This artificially reduces the variance of the variable (because extreme values are pulled back to the same value).

3.2 Truncation / Trimming—"Directly Delete"

Operation: Delete all observations below a certain percentile and above a certain percentile.

Raw data: [-50, 2, 3, 5, 8, 12, 15, 20, 200, 5000]
After 1%/99% Trimming: [2, 3, 5, 8, 12, 15, 20]

drop if x < `r(p1)' | x > `r(p99)'

Key point: Trimming discards observations—the sample size decreases. But the retained values are not modified—they are the original true values.

3.3 The Core Difference Between the Two—Not Just "One Modifies, One Deletes"

The relationship between Winsorizing and Trimming: Both "trim" the extreme tails of the data—Winsorizing pushes the exposed parts back in, while Trimming cuts the exposed parts off. Winsorizing is more conservative (sample size unchanged, but values modified), while Trimming is more aggressive (sample size decreases, but values are not modified). Neither is absolutely superior to the other—the choice depends on the nature of the outliers and your research objectives.

4. How to Choose?—Four Judgment Principles

4.1 Principle 1: If It Is a Data Error → Correct or Delete, Do Not Winsorize

Data errors should not be "Winsorized" to boundary values—they should be corrected (if you know the true value) or deleted (if you do not). Winsorizing is for handling genuine but extreme observations—not for masking data errors.

4.2 Principle 2: If It Is a Genuine Extreme Individual, and You Have Reason to Believe They Belong to the Same Data-Generating Process → Winsorize

Apple's market capitalization is an extreme value, but Apple's business logic belongs to the same DGP as other technology companies. You should not delete Apple—it is one of the most valuable companies in the U.S. economy. But you can Winsorize its market capitalization so that its influence is not overly dominant in the regression.

The philosophy of Winsorizing: "I acknowledge the existence of this individual and that its value contains information—but I do not want the information from this one individual to outweigh the information from 9,999 others."

4.3 Principle 3: If Extreme Individuals May Come from a Different Data-Generating Process → Trim + Discuss

If you study "innovation in SMEs," and your sample includes a few giants with annual revenues exceeding 100 billion—they are likely from a completely different DGP (the innovation logic of large firms and small firms is entirely different). In this case, Trimming (deleting these giants) may be more reasonable than Winsorizing (pulling them back to SME boundary values)—because Winsorizing forcibly fits a firm that essentially belongs to another world into the SME pattern.

4.4 Principle 4: Prefer Winsorizing, but Report Trimming Results as a Robustness Check

In empirical economics, Winsorizing (1%/99%) is by far the more common practice—it retains the sample size, reviewers are more familiar with it, and "Winsorizing" signals to readers that you are "handling the problem rather than avoiding it" more than Trimming does. But regardless of which method you choose, you should report the results of the other method in your robustness checks.

5. Thresholds for Winsorizing/Trimming—Choose 1% or 5%?

1%/99% (1st/99th percentile) is the most common threshold in empirical work—it is almost standard practice in finance and accounting research. For most economic variables, this threshold is sufficient to remove genuine extreme noise while retaining the vast majority of observations.

0.5%/99.5% is used when extreme values are fewer but more extreme in magnitude (such as extreme returns in high-frequency trading data).

5%/95% is relatively aggressive—it modifies or deletes 10% of the data. Only consider this when extreme values are densely distributed (such as in certain count data or power-law distributed data).

Selection principle: Look at your data. Before Winsorizing, run sum x, detail—look at p1, p5, p95, p99. If p1 and p5 are not far apart, the tail beyond 1% is not particularly extreme, and 1% is appropriate. If p1 and p5 are vastly different (e.g., p1 = 0, p5 = 1000), the distribution of the first 5% of the data is very sparse—in this case, Winsorizing at 5% may be more reasonable.

6. How to Address Potential Adverse Effects of Winsorizing/Trimming?

6.1 Adverse Effect 1: Artificially Compressed Variance → Inflated Significance

Winsorizing "compresses" multiple different extreme values to the same boundary value → the variance of the variable is artificially reduced → standard errors are underestimated → it becomes easier to obtain "significant" conclusions.

Check method: Compare the variable variance and regression standard errors before and after Winsorizing—if the SE decreases substantially after Winsorizing (by more than 20%–30%) while the coefficient changes little → part of your "significance" may be manufactured by Winsorizing.

6.2 Adverse Effect 2: Threshold Choice Drives Conclusions

Your core coefficient is significant at 1% Winsorizing but not significant at 5% Winsorizing—your conclusion is driven by the choice of threshold.

Check method: Conduct a threshold sensitivity analysis. Winsorize at thresholds from 0.5% to 5% in 0.5% increments, and plot the point estimates and 95% CIs of the core coefficient against the threshold (a "funnel plot"-style sensitivity analysis). If the core coefficient remains roughly stable across the threshold range → the conclusion is not sensitive to the threshold. If the coefficient suddenly becomes insignificant after a certain threshold → your conclusion has been artificially "trimmed" into existence.

6.3 Adverse Effect 3: Winsorizing Control Variables Contaminates the Core Coefficient in Reverse

You Winsorize income—but you forget about the impact on the interaction effects between income and other control variables. Winsorizing changes the covariance structure between income and other variables, which may indirectly affect the estimation of the core coefficient.

Check method: In robustness checks, Winsorize all continuous variables in the model and see whether the core coefficient changes. If the results from Winsorizing only the dependent variable and from Winsorizing all continuous variables are similar → the core coefficient is not sensitive to the Winsorizing approach.

6.4 Adverse Effect 4: Group-Wise Winsorizing in Panel Data May Introduce Between-Group Bias

If you Winsorize firms' R&D expenditure by industry, and some industries have very small samples—in an industry with as few as 10 firms, Winsorizing at 1% means no extreme values in that industry are adjusted at all. "Normal high values" in small industries (which appear extreme only because of the small sample size) may be incorrectly Winsorized, while genuine extreme values in large industries may not be adequately treated.

Check method: Compare the core coefficients from full-sample Winsorizing and group-wise Winsorizing—if the difference is large, group-wise Winsorizing may have introduced between-group bias.

7. A Complete Outlier Treatment and Diagnostic Workflow

* 1. Univariate checks
sum y x1 x2 x3, detail
histogram y, freq
graph box y, over(year)    // panel: box plots by year
 
* 2. Panel time-series jump checks
bysort id (year): gen dx_pct = (x1 - x1[_n-1]) / x1[_n-1]
sum dx_pct, detail
list id year x1 dx_pct if abs(dx_pct) > 3 & !missing(dx_pct)
 
* 3. Regression outlier checks (residuals & influence points)
reg y x1 x2 x3
predict rstud, rstudent
predict cook, cooksd
list id if abs(rstud) > 2 | cook > 4/e(N)
 
* 4. Winsorizing
winsor2 y x1 x2 x3, replace cuts(1 99)
 
* 5. Compare regressions before and after Winsorizing
reg y x1 x2 x3              // raw data (not Winsorized)
estimates store m_raw
reg y x1 x2 x3              // Winsorized data
estimates store m_winsor
estimates table m_raw m_winsor, star
 
* 6. Sensitivity analysis across different thresholds
foreach pct in 0.5 1 2 3 5 {
    * restore raw data first
    winsor2 x1, replace cuts(`pct' 100-`pct')
    reg y x1 x2 x3
    estimates store m_`pct'
}
estimates table m_*, star

Standard reporting in a paper:

• Report both raw and Winsorized means and standard deviations in descriptive statistics tables
• Use Winsorized data for the baseline regression, with a footnote stating "continuous variables are Winsorized at the 1st and 99th percentiles"
• Use raw data or different thresholds (3%/97%, 5%/95%) in robustness checks to confirm that core conclusions remain unchanged

8. Common Misconceptions

8.1 Misconception 1: Applying the Same Winsorizing to All Variables

Winsorizing the dependent variable and the independent variables has different implications—Winsorizing the dependent variable directly changes the distribution of Y and the objective function of the regression; Winsorizing the key explanatory variable changes the variation range of X and the source of identification. The impact of each on your core conclusions may be completely different.

Recommendation: Winsorize the dependent variable and the key explanatory variable; Winsorize control variables selectively (or Winsorize all variables in robustness checks). Clearly report in the paper which variables were Winsorized.

8.2 Misconception 2: Not Checking Changes in Sample Composition After Winsorizing

Winsorizing does not change the sample size (unlike Trimming), but it does change the distribution of the variable—extreme values are compressed to boundary values. If a large number of observations pile up at your boundary values (e.g., at the 1st and 99th percentiles), these observations become identical after Winsorizing—this creates a spurious "peak" in the distribution of a continuous variable.

Check method: Run histogram x, freq before and after Winsorizing. If an unusually tall bar suddenly appears at the boundary value → a large number of observations have been compressed to the same value → the Winsorizing threshold may be set too aggressively.

8.3 Misconception 3: Winsorizing Replaces Analyzing Outliers

Winsorizing should not be an "automated" operation—you should not routinely Winsorize all variables before running any regression without looking at the data. Before Winsorizing, you need to know: Which observations are outliers? Why are they outliers? Do they contain noise or information?

Winsorizing is one tool in the toolbox—not an automatic gatekeeping system. A responsible empirical researcher should flag potential outliers during the data inspection phase, list them (in a footnote or appendix), and then explain why Winsorizing/Trimming is the appropriate treatment.

9. Summary

Five core takeaways for outlier treatment:

1. Defining outliers requires three dimensions: cross-sectional extremity, time-series jumps, and economic meaning. It is not just "greater than 3 standard deviations"—panel data has an additional time dimension to exploit.

2. Winsorizing = pulling back to the boundary without deleting (conservative); Trimming = direct deletion (aggressive). Winsorizing retains the sample size but modifies values; Trimming does not modify values but loses sample size and information. Prefer Winsorizing, but report Trimming in robustness checks.

3. The choice depends on the nature of the outliers: data errors → correct or delete; genuine extreme individuals → Winsorize; possibly from a different DGP → Trim + discuss.

4. Adverse effects must be checked: variance compression, threshold sensitivity, the impact of Winsorizing on covariances, and between-group bias from group-wise Winsorizing. Conduct threshold sensitivity analyses (0.5% → 5%) to confirm that core conclusions do not drift with the threshold.

5. Transparency matters more than precision. Report which variables were treated, what thresholds were used, descriptive statistics comparisons before and after Winsorizing, and baseline results without Winsorizing. Let readers judge for themselves how dependent your conclusions are on the treatment method.

One-sentence conclusion:

"Winsorizing is not cutting off the tail of your data—it is rolling the tail up so it cannot whip the entire regression line. Trimming is directly chopping the tail off. Whether you choose to roll or chop does not depend on which is cleaner—it depends on whether the person on that tail is genuinely anomalous, or a part of the real world you do not want to face. If you do not know—do both, and tell your readers whether your conclusions are consistent under both choices."

10. Presentation Suggestions for Bilibili/WeChat Official Account

• Bilibili video: Suggest using the "tail" as the visual metaphor throughout. Opening scene: a bell-shaped normal distribution curve with a long tail on each side (extreme values). A pair of scissors appears at the left tail—"Trimming: directly cut off." A finger presses down on the right tail—"Winsorizing: roll the tail back and press it against the boundary value." Narration: "Your data has two tails—on the left live people with extremely low incomes, on the right lives the person earning 50 million a year. The question is—should you cut them off, or roll them back?" Act 1 "Who is the outlier": three panels—cross-sectional extremity (a point at the far right of the distribution, isolated), time-series jump (a firm's line suddenly shoots vertically upward in one period), data error (a point outside the scatter plot, labeled "an extra zero?"). Act 2 "Winsorizing vs. Trimming": split-screen animation—left screen "Winsorizing": extreme values are pulled from a point far from the distribution to the 99th percentile boundary line (arrow + highlighted boundary line), labeled "retains observations, modifies values." Right screen "Trimming": extreme values directly fade out and disappear, labeled "deletes observations, does not modify values." Show sample size changes—N unchanged after Winsorizing, N decreases by the labeled proportion after Trimming. Act 3 "How to choose": four scenario cards slide in one by one—"data error → delete" (red card), "genuine extreme → Winsorize" (yellow card), "different DGP → Trim + discuss" (orange card), "don't know → do both" (green card, labeled "robustness check"). Act 4 "Adverse effects check": three "alert dashboards"—variance compression (distribution bulges in the middle after Winsorizing, labeled "SE may be underestimated"), threshold sensitivity (line chart of the core coefficient varying with the threshold, suddenly becoming insignificant after a certain threshold → alarm flashing), group-wise Winsorizing trap (animation of Winsorizing in large vs. small industries, labeled "small industries—normal values may be incorrectly Winsorized").
• WeChat Official Account: The two-column comparison table of Winsorizing vs. Trimming (operation, sample size, value authenticity, variance impact, bias risk, applicable scenarios) is recommended as the core infographic. The three sources of outliers (data errors, genuine extremes, different DGP) with specific examples and judgment flows should be made into three diagnostic cards. The "funnel plot" for threshold sensitivity analysis is recommended as a visual example (x-axis = threshold from 0.5% to 5%, y-axis = core coefficient ± 1.96 SE). The complete diagnostic workflow should be presented as a code block plus flowchart. The four adverse effects (variance compression, threshold-driven conclusions, covariance contamination, group-wise bias) should each be made into a warning card—each card contains "phenomenon → check method → response measure." The Stata code for the panel-specific dimension (time-series jump detection) should be accompanied by a scatter plot with annotations.
• Recommended titles:
  • Main title: "Outliers in Panel Data—How to Define Them? How to Choose Between Winsorizing and Trimming?"
  • Alternative title: "Winsorizing or Trimming?—A Pair of Brothers for Outlier Treatment, One Conservative, One Aggressive"
  • New media title: "A Person Earning 50 Million a Year Ruined Your Regression Coefficient—Should You Delete Him or Squash Him?"
• Key quotes:

  "Winsorizing is rolling the exposed tail back—'I know you are extreme, but I will not let you drag the entire regression line.' Trimming is chopping the tail off—'I do not believe you belong to this data.' Rolling or chopping depends on whether you trust the person on that tail."

  "A person earning 50 million a year appears in your sample—he could be a data entry error (an extra zero), a genuine wealthy individual (a successful college dropout entrepreneur), or he may simply not belong to the type of people you are studying (he is an entrepreneur, and your sample is wage earners). Three possibilities, three treatment methods—none of them is an automatic 'Winsorize at 1% and done.'"

  "Threshold sensitivity analysis is what you must do after Winsorizing—if your core coefficient is just barely significant at 1% Winsorizing and disappears at 2% Winsorizing, then you are not 'handling outliers'—you are using outliers to manufacture significance."

  "Winsorizing changes your data—honestly show the descriptive statistics comparison before and after Winsorizing, so your readers know what your hands did to the data. Transparency matters more than precision."