This document provides comprehensive guidance on checking and validating statistical assumptions for various analyses.
What it means: Each observation is independent; measurements on one subject do not influence measurements on another.
How to check: - Review study design and data collection procedures - For time series: Check autocorrelation (ACF/PACF plots, Durbin-Watson test) - For clustered data: Consider intraclass correlation (ICC)
What to do if violated: - Use mixed-effects models for clustered/hierarchical data - Use time series methods for temporally dependent data - Use generalized estimating equations (GEE) for correlated data
Critical severity: HIGH - violations can severely inflate Type I error
What it means: Data or residuals follow a normal (Gaussian) distribution.
When required: - t-tests (for small samples; robust for n > 30 per group) - ANOVA (for small samples; robust for n > 30 per group) - Linear regression (for residuals) - Some correlation tests (Pearson)
How to check:
Visual methods (primary): - Q-Q (quantile-quantile) plot: Points should fall on diagonal line - Histogram with normal curve overlay - Kernel density plot
Formal tests (secondary):
- Shapiro-Wilk test (good default; scipy handles n up to ~5000 and warns above that)
- Lilliefors test (statsmodels.stats.diagnostic.lilliefors) — use this instead of a plain Kolmogorov-Smirnov test, which is invalid when the mean/SD are estimated from the data
- Anderson-Darling test
Python implementation:
from scipy import stats
import matplotlib.pyplot as plt
# Shapiro-Wilk test
statistic, p_value = stats.shapiro(data)
# Q-Q plot
stats.probplot(data, dist="norm", plot=plt)
Interpretation guidance: - For n < 30: Both visual and formal tests important - For 30 ≤ n < 100: Visual inspection primary, formal tests secondary - For n ≥ 100: Formal tests overly sensitive; rely on visual inspection - Look for severe skewness, outliers, or bimodality
What to do if violated: - Mild violations (slight skewness): Proceed if n > 30 per group (this CLT heuristic assumes mild skewness; severely skewed or heavy-tailed data can require much larger samples) - Moderate violations: Use non-parametric alternatives (Mann-Whitney, Kruskal-Wallis, Wilcoxon) - Severe violations: - Transform data (log, square root, Box-Cox) - Use non-parametric methods - Use robust regression methods - Consider bootstrapping
Critical severity: MEDIUM - parametric tests are often robust to mild violations with adequate sample size
What it means: Variances are equal across groups or across the range of predictors.
When required: - Independent samples t-test - ANOVA - Linear regression (constant variance of residuals)
How to check:
Visual methods (primary): - Box plots by group (for t-test/ANOVA) - Residuals vs. fitted values plot (for regression) - should show random scatter - Scale-location plot (square root of standardized residuals vs. fitted)
Formal tests (secondary): - Levene's test (robust to non-normality) - Bartlett's test (sensitive to non-normality, not recommended) - Brown-Forsythe test (median-based version of Levene's) - Breusch-Pagan test (for regression)
Python implementation:
from scipy import stats
import pingouin as pg
# Levene's test
statistic, p_value = stats.levene(group1, group2, group3)
# For regression
# Breusch-Pagan test
# Note: exog must include the constant column (e.g. exog = sm.add_constant(X),
# or pass the fitted model's model.exog)
from statsmodels.stats.diagnostic import het_breuschpagan
_, p_value, _, _ = het_breuschpagan(residuals, exog)
Interpretation guidance: - Variance ratio (max/min) < 2-3: Generally acceptable - For ANOVA: Test is robust if groups have equal sizes - For regression: Look for funnel patterns in residual plots
What to do if violated: - t-test: Use Welch's t-test (does not assume equal variances) - ANOVA: Use Welch's ANOVA or Brown-Forsythe ANOVA - Regression: - Transform dependent variable (log, square root) - Use weighted least squares (WLS) - Use robust standard errors (HC3) - Use generalized linear models (GLM) with appropriate variance function
Critical severity: MEDIUM - tests can be robust with equal sample sizes
Assumptions: 1. Independence of observations 2. Normality (each group for independent t-test; differences for paired t-test) 3. Homogeneity of variance (independent t-test only)
Diagnostic workflow:
import scipy.stats as stats
import pingouin as pg
# Check normality for each group
stats.shapiro(group1)
stats.shapiro(group2)
# Check homogeneity of variance
stats.levene(group1, group2)
# If assumptions violated:
# Option 1: Welch's t-test (unequal variances)
pg.ttest(group1, group2, correction=True) # correction=True applies Welch's
# (correction='auto' applies Welch only when variances/group sizes are unequal;
# correction=False forces Student's t-test)
# Option 2: Non-parametric alternative
pg.mwu(group1, group2) # Mann-Whitney U
Assumptions: 1. Independence of observations within and between groups 2. Normality in each group 3. Homogeneity of variance across groups
Additional considerations: - For repeated measures ANOVA: Sphericity assumption (Mauchly's test)
Diagnostic workflow:
import pingouin as pg
from scipy import stats
# Check normality per group
for group in df['group'].unique():
data = df[df['group'] == group]['value']
w, p = stats.shapiro(data)
print(f"{group}: W = {w:.3f}, p = {p:.4f}")
# Check homogeneity of variance
print(pg.homoscedasticity(df, dv='value', group='group'))
# For repeated measures: Check sphericity
# Automatically tested in pingouin's rm_anova
What to do if sphericity violated (repeated measures): - Greenhouse-Geisser correction (ε < 0.75) - Huynh-Feldt correction (ε > 0.75) - Use multivariate approach (MANOVA)
Assumptions: 1. Linearity: Relationship between X and Y is linear 2. Independence: Residuals are independent 3. Homoscedasticity: Constant variance of residuals 4. Normality: Residuals are normally distributed 5. No multicollinearity: Predictors are not highly correlated (multiple regression)
Diagnostic workflow:
1. Linearity:
import matplotlib.pyplot as plt
import seaborn as sns
# Scatter plots of Y vs each X
# Residuals vs. fitted values (should be randomly scattered)
plt.scatter(fitted_values, residuals)
plt.axhline(y=0, color='r', linestyle='--')
2. Independence:
from statsmodels.stats.stattools import durbin_watson
# Durbin-Watson test (for time series)
dw_statistic = durbin_watson(residuals)
# Values between 1.5-2.5 suggest independence
3. Homoscedasticity:
# Breusch-Pagan test
# Note: exog must include the constant column (e.g. sm.add_constant(X))
from statsmodels.stats.diagnostic import het_breuschpagan
_, p_value, _, _ = het_breuschpagan(residuals, exog)
# Visual: Scale-location plot
plt.scatter(fitted_values, np.sqrt(np.abs(std_residuals)))
4. Normality of residuals:
# Q-Q plot of residuals
stats.probplot(residuals, dist="norm", plot=plt)
# Shapiro-Wilk test
stats.shapiro(residuals)
5. Multicollinearity:
from statsmodels.stats.outliers_influence import variance_inflation_factor
# Calculate VIF for each predictor
vif_data = pd.DataFrame()
vif_data["feature"] = X.columns
vif_data["VIF"] = [variance_inflation_factor(X.values, i) for i in range(len(X.columns))]
# VIF > 10 indicates severe multicollinearity
# VIF > 5 indicates moderate multicollinearity
What to do if violated: - Non-linearity: Add polynomial terms, use GAM, or transform variables - Heteroscedasticity: Transform Y, use WLS, use robust SE - Non-normal residuals: Transform Y, use robust methods, check for outliers - Multicollinearity: Remove correlated predictors, use PCA, ridge regression
Assumptions: 1. Independence: Observations are independent 2. Linearity: Linear relationship between log-odds and continuous predictors 3. No perfect multicollinearity: Predictors not perfectly correlated 4. Large sample size: At least 10-20 events per predictor
Diagnostic workflow:
1. Linearity of logit:
# Box-Tidwell test: Add interaction with log of continuous predictor
# If interaction is significant, linearity violated
2. Multicollinearity:
# Use VIF as in linear regression
3. Influential observations:
# Cook's distance, DFBetas, leverage (statsmodels >= 0.10)
# Do NOT use OLSInfluence on Logit/GLM results; use get_influence(),
# which returns MLEInfluence (Logit) or GLMInfluence (GLM)
influence = model.get_influence()
cooks_d, cooks_p = influence.cooks_distance # returns a tuple: (distances, p_values)
4. Model fit / calibration:
# Calibration curve: compare predicted probabilities to observed event rates
# (e.g. sklearn.calibration.calibration_curve), plus the Brier score
# Pseudo R-squared
# Classification metrics (accuracy, AUC-ROC)
# Note: the Hosmer-Lemeshow test is not implemented in scipy/statsmodels
# and is sensitive to the choice of bins; prefer calibration curves
Methods: 1. Visual: Box plots, scatter plots 2. Statistical: - Z-scores: |z| > 3 suggests outlier - IQR method: Values < Q1 - 1.5×IQR or > Q3 + 1.5×IQR - Modified Z-score using median absolute deviation (robust to outliers)
For regression: - Leverage: High leverage points (hat values) - Influence: Cook's distance > 4/n suggests influential point - Outliers: Studentized residuals > ±3
What to do: 1. Investigate data entry errors 2. Consider if outliers are valid observations 3. Report sensitivity analysis (results with and without outliers) 4. Use robust methods if outliers are legitimate
For small samples: - Assumptions become more critical - Use exact tests when available (Fisher's exact, exact logistic regression) - Consider non-parametric alternatives - Use permutation tests or bootstrap methods - Be conservative with interpretation
When reporting analyses, include:
Example reporting statement:
"Normality was assessed using Shapiro-Wilk tests and Q-Q plots. Data for Group A (W = 0.97, p = .18) and Group B (W = 0.96, p = .12) showed no significant departure from normality. Homogeneity of variance was assessed using Levene's test, which was non-significant (F(1, 58) = 1.23, p = .27), indicating equal variances across groups. Therefore, assumptions for the independent samples t-test were satisfied."