Convergence Modes and Stochastic Orders
Y. Eddie Lu, Summer 2026
ECON 8002 course index · Lecture 5 of 9
Orientation
Four ways a random sequence can approach a limit
How to use this page
Never write \(X_n\to X\) without its mode. First ask: are we comparing paths, error probabilities, expected losses, or probability laws?
How should I think about Lecture 5?
a.s.: almost every path settles.
\(L^r\): expected loss vanishes.
\(\longrightarrow\)
Convergence chooses what kind of approximation an estimator has achieved.
\(\longrightarrow\)
in probability: large errors become rare.
in distribution: laws become close.
Course map
- Lectures 1–4: probability and conditioning
- Lecture 5: convergence modes and rates (current lecture)
- Lecture 6: transformations
- Lecture 7: laws of large numbers
- Lectures 8–9: weak limits, central limit theorems, and inference
Prerequisite retrieval
Know norms of random vectors, c.d.f.s, Markov’s inequality, subsequences, and \(E|X|^r<\infty\). For a.s., probability, and \(L^r\) convergence, variables live on a common probability space.
Four convergence modes
Concrete opening
For an estimator \(\hat\theta_n\), these claims differ:
\[\hat\theta_n\xrightarrow{p}\theta_0,\qquad \sqrt n(\hat\theta_n-\theta_0)\xrightarrow{d}N(0,V).\]
The first is consistency. The second describes remaining sampling noise at a finer scale.
Almost-sure convergence
\[X_n\xrightarrow{\mathrm{a.s.}}X \iff P\{\omega:\lim_{n\to\infty}X_n(\omega)=X(\omega)\}=1.\]
The same sample path must eventually settle, except on one null set.
Retrieval check.
Does a.s. convergence mean \(X_n(\omega)\to X(\omega)\) for every \(\omega\)?
No. Failure is allowed on a set of probability zero.
Convergence in probability
\[X_n\xrightarrow{p}X \iff \forall\varepsilon>0,\ P(\|X_n-X\|>\varepsilon)\to0.\]
It controls the chance of a fixed-size error, not its expected size.
\(L^r\) convergence
For \(r>0\),
\[X_n\xrightarrow{L^r}X\iff E\|X_n-X\|^r\to0.\]
This is stronger because it penalizes the magnitude of errors as well as their probability.
Convergence in distribution
\[X_n\xrightarrow{d}X\iff F_{X_n}(x)\to F_X(x)\]
at every continuity point \(x\) of \(F_X\). It compares laws; the variables need not share a probability space.
Retrieval check.
Why are discontinuity points of \(F_X\) excluded?
Weak convergence need not force c.d.f. convergence there; point masses create jumps.
Implications and separating examples
One map, not four synonyms
\[ \begin{aligned} X_n\xrightarrow{\mathrm{a.s.}}X&\searrow\\ &X_n\xrightarrow{p}X\Longrightarrow X_n\xrightarrow{d}X,\\ X_n\xrightarrow{L^r}X&\nearrow \end{aligned} \]
None of the displayed implications reverses in general.
Proof: \(L^r\) implies probability
Markov’s inequality gives, for every \(\varepsilon>0\),
\[P(\|X_n-X\|>\varepsilon) \le \varepsilon^{-r}E\|X_n-X\|^r\to0.\]
A.s. implies probability
The tail-supremum characterization is
\[X_n\to X\ \mathrm{a.s.}\iff P\left(\sup_{k\ge n}\|X_k-X\|>\varepsilon\right)\to0\quad\forall\varepsilon>0.\]
Since \(\{\|X_n-X\|>\varepsilon\}\) lies inside that tail event, the probability conclusion follows.
Retrieval check.
Which event is larger: \(\{\|X_n-X\|>\varepsilon\}\) or \(\{\sup_{k\ge n}\|X_k-X\|>\varepsilon\}\)?
The tail-supremum event is larger.
Probability implies distribution
Use the Portmanteau theorem or bounded continuous test functions. A small input error makes \(g(X_n)-g(X)\) small in probability for bounded uniformly continuous \(g\), hence \(E[g(X_n)]-E[g(X)]\to0\).
Nonexample: rare, enormous errors
Let \(U\sim\mathrm{Uniform}(0,1)\) and \(X_n=n\mathbf1_{\{U\le1/n\}}\). Then
\[X_n\xrightarrow{p}0,\qquad E[X_n]=1.\]
Thus convergence in probability need not imply \(L^1\) convergence or convergence of means.
Retrieval check.
What feature prevents \(L^1\) convergence in the moving-spike example?
The vanishing probability is exactly offset by an exploding magnitude.
Nonexample: probability without a.s.
For independent \(A_n\) with \(P(A_n)=1/n\), \(\mathbf1_{A_n}\xrightarrow{p}0\) but not almost surely: Borel–Cantelli gives infinitely many spikes almost surely.
Distribution without probability
Let \(X,X_1,X_2,\ldots\) be independent \(N(0,1)\) variables. Then \(X_n\xrightarrow{d}X\) because their laws coincide, but \(X_n-X\) does not converge in probability to zero.
A constant limit is special
\[X_n\xrightarrow{d}c\quad\Longleftrightarrow\quad X_n\xrightarrow{p}c.\]
The equivalence fails for a nondegenerate random limit. This fact will turn a Slutsky conclusion into an LLN conclusion in Lecture 7.
Retrieval check.
Why does \(X_n\xrightarrow{d}X\) not generally imply \(X_n\xrightarrow{p}X\)?
Distribution ignores how \(X_n\) and \(X\) are coupled on the probability space.
Supporting convergence results
Pólya’s theorem
If \(F_n\) and \(F\) are c.d.f.s on \(\mathbb R^k\), \(F_n(x)\to F(x)\) at every \(x\in\mathbb R^k\), and \(F\) is continuous, then
\[\sup_x|F_n(x)-F(x)|\to0.\]
Continuity of the limit upgrades pointwise c.d.f. convergence to uniform c.d.f. convergence.
Skorokhod representation: a warning label
\(X_n\xrightarrow{d}X\) permits construction of new variables \(Y_n,Y\) with the same respective laws and \(Y_n\to Y\) a.s. It does not prove a.s. convergence of the original \(X_n\).
Borel–Cantelli sufficient condition
If for every \(\varepsilon>0\),
\[\sum_{n=1}^{\infty}P(\|X_n-X\|>\varepsilon)<\infty,\]
then \(X_n\to X\) a.s. No independence is required for this direction.
Retrieval check.
Does \(X_n\xrightarrow{p}X\) alone make the displayed probability series summable?
No. Terms can tend to zero while their series diverges.
Subsequence principle
\[X_n\xrightarrow{p}X\iff\text{every subsequence has a further subsequence converging a.s. to }X.\]
Construction: choose \(n_k\) with \(P(\|X_{n_k}-X\|>2^{-k})<2^{-k}\), then apply Borel–Cantelli.
Weak convergence and moments
Weak convergence alone cannot control unbounded functions such as \(x\mapsto |x|^r\). Uniform integrability supplies the missing tail control:
\[\lim_{K\to\infty}\sup_nE[|X_n|^r\mathbf1_{\{|X_n|>K\}}]=0.\]
Uniform-integrability theorem
For \(r>0\), if \(X_n\xrightarrow{d}X\), then
\[E|X_n|^r\to E|X|^r<\infty\]
iff \(\{|X_n|^r\}\) is uniformly integrable. Treat this as a tail theorem, not a mechanical add-on.
Retrieval check.
Does \(X_n\xrightarrow{d}0\) imply \(E[X_n^2]\to0\)?
No. A rare-spike sequence can violate it; uniform integrability is needed.
Deterministic and stochastic orders
Deterministic orders
For nonrandom sequences with \(b_n>0\) eventually, \(a_n=O(b_n)\) if \(|a_n|\le Cb_n\) eventually, and \(a_n=o(b_n)\) if \(a_n/b_n\to0\).
Stochastic orders
For deterministic \(a_n>0\) for every \(n\),
\[X_n=O_p(a_n)\iff\forall\varepsilon>0\ \exists C_\varepsilon<\infty: \sup_nP(\|X_n\|>C_\varepsilon a_n)<\varepsilon,\]
\[X_n=o_p(a_n)\iff X_n/a_n\xrightarrow{p}0.\]
Almost-sure orders and random scales
For deterministic \(a_n>0\) eventually,
\[X_n=o_{\mathrm{a.s.}}(a_n)\iff X_n/a_n\to0\quad\mathrm{a.s.},\]
\[X_n=O_{\mathrm{a.s.}}(a_n)\iff P\!\left(\limsup_n\|X_n/a_n\|<\infty\right)=1.\]
For a random scale satisfying \(P(Y_n=0)=0\) for every relevant \(n\), write \(X_n=O_p(Y_n)\) when \(X_n/Y_n=O_p(1)\), and similarly for \(o_p\).
What the symbols say
\(O_p(a_n)\) means “contained at scale \(a_n\) with high probability.” \(o_p(a_n)\) means “negligible relative to \(a_n\).”
\[\bar X_n-\mu=O_p(n^{-1/2})\quad\text{does not mean}\quad o_p(n^{-1/2}).\]
Retrieval check.
Under a nondegenerate CLT, classify \(\sqrt n(\bar X_n-\mu)\).
It is \(O_p(1)\), but generally not \(o_p(1)\).
Order algebra
\[ o_p(a_n)\implies O_p(a_n), \]
and
\[ X_n\xrightarrow{d}X\implies X_n=O_p(1). \]
If \(X_n=O_p(a_n)\) and \(a_n=O(b_n)\), then \(X_n=O_p(b_n)\).
For compatible scalar, vector, or matrix products,
\[O_p(1)o_p(1)=o_p(1),\]
and random-scale transitivity gives
\[X_n=O_p(Y_n),\quad Y_n=O_p(Z_n)\implies X_n=O_p(Z_n).\]
Proof of the product rule
Let \(Z_n=O_p(1)\) and \(R_n=o_p(1)\). Fix \(\varepsilon,\eta>0\). Choose \(M<\infty\) so \(P(\|Z_n\|>M)<\eta/2\) eventually. Then
\[ P(\|Z_nR_n\|>\varepsilon) \le P(\|Z_n\|>M)+P(\|R_n\|>\varepsilon/M)<\eta \]
eventually. Hence \(Z_nR_n=o_p(1)\).
Synthesis and retrieval
Assumption audit
| conclusion | key assumption |
|---|---|
| \(L^r\implies p\) | finite \(r\)-loss and Markov |
| \(p\implies d\) | common-space closeness / Portmanteau |
| c.d.f. test | continuity points of limit c.d.f. |
| moments converge | uniform integrability |
| \(X_n=O_p(1)\) from weak convergence | tightness of a weakly convergent sequence |
Common errors
- Never omit the convergence mode.
- Do not reverse implication arrows.
- \(O_p(1)\) does not mean convergence to zero.
- Weak convergence does not imply moment convergence.
- Skorokhod changes the coupling.
Theorem-selection blueprint
- Need a path claim? Try a.s. or Borel–Cantelli.
- Need consistency? Prove convergence in probability.
- Need expected loss? Prove \(L^r\) or add UI.
- Need an asymptotic law? Prove convergence in distribution.
- Need a remainder? state its scale with \(O_p\) or \(o_p\).
Exit ticket 1
Give a sequence that converges in probability but not in \(L^1\).
Exit ticket 2
State the implication map and one non-reversible arrow.
Exit ticket 3
Explain in one sentence why \(X_n\xrightarrow{d}c\) implies \(X_n\xrightarrow{p}c\).
Mastery checklist
- I can define all four convergence modes.
- I can prove \(L^r\implies p\) and explain \(p\implies d\).
- I can construct failures of reverse implications.
- I can state the role of uniform integrability.
- I can interpret \(O_p\) and \(o_p\) at an estimator scale.