Power-system state estimation — WLS, observability, bad data, PMU

Reference notes

State estimation (SE) is the algorithm at the heart of every transmission control center. It runs every 30 seconds to 4 minutes in production EMS, turning noisy and incomplete SCADA + PMU measurements into a self-consistent best estimate of bus voltages and angles. Downstream EMS applications — contingency analysis, security-constrained economic dispatch (SCED), market clearing — all consume SE output.

The state vector and measurements

• State vector x: bus voltage magnitudes |V| and phase angles θ at every bus. For N buses with one slack (θ_slack = 0): state dimension n = 2N - 1.
• Measurements z from SCADA: voltage magnitudes, real and reactive power injections, line flows P and Q, current magnitudes.
• PMU measurements: synchronized voltage AND angle phasors at PMU-equipped buses, plus branch currents. Sampled 30-120 Hz via IEEE C37.118.1.
• Measurement model: z_i = h_i(x) + e_i, where e_i ~ N(0, σ_i²) is zero-mean Gaussian noise with variance σ_i².
• Total measurements m typically 2-5× state dimension n. This redundancy is essential for robustness and bad-data detection.

Weighted Least Squares (WLS) — the standard algorithm

• Each measurement weighted INVERSELY by its variance — accurate measurements (PMU, small σ) carry more weight than noisy SCADA.
• h is the measurement function: linear for voltage magnitudes and PMU phasors; nonlinear (products of |V| and trig of θ) for power injections and flows.

Gauss-Newton iteration

1. Flat start: |V| = 1.0, θ = 0 at all buses.
2. Compute residuals: r = z − h(x).
3. Build Jacobian: H = ∂h/∂x.
4. Solve normal equation: (Hᵀ R⁻¹ H) · Δx = Hᵀ R⁻¹ r, where R = diag(σ_i²).
5. Update: x ← x + Δx.
6. If ||Δx|| > tolerance, repeat from step 2.

• Typical convergence: 3-5 iterations.
• Gain matrix G = Hᵀ R⁻¹ H — typically sparse; solved by Cholesky or LU factorization with reordering.
• Modern hardware: 1-5 second solve time for 5,000-bus system; 5-15 seconds for 30,000-bus.

Observability

• SE only solves if the system is OBSERVABLE — measurements collectively span the state vector.
• Mathematical: Jacobian H must have FULL RANK equal to state dimension (2N - 1).
• Practical: every bus must be "reachable" from a reference through measurement equations.
• Topology-based check (Krumpholz-Clements, Monticelli) identifies observable islands BEFORE running WLS.
• Remedies for unobservable regions:
  • Add more measurements (install PMU or SCADA RTU at the unobservable bus).
  • Pseudo-measurements (load forecasts, historical patterns) with high σ (low weight) — last resort.
  • Topology adjustment.
• Typical transmission redundancy ratio η = m/n: 2-5. Above 2 generally observable; above 3 robust to single bad data.

Bad-data detection — χ² test + normalized residual

1. χ² TEST: at the converged solution, J(x̂) follows χ²(m − n) IF all measurements are good. Reject H_0 (no bad data) if J(x̂) > χ²(m − n, 1 − α) critical value (typical α = 0.05 or 0.01).
2. NORMALIZED RESIDUAL: r_N,i = r_i / sqrt(Ω_ii), where Ω = R − H G⁻¹ Hᵀ. Flag |r_N,i| > 3 (three-sigma rule).
3. Identify measurement with LARGEST |r_N,i| as most likely bad data; remove it and re-run SE.
4. Repeat until χ² test passes.

Limitations: single-bad-data identification. Multiple simultaneous or interacting bad data may require more sophisticated algorithms (Monticelli-Garcia, Hyper-Tester).

PMU-enhanced state estimation

• Phasor Measurement Units: GPS-synchronized devices measuring voltage magnitude AND angle (plus branch current phasors) at 30-120 samples/sec.
• Accuracy: < 1% magnitude, < 0.5° angle.
• PMU voltage measurement is LINEAR: h(x) = V at that bus.
• Linear state estimator: if every bus has a PMU, the measurement equations become linear in rectangular voltages (V_real, V_imag). WLS reduces to a single matrix solve — no iteration needed. Solution in milliseconds.
• Hybrid (typical): few PMUs + many SCADA. Combine linear (PMU) and nonlinear (SCADA) equations.
• Accuracy gain: PMU-enhanced SE has 10-100× better angle accuracy than SCADA-only.
• Standards: IEEE C37.118.1 (measurement); C37.118.2 (data transmission); NERC PRC-002 (PMU disturbance recording).
• Suppliers: Schweitzer Engineering Labs (SEL), GE, ABB-Hitachi, Siemens, Arbiter, Macrodyne.

EMS workflow — where SE fits

1. SCADA polls field RTUs every 2-10 seconds. PMU data streams at 30-60 Hz.
2. State estimation runs every 30 seconds to 4 minutes. Outputs estimated state to EMS database.
3. Contingency Analysis (CA): simulates loss of every credible single contingency (N-1 line, transformer, generator) from SE state, checks for overloads / voltage violations. Every 5-30 min.
4. SCED (Security-Constrained Economic Dispatch): solves OPF with N-1 contingency constraints; sets generator dispatch and LMPs (locational marginal prices). Every 5-15 min.
5. Market clearing: RTOs (PJM, MISO, CAISO, ERCOT, NYISO, ISO-NE) clear day-ahead and real-time markets from SCED output.

Alarm processing and topology error detection

• Measurement alarms: SE flags large normalized residuals → operator investigates the suspect RTU or field device.
• Topology Error Detection (TED): if a line is actually open but the topology processor thinks it's closed (or vice versa), SE residuals spike pattern-wise — TED algorithms catch and auto-update the network model.

Robust state estimators (alternatives to WLS)

• WLS strength: statistically efficient (minimum variance) if errors are normally distributed.
• WLS weakness: vulnerable to outliers — a single grossly bad measurement squared in J can heavily bias the estimate.
• LAV (Least Absolute Value): minimize Σ |r_i| / σ_i. Outlier cost grows linearly. Solution via linear programming. 5-10× slower than WLS.
• SHGM (Schweppe-Huber Generalized M): Huber loss — quadratic for small residuals, linear for large. Best of both worlds. 2-3× WLS cost.
• Trade-off: robust estimators tolerate bad data better but are slower, harder to tune. WLS + good bad-data detection remains the industry default for real-time EMS.

Vendors & benchmarks

• EMS vendors: GE iDM (formerly e-terra), Siemens Spectrum Power, ABB-Hitachi Network Manager, OSI Monarch, ETAP RT, plus open-source OpenPDC.
• Benchmark: modern utility SE handles 10,000-30,000 buses with redundancy ratio 3-5, completes in 1-5 seconds, runs every 30-60 seconds, identifies single bad data reliably.