Source Guide: Orthogonality and Least Squares

A curated source guide for learning orthogonality and least squares from first pass through research bridge.

Keywords

source guide, least squares, orthogonality

1 How To Use This Guide

Use exactly one source from each layer at a time:

• First pass for intuition and basic fluency
• Second pass for proofs, exercises, and numerics
• Paper bridge for reading current research

Do not try to read everything in parallel.

2 Stable Core

• MIT 18.06SC Linear Algebra resource index - First pass - best official structured path if you want lectures, problems, and projection-focused least-squares coverage. Checked 2026-04-24.
• Stanford Math 51 - First pass - strong current course framing for applied linear algebra with regression and data-facing motivation. Checked 2026-04-24.
• Hefferon, Linear Algebra - Second pass - especially strong for motivated proofs and self-study exercises. Checked 2026-04-24.
• MIT 2.086 Unit 3: Matrices and Least Squares; Regression from Math, Numerics, and Programming - Second pass - excellent when you want regression, numerics, and programming concerns in one place. Checked 2026-04-24.

3 Current Bridge

• Randomized Numerical Linear Algebra: Foundations and Algorithms - Second pass - modern survey context for how least squares fits into sketching and large-scale numerical methods. Checked 2026-04-24.
• The Implicit Bias of Benign Overfitting - Paper bridge - current perspective on minimum-norm interpolation and high-dimensional regression. Checked 2026-04-24.
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression - Paper bridge - current view linking least squares to optimization dynamics. Checked 2026-04-24.
• Distributed Least Squares in Small Space via Sketching and Bias Reduction - Paper bridge - useful current example when the systems side matters. Checked 2026-04-24.

4 Paper Bridge

• A Statistical Perspective on Randomized Sketching for Ordinary Least-Squares - Paper bridge - best first bridge paper for this topic package because it keeps the core least-squares geometry visible. Checked 2026-04-24.
• Iterative Hessian Sketch: Fast and Accurate Solution Approximation for Constrained Least-Squares - Paper bridge - stronger optimization-facing follow-up once the first sketching paper is comfortable. Checked 2026-04-24.

5 Recommended Reading Paths

5.1 Path 1: First serious understanding

1. MIT 18.06SC resource index
2. Orthogonality and Least Squares
3. Hefferon for more exercises

5.2 Path 2: Regression and computation

1. Orthogonality and Least Squares
2. MIT 2.086 Unit 3 notes
3. Computation Lab: Projection Geometry and Regression Residuals
4. Linear Regression Through Projection

5.3 Path 3: Bridge to research

1. Orthogonality and Least Squares
2. Projection Theorem and Normal Equations
3. Paper Lab: Randomized Sketching for Least Squares
4. Research Direction: Least Squares, Sketching, and Overparameterized Regression
5. Acta Numerica survey or current JMLR paper

6 Sources Checked Online

All links above were checked on 2026-04-24.

The online verification pass included:

• official course pages
• official author-maintained textbook pages
• official journal or proceedings pages
• official conference proceedings page for the 2024 NeurIPS example