High-Dimensional Phenomena

Why maxima, distances, norms, and random directions behave differently in high dimension, and why low-dimensional intuition starts to fail.
Modified

April 26, 2026

Keywords

high-dimensional geometry, concentration of measure, almost orthogonality, maxima, distances

1 Role

This is the fifth page of the High-Dimensional Probability module.

The previous page explained why random vectors naturally turn into random-matrix questions.

This page steps back and asks:

what geometric effects become typical once dimension is large?

2 First-Pass Promise

Read this page after Random Matrices and Spectral Concentration.

If you stop here, you should still understand:

  • why maxima, norms, and pairwise relations scale differently with dimension
  • why random high-dimensional vectors are often nearly orthogonal
  • why distances and averages can look surprisingly stable in high dimension
  • why these effects matter for statistics, random features, and learning theory

3 Why It Matters

High dimension is not just “many coordinates.”

It changes which events are typical.

For example:

  • the Euclidean norm of a random vector may be very stable
  • the largest coordinate may still grow like \sqrt{\log d}
  • two random directions may be almost orthogonal
  • many points may lie near the boundary of the region that contains them

These effects explain why high-dimensional statistics and machine learning often feel counterintuitive if we reason with only low-dimensional pictures.

4 Prerequisite Recall

  • scalar concentration controls one quantity at a time
  • vector concentration moves from coordinates to norms and directions
  • matrix concentration controls geometry through operator norms and spectra
  • dimension changes the number of directions, coordinates, and comparisons we must control simultaneously

5 Intuition

5.1 Typical Scale Versus Extreme Scale

A high-dimensional random object can have several natural scales at once.

For a Gaussian vector in \(\mathbb R^d\):

  • the norm lives near \sqrt d
  • the largest coordinate lives near \sqrt{\log d}

Both statements are true at the same time.

That is the first big lesson:

different summaries of the same object respond to dimension in different ways

5.2 Almost Orthogonality

In two or three dimensions, random directions do not feel especially orthogonal.

In high dimension, they often do.

That is because the inner product between two independent random unit directions is usually very small.

So “generic directions” tend to interact weakly.

5.3 Distance Stability

When norms concentrate and pairwise inner products stay small, distances between random points often become much more predictable than low-dimensional intuition suggests.

This is one reason random design and random features can preserve geometry well enough to support useful theory.

6 Formal Core

Definition 1 (Definition: High-Dimensional Phenomenon) In this module, a high-dimensional phenomenon means a geometric or probabilistic effect whose typical behavior becomes clear only when dimension-dependent scaling is made explicit.

Typical examples are:

  • maxima scaling like \sqrt{\log d}
  • norms scaling like \sqrt d
  • random directions becoming nearly orthogonal
  • distances concentrating around a narrow band

Theorem 1 (Theorem Idea: Maxima Grow More Slowly Than Norms) If \(g\in\mathbb R^d\) has independent sub-Gaussian coordinates, then

\[ \max_{1\le i\le d}|g_i| \]

typically scales like \sqrt{\log d}, while

\[ \|g\|_2 \]

typically scales like \sqrt d.

This is one of the cleanest examples of dimension creating multiple natural scales at once.

Theorem 2 (Theorem Idea: Random Directions Are Nearly Orthogonal) If \(u\) and \(v\) are independent random unit vectors in high dimension, then

\[ \langle u,v\rangle \]

is usually very small, often on the scale 1/\sqrt d.

So in high dimension, “generic directions” tend to be almost perpendicular.

Theorem 3 (Theorem Idea: Distances Concentrate) For many isotropic sub-Gaussian random vectors, pairwise distances and norms fluctuate much less than a low-dimensional picture would suggest.

The exact theorem depends on the model, but the first-pass point is:

  • geometry becomes more regular
  • but only when we measure the right quantity on the right scale

7 Worked Example

Let \(g=(g_1,\dots,g_d)\) with independent standard Gaussian coordinates.

Then:

  • the norm satisfies \|g\|_2 \approx \sqrt d
  • the largest coordinate satisfies \max_i |g_i| \approx \sqrt{\log d}

These are not competing statements.

They describe different summaries of the same vector.

Now take two independent copies \(g,h\) and normalize them:

\[ u=\frac{g}{\|g\|_2}, \qquad v=\frac{h}{\|h\|_2}. \]

Then the inner product \(\langle u,v\rangle\) is typically small.

So the same high-dimensional world gives us:

  • stable norms
  • small coordinatewise extremes relative to the norm scale
  • almost orthogonal random directions

That combination is exactly why high-dimensional geometry is so useful in modern theory.

8 Computation Lens

When reading a theorem or running a simulation, always ask:

  1. what quantity is being measured
  2. what its natural dimension-dependent scale is
  3. whether the theorem is about a typical value, an extreme value, or a worst-case value

This is often the difference between understanding a result and misreading it.

9 Application Lens

9.1 High-Dimensional Statistics

Modern covariance estimation, sparse recovery, and random design theory all rely on knowing that certain random objects are more regular than raw coordinate-level intuition suggests.

9.2 Learning Theory

Capacity control, random features, and overparameterized linear models all use the fact that high-dimensional randomness can preserve geometry while still creating dimension-dependent penalties like \log d.

9.3 Machine Learning Practice

Embedding spaces, random projections, nearest-neighbor behavior, and feature conditioning all feel strange without this page’s geometric viewpoint.

10 Stop Here For First Pass

If you can now explain:

  • why norms and maxima live on different scales
  • why random high-dimensional directions are often nearly orthogonal
  • why distance stability is plausible in high dimension
  • why low-dimensional pictures can be misleading

then this page has done its job.

11 Go Deeper

After this page, the next natural step is:

The strongest adjacent live pages right now are:

12 Optional Deeper Reading After First Pass

The strongest current references connected to this page are:

13 Sources and Further Reading

Back to top