Rewriting the statistical mechanics of learning in function space

A Function-Space Approach to the Statistical Mechanics of Learning Dynamics is a preprint posted on arXiv as 2609.09589. It argues that learning shows a more stable collective order when we treat functions and their dynamical operators as macroscopic variables instead of parameters. The link to the original paper sits under the title.

I picked up this paper for a simple reason. Neural networks entangle millions of parameters in nonlinear ways, yet their learning curves move with a strange regularity, and I have always wondered where that regularity comes from. If you follow every parameter trajectory, the explanation gets tangled very fast, so the authors propose to change the level of description. They push microscopic parameter arrangements down to realizations and lift the way functions reduce error up to the macroscopic scene. So what does this move in level actually show. Let me follow that question from the start.

Why look at functions instead of parameters

Training a neural network usually means updating weights. Yet anyone with training experience knows that many different weight arrays can produce the same function. Permuted neurons, flipped signs, and canceling paths can give identical outputs, and such redundancy grows explosively with scale. So parameter trajectories often zigzag at the microscopic level while the function the model expresses changes slowly and smoothly.

I think this gap is exactly where statistical mechanics belongs. Just as we use pressure and temperature instead of tracking every gas molecule, the proposal is to use functions and the operators that drive their change instead of tracking every parameter. That is the scene this paper puts forward. Deep networks are fiercely nonlinear in parameter space, but at the macroscopic level of function space, regular collective behavior appears.

An interesting question follows from this. Which operators deserve to be macroscopic variables. The authors answer with the learning operator M. For mean squared loss, the exact dynamics of error are governed by M, which is built from the parameter to function Jacobian J in the form M = J J*. The formula looks dry, but the meaning is intuitive. J collects how sensitively the function changes in each direction when parameters move a little, and M sums that sensitivity over function directions. In other words, M is a map that tells us which function directions learn fast and which ones lag behind.

What many roads to the same answer means

The many to one correspondence between parameters and functions is a condition a theory cannot simply skip. Countless parameter arrays stand behind the same function, and dynamics alone does not reveal that redundancy. So the authors take a path that uses both the weight coming from dynamics and the density of states in parameter space.

The first part is easier to unpack. A dynamical Boltzmann weight appears, derived from conditional stochastic dynamics. You can read it as a scale that measures how reachable each function trajectory was while learning proceeded with stochastic wobble. But one scale is not enough. Unless we also count how many parameter realizations hide behind the same function trajectory, we cannot separate what is frequently observed from what merely occupies a huge volume. That is why the parameter space density of states enters, and the product of the two becomes the starting point of the theory.

Human work runs in a similar way if you think about it. People in a meeting may reach the same conclusion through slightly different reasoning, and some of those paths are common routes many people could have found while others are rare routes only one or two people would find. Judged only by persuasiveness, the conclusions look the same, but counting the paths behind them separates common conclusions from rare ones. The paper brings in the density of states for the same reason. To explain collective behavior, we have to measure both the plausibility of a function trajectory and the parameter volume that realizes it.

The next problem is immediate. The density of states is not spread evenly, and it curves locally around each function. If we ignore that curvature, the theory turns neat but drifts from reality, and if we keep it as is, the calculation becomes hard to carry. So how can we handle this local bending.

How error actually moves

Under mean squared loss, the story turns quite crisp. Error in function space shrinks along the directions and speeds set by the learning operator M, and this is written as exact dynamics rather than an approximation. Without chasing complicated nonlinear trajectories in parameter space, error in function space moves in an ordered form governed by a single operator M. I liked the attitude in this part. It does not deny microscopic complication, yet it lifts the level of description and pulls regularity out of it.

Real learning still contains fluctuations. Minibatch randomness, initialization differences, and numerical noise keep nudging the trajectory, and the spectrum of M decides which directions amplify or damp that nudge. Directions with large eigenvalues are where error settles fast, while directions with small eigenvalues are the residue that lingers. A common simple reading here is that fast directions are the important ones, but this is where I would be careful not to overread the result. Fast settling does not always mean good for generalization, because we still have to ask what exactly settles fast.

The statistical operator B is the device that answers that question. Defined by the local curvature of the density of states, B couples to dynamics through the fluctuation term and measures which function directions are statistically heavy. If M sets the speed of motion, B plays a role like the lay of the land on which that motion runs. When the two operators are viewed separately, the explanation stays at half strength, and the full picture arrives only when we see how they mesh.

What you see when you count the wobble

Once local fluctuations are integrated out, the most striking formula in the theory appears. Apart from a constant, the fluctuation contribution reads Phi_fluc(M;B) = (sigma_xi^2/2) log det(M^{-1} + B). It looks heavy with symbols at first, but taken apart, the meaning is clear. The variance sigma_xi^2 measures the strength of fluctuations, M^{-1} contributes more in directions where learning is slow, and B adds the local curvature of the density of states. The determinant and logarithm are the statistical mechanical way of multiplying and adding contributions from all directions.

What interests me more is the structure behind the numbers. This formula insists that M and B be judged together, never separately. Good M alone does not stabilize learning, and flat B alone does not remove fluctuations. The sum of the two operators moves as one body inside the determinant, sets the volume of possibilities, and the theory favors arrangements that shrink that volume. Just as statistical mechanics favors states with lower free energy, this picture favors operator arrangements with smaller fluctuation contribution.

One misunderstanding should be cleared early. It would be wrong to read fluctuations here as mere noise to remove. Fluctuations are closer to a signal that carries information, a probe that reveals which directions are statistically heavy. A direction with large fluctuations may sit where the density of states changes steeply, while a direction with small fluctuations may sit in a broad flat valley covered by many parameter realizations. So integrating over fluctuations does not mean erasing noise, it means reading the shape of the terrain from the shape of the noise. Then what happens when these two terrains point in different directions.

When two operators look in the same direction

With the spectrum fixed and only rotations allowed, the fluctuation term becomes rotationally stationary when M and B commute ([M,B]=0). The phrasing sounds difficult, but the mental image is simple. When the fast directions of M and the light directions of B share the same axes, small rotations change nothing in total volume. Once the axes line up, turning either side a little costs nothing.

I found this condition one of the more elegant moments in the paper. Instead of assuming alignment from the start, it shows why an aligned state is preferred, through the stationarity of an energy like quantity. In other words, the matching arrangement is not fitted by hand, it becomes a fixed point on the scale of the fluctuation contribution. Actual learning does not sit on this fixed point at every moment of course, because M and B both keep changing as learning proceeds, so the fixed point is closer to a moving target. Still, the direction the theory points to is clear. Learning leans toward states where the axes of the two operators interlock rather than states where they drift apart.

From a systems perspective, the next question is obvious. Sharing eigenvectors sounds strict, so how exact can that sharing be in millions of dimensions. The stationarity the paper describes lives under local and idealized conditions, so in practice it is healthier to read it as partial alignment rather than perfect commutation. Even so, this condition prepares the ground for the minimization principle that follows. If aligned axes are stable, we then have to ask which axes should pair with which.

Why the fast path overlaps the smooth path

At fixed spectrum, the fluctuation term is smallest when large eigenvalues of M pair with small eigenvalues of B. The sentence is long, but the point is compact. Total volume shrinks when fast learning directions overlap statistically light directions. That is why fast relaxation tends to happen in low B subspaces.

An interesting question opens here. Why should fast directions meet light directions. Unpacked intuitively, large M directions are where error vanishes quickly so fluctuations do not linger, and small B directions are where the density of states is gentle so parameter realizations spread widely. When the two overlap, error settles fast while landing in a broad valley, which keeps the fluctuation contribution small. When large M meets large B instead, motion is fast but rides a narrow steep ridge, so fluctuations charge a higher price. The theory prefers the former and avoids the latter.

When our team looks at a new AI architecture, we do not look at benchmark numbers alone. We look together at what actually changed and what cost structure that change creates in a system. Reading this result with the same attitude, where speed happens matters more than speed itself. Fast relaxation in a low B subspace arrives with a broad realization volume, so it reads as stable, while fast relaxation in a high B subspace may amount to a rushed fit into a narrow solution. I therefore understood this pairing principle as a story about stable speed rather than speed alone.

What pulls things back when they drift apart

When the axes of M and B misalign, they do not simply stay misaligned, and a local restoring contribution appears against the rotational mismatch. Like a spring that pulls back when twisted a little, the theory leans toward reducing the twist when the two operators point in different directions. If the stationarity condition is the fixed point, this restoring part is the slope around it.

This part makes learning feel like a living process. M keeps changing with the data and the current function, and B also varies with position because it reflects local curvature of the density of states. The two operators do not align once and stop while fixed in place, they chase each other and repeat cycles of drifting and returning. During that process, the restoring contribution works like a signpost. As mismatch grows, the tendency to return grows with it, which makes it hard for the two axes to wander completely separately.

Still, this force should not be taken too mechanically. Restoring here does not mean autopilot toward a fixed destination. It is closer to a statistical lean toward smaller local volume than a device that guarantees a global optimum. I think this distinction matters. Read as a guarantee of training success, the theory overreaches, but read as an explanation for why learning often leans toward axis alignment, it persuades. So where does this abstract alignment story meet the properties of real networks.

What smoothness means in a ReLU space

The theory feels most concrete when stable statistical conditions are placed on ReLU type function spaces. There B takes the form B = sigma_xi^2 L* K L, where L measures coarse grained second order structure. The symbols may look unfamiliar, but the message is surprisingly plain. Instead of measuring local curvature directly, which is hard, it blurs the function a little and then measures second order bending on that blurred view.

Small B in this form is close to saying structural curvature is low, and as long as the anisotropy of K stays bounded, smooth and data adaptive directions overlap low B directions. In plain terms, gently bending directions relax before sharply kinked ones, and directions that follow the grain of the data relax before directions that stick out against it. This connects naturally with the familiar observation that networks often capture rough outlines early in training and fill in details later.

This is where I would be careful not to overread the experiment. Smoothness equaling goodness always invites counterexamples. That smooth directions relax first does not mean smooth solutions are always right, nor does following the grain of the data guarantee every problem gets solved. The paper states a preferred direction, not a victory announcement. I therefore think it is more accurate to read this as terrain that explains why networks often lean toward smooth solutions than as a magic formula for generalization. The role of K should be understood in the same way. When K is severely distorted, the same curvature can feel different by direction, and the paper links smoothness to low B only under the condition that this distortion stays within bounds.

One concrete scene makes this easier to picture. Imagine training an image classifier late at night. In early epochs, large structures such as the overall silhouette of a cat settle first, while fine textures such as single whiskers follow later. Through the eyes of L, the former sit in directions with small coarse grained second order structure and the latter in directions with large structure. What the theory says is that the former carry small B, pair easily with fast relaxation, and so the broad outline settles first. Actual training can shuffle this order through augmentation or optimizer settings of course, which is why this picture should be read as a tendency rather than a law.

How far should we trust this picture

A beautiful theory should not be mistaken for reality right away. The premises of this paper are explicit, and where those premises break, the conclusions shake with them. Exact dynamics under mean squared loss are the starting point, but they do not transfer as is to cross entropy for classification or to contrastive objectives. The fluctuation strength sigma_xi^2 is not a constant through training either, it wobbles with schedules, batches, and optimizers. The analysis at fixed spectrum with only rotations allowed also cuts out one slice of real learning, so it stands apart from reality where eigenvalues themselves keep moving.

Even so, the paper did not feel hollow to me, because it does not hide its assumptions and pushes as far as it can within them. It writes error dynamics exactly without leaning on rough intuition, shows where the curvature of the density of states enters in equations, and resolves why alignment and pairing are preferred inside the same equations. Such honest scoping often lasts longer than exaggerated generalization. What I would watch next is exactly this extension. How the operator structure deforms under other losses, whether the alignment tendency survives while spectra move, and how much finite batches and learning rate schedules shake the conclusions through sigma_xi^2. If those threads connect, the weight of the theory will change.

Another point worth naming is what this theory does not explain. It does not directly answer empirical questions such as how many points a benchmark rose or at which model scale a reversal happens. Instead it explains in the language of terrain why numbers often gather in a certain direction. Papers that dispute numbers and papers that ask why numbers gather have different jobs, and this one clearly belongs to the second group. So when reading it, it helps to ask why the scoreboard tilts the way it does rather than hunting for a new row in a table.

Why treating function space as the macro level matters

Returning to the start, the largest question in this paper is not a technical formula but a choice of level. The proposal is to demote parameters to microscopic realizations and promote functions with their dynamical operators to macroscopic variables, because that level shows the stable collective organization of learning more clearly. Many people share the observation that deep networks behave regularly at the macro level despite nonlinearity across vast parameter spaces, but which variables should record that regularity has always stayed open. The authors point to function space as the answer.

This choice also gives a practical hint. When watching training, do not chase weight trajectories alone, watch which function directions settle first and whether those directions are smooth ones that follow the grain of the data. Instead of feeling reassured because the loss curve descends smoothly, build the habit of asking in which subspace it descends. This insight does not turn into a new optimizer right away of course. But a theory that tells us what to measure can be as useful as a prescription that tells us what to change.

So what remains for people to do. As theories grow precise, the responsibility of interpretation stays with humans. Equations do not decide for us under which conditions a result was read, how far it may generalize, or what stays open. Closing this paper left those distinctions sharper for me. Viewing learning from the macroscopic level of function space is clearly attractive, and the account of alignment and smoothness meshes well with intuition. What remains is to test how far this attractive map carries across the rough terrain of real training. If that testing is the task of future work, then what we understood today makes a fairly solid starting point for the next step.