The Direction–Execution Gap in Physics-Informed Neural Network Optimization

By Dr. Barak Or, Founder & CEO, State16

A neural network can point downhill - and the optimizer can still take a step that makes one of its objectives worse.

We are pleased to share that our paper, The Direction–Execution Gap in Physics-Informed Neural Network Optimization, has been accepted to IEEE Computational Intelligence Letters. The paper asks a practical question: when an optimizer claims to be improving a physics-informed model, does the full update actually improve the parts of the problem we care about?

The problem in plain language

Physics-informed neural networks (PINNs) learn to approximate solutions to differential equations. They are trained by combining several objectives: matching the governing equation, satisfying boundary conditions, respecting initial conditions, and fitting any observations.

The gradient gives a local, first-order picture of what should happen immediately around the current parameters. But an optimizer does not move by an infinitesimal amount. Adam, SGD, and conflict-resolution methods execute a finite step, sometimes with momentum, adaptive scaling, and interactions between objectives. That full step can behave differently from the direction suggested at its starting point.

In everyday terms: a car can be pointed downhill at the instant you release the brake, yet momentum and steering can carry it uphill over the actual movement. The paper names this mismatch the direction–execution gap.

What the paper measures

We evaluate the same fixed audit set before and after an accepted optimizer update. The audit points are disjoint from the training and evaluation sets, and they never generate the update. This makes the comparison about the executed step itself, not about changing collocation samples.

Because the two measurements are paired on the same points, the estimator also avoids a major source of noise: comparing losses computed on different random samples. We then use one directional Hessian-vector product to add a second-order correction and test whether curvature explains a predicted reversal.

The experimental picture

The study covers 111 paired runs across four forward PDE families: 2-D Poisson, Burgers, Allen-Cahn, and Helmholtz. We examine Adam, plain SGD, and ConFIG+Adam, with additional optimizer controls, using 80 checkpoints in each 6,000-step run. In total, the fixed audit set records 24,960 component events.

The first-order signal is useful - but not sufficient

Across the paired events, the first-order sign prediction was correct 86.39% of the time. Yet the pooled reversal prevalence was 13.56%, and the conditional risk was 17.85% when the first-order signal predicted improvement.

That is the central result: a favorable local projection is informative, but it is not a guarantee that the complete optimizer step will improve the same component.

Curvature makes the failure measurable

The second-order term captures directional curvature accumulated along the actual step segment. When that curvature contribution becomes large enough relative to the linear term, the sign can reverse. This is why gradient alignment alone cannot fully describe finite-step reliability.

With the curvature correction, sign accuracy rose to 99.63%. The measured second-order reversal prevalence fell to 0.128%, with conditional risk at 0.245%. The median normalized magnitude-error ratio was 1.284 x 10^-3, compared with a baseline of 1.0 - a 99.87% reduction.

What this means for practitioners

The audit is a diagnostic instrument, not a replacement solver. It can reveal when an optimizer’s internal signal and its executed update disagree, giving researchers a way to compare update rules and decide when additional monitoring may be warranted.

A separate frozen-set guard reduced local reversal prevalence in the tested settings. However, it did not improve final relative L2 error, and the guarded procedure increased runtime by 3.65x. Local step reliability and final solution accuracy are related, but they are not the same metric.

The scope is deliberately limited

The experiments use one tanh multilayer perceptron, four low-dimensional forward PDE families, and fixed sampled audit sets. They are not a claim about every architecture, optimizer, PDE, or inverse problem. Extending the analysis to other architectures, higher-dimensional systems, operator learning, inverse problems, and independent validation sets is an important next step.

The takeaway

For physics-informed optimization, the question is not only whether the gradient points in a promising direction. It is whether the optimizer’s complete finite step executes that promise. The direction–execution gap turns that question into something measurable - and curvature provides a practical way to make the measurement much more reliable.

This work is part of STATE16’s broader focus on making intelligent systems more observable, testable, and dependable as they move from promising local signals to real-world execution.