Watching Training Move: Causal Forecasting of Neural Updates Across 8,192 SGD Transitions

TiGa-RCE
Independent researcher; founder and sole operator of Brewster Jennings

Article prose and original figures © 2026 TiGa-RCE, licensed under CC BY-ND 4.0.

If we change one training update, how much of that change will still be visible thousands of updates later?

The Synthetic Neural Observatory (SNO) grew out of trying to understand that question. The project began with a visual introduction to MNIST, gradient descent, and backpropagation. It became a set of small, deterministic experiments in which a training event can be changed, its effect predicted, and the rest of training replayed to find out what actually happened.

Two results are worth looking at together. In a 90-parameter network, low-order predictions tracked sampled interventions through 8,192 SGD transitions. That result includes a numerical-resolution rule changed after an earlier run, which I describe below. In a separate MNIST experiment, predictions correlated almost perfectly with exact results and still picked the wrong winner in three of 250 candidate pools. Evaluating the two highest-ranked candidates exactly recovered all 250 winners in that audit.

Those experiments suggest a useful role for prediction: choosing which counterfactual training runs to evaluate first. They also show why prediction error and selection error need separate measurements. This article reports the experiments, their failures, and the evidence needed to reproduce them. The work has not yet been peer reviewed.

What the observatory records

The first SNO fixture was an 85-parameter 2 → 8 → 6 → 1 network on an XOR-like task. At that size, it was practical to enumerate interventions and replay every alternative. The later fixtures kept that emphasis on comparisons that could be checked exactly.

SNO keeps five operations distinct. Training changes persistent parameters. Probing reads a frozen representation. An intervention changes a specified component. Prediction estimates the consequence. Exact replay executes the altered training trajectory. Keeping those operations separate avoids a common mistake: assuming that a feature is causally important because a probe can decode it.

Let q_t contain the complete differentiable training state at step t. That includes parameters, optimizer state, differentiable buffers, and any other modeled memory that can affect later computation. A deterministic transition is

$$q_{t+1}=F_t(q_t;c_t),q\_\{t+1\}\; =\; F\_t(q\_t;\; c\_t),$$

where c_t holds the fixed batch identity, hyperparameters that are not being differentiated, and deterministic control context. An intervention changes an explicit part of the state or an input to this transition.

Events and measurements have stable identifiers. Paths distinguish forward computation, reverse derivatives, parameter updates, propagation between steps, and effects between task heads through shared parameters. This makes it possible to ask where a prediction went wrong. A residual with the wrong derivative order calls for checking missing state, numerical error, and implementation defects before giving it a causal interpretation.

The experiments concern finite changes along an active training trajectory. Training does not have to end at a stationary point. The small models make exact comparisons affordable; they say little on their own about performance at production scale.

The experiments use different models. Parameter counts below refer to the network being trained in each experiment; frozen feature extractors are excluded.

When an accurate prediction picks the wrong update

The MNIST experiment used a deterministic 784 → 16 → 16 → 10 sigmoid classifier with 13,002 parameters. Its architecture follows the compact teaching example in 3Blue1Brown's neural-network sequence. The trained classifier reached 94.25% accuracy on the sealed test split. That accuracy describes the fixture and was not used to choose counterfactual branches.

At a frozen epoch-5 checkpoint, each of the final two layers had 24 possible update actions. Together they produced 576 joint one-step actions, all evaluated exactly. A ridge-regression connector learned the interaction between the two layers beyond what isolated predictions captured. It was fitted on 345 actions, calibrated on 115, and evaluated on the remaining 116.

On the held-out actions, mean absolute error fell from 5.67995e-5 to 9.33523e-6, an 83.6% reduction. Pearson correlation reached 0.999967. To test whether that accuracy was useful for choosing an update, the experiment ranked repeated pools of 20 candidates:

• The highest predicted candidate was the true winner in 247 of 250 pools, or 98.8%.
• Exact evaluation of the top two recovered the winner in all 250 pools, with zero regret.

The three misses matter. Correlation summarizes the predictions across the action set; it does not tell us whether a close contest at the top was decided correctly. Exact evaluation selects the better shortlisted candidate, but says nothing about an omitted candidate unless coverage is checked too.

These were correlated pools formed by repeated permutations of the same 116 held-out actions. They were not 250 independent training runs. The top-two shortlist was assessed on that same pool audit, so its perfect recovery needs a fresh confirmation before being treated as a reliable policy elsewhere. All 576 actions had already been evaluated to build the reference. The experiment therefore does not establish total cost savings after fitting, calibration, and audit costs.

The learning-rate test exposed another limit. Fitting used multipliers of 0.25, 0.75, and 1.5. At an unseen multiplier of 4.0, exact evaluation of 64 joint actions found that 61, or 95.31%, exceeded the calibration q99 error bound of approximately 3.44e-5. The script assigns ABSTAIN_EXACT_ALL to this known shifted condition. It is a fixed policy, not a detector that discovered the shift before replay. The exceedance rate itself requires the exact losses.

Figure 1. Winner misses across 250 correlated pools of held-out MNIST actions: three for top-1 and zero after exact evaluation of the top two. These are miss counts, not loss-regret magnitudes. The top-two result is observed coverage in this fixture. The shifted-action errors in panel B were measured retrospectively using all 64 exact outcomes.

Transfer and checkpoint placement in separate fixtures

The continuation tests used a nine-parameter binary linear head trained with Adam on eight frozen PCA features from the XOR network. Each decision ranked 1,280 possible first-step updates. Each first step had 1,280 possible second-step continuations, giving 1,638,400 terminal paths. The predictor ranked first-step branches by their estimated best continuation loss; exact replay then evaluated all second steps for the shortlisted branches.

A top-64 policy passed confirmation on the same training sequence but missed a transferred winner whose first-step branch ranked 650th out of 1,280. That shortlist covered 81,920 terminal paths. A coordinate-repair procedure changed which failures occurred without eliminating them. A later, frozen top-128 policy passed its specified untouched confirmation set. The earlier failures remain in the record.

The exploratory checkpoint-placement test used a shared 8 → 8 tanh adapter and eight binary heads, totaling 144 trainable parameters over frozen XOR PCA features. It followed one deterministic 64-transition sequence with alternating training batches. At each measured time, exact replay followed eight possible head-gradient deletions for four updates and measured all eight validation-head losses. Winner recovery covered 256 decisions across 32 evaluation times and eight target losses, with eight deletion events to choose from in each decision.

Eight checkpoints chosen using accumulated gradient drift and eight chosen using representation drift each matched or slightly improved on a uniform sixteen-checkpoint atlas under the fixed relative comparison criteria. The representation-based selection reached 92.68% material-sign accuracy and 89.45% winner recovery. Every policy retained 100% recall of routes above the stated materiality floor. The selected positions were nearly uniform because the sequence had no strong change point. This is evidence for using fewer observations on that sequence, but it leaves adaptive placement on a new sequence untested.

Interactions can change the search itself

This experiment uses a 24-parameter model over frozen PCA features from the XOR network: an 8 → 2 tanh adapter with two binary heads. Head A predicts XOR; head B predicts whether the first input coordinate is positive. The checkpoint is the original capacity_competing configuration with bottleneck width 2 and seed index 0, qualified after 920 warmup updates. Training uses 600 examples, and action selection uses 200 validation examples. The test split is kept out of selection.

Each head has a lane of possible updates. An action chooses one of 256 binary masks over the adapter's eight input rows and a learning-rate multiplier from 0.25, 0.5, 1, 2, and 4, applied to a base rate of 0.02. That gives 1,280 actions per lane. The mask affects the adapter-weight gradient; the shared bias and private head updates still receive the learning-rate multiplier. Both head gradients are computed at the same state, combined, and applied once. In a second step, gradients are recomputed on the changed state.

An early search rule kept only 40 actions per lane and excluded the true one-step optimum. The complete reference contains 1,638,400 action pairs. To describe their interaction, let L_0 be the baseline validation loss sum, L_{i0} and L_{0j} the isolated-lane scores, and L_{ij} the joint score. The matrix-completion model fits the residual left by adding the isolated effects:

Here u_i and v_j each have two components. Their rank is a choice for the surrogate, separate from the network's two-unit bottleneck. The implementation fits them with 80 alternating least-squares sweeps and ridge regularization.

The fit uses an expanded 80 × 80 action matrix containing 6,400 entries. A random 25% observation mask, supplemented to cover every row and column, selected 1,703 entries, or 26.6%, for fitting; 4,697 entries remained held out. The rank-2 model achieved mean absolute error 3.24e-8 and selected the best held-out pair with zero regret. The expanded action set had already been chosen using outcomes from a 100,000-pair sample. These entries were held out from the matrix fit, not from the entire selection procedure. The reported errors also depend on the archived loss-evaluation conventions described in the reproduction notes.

Adaptive expansion uses a different predictor:

$${\widehat{R}}_{ij}={\varphi }_i^{\mathrm{\top }}W{\varphi }_j\mathrm{.}\backslash widehat\; R\_\{ij\}=\backslash phi\_i^\backslash top\; W\backslash phi\_j.$$

Each 11-component action feature vector contains a constant, eight mask bits, the learning-rate multiplier, and its square. W is fitted by minimum-norm least squares; it is not constrained to rank 2. Starting from the incomplete original action set, this procedure first included the global optimum identified by the exhaustive reference when its evaluated set reached 2,025 joint paths, about 0.124% of the full universe.

The predictor also used 2,560 isolated-lane scores, 1,280 for each lane, and one unchanged-state baseline score. Together with the 2,025 joint scores, that is 4,586 scores, about 0.28% of the pair universe. The script obtains the joint scores from an exhaustive reference built beforehand. This count describes score coverage with reuse; it excludes construction of that reference, checkpoint training, gradient calculations, predictor fitting, and prediction work. It is not a measured total compute saving, and these scores alone could not certify a global optimum.

For two successive updates, the experiment sampled 30,000 paths with replacement from the full action alphabet: 1,280 choices for each lane at each step, or 1,280^4 = 2,684,354,560,000 possible ordered paths. It evaluated the sampled paths exactly and held out 7,500. Disconnected propagation had mean absolute error 1.95e-5. Adding independent one-step interaction terms did not improve it. A separate ridge connector, fitted on 22,500 paths using 61 features describing cross-step action products and mask agreement, reduced error to 9.54e-6 and selected the best held-out path. Its ridge coefficient is 1e-6. The one-step reference is exhaustive; the two-step result covers a sample.

Following a change through training

SNO uses Taylor coefficients to follow a specified intervention through later updates. For intervention scale ε, the state is represented with factorial-scaled coefficients:

Through a transition F, these coefficients propagate as

$$b_0=F(a_0),b\_0\; =\; F(a\_0),$$

$$b_1=DF[a_1],b\_1\; =\; D\; F[a\_1],$$

$$b_2=DF[a_2]+\frac{1}{2}{D}^{2}F[a_1,a_1],b\_2\; =\; D\; F[a\_2]\; +\; \backslash frac\{1\}\{2\}D^2F[a\_1,a\_1],$$

These are univariate directional jets. A single ε moves along one intervention direction, even if that direction changes many coordinates. Independently varying several interventions would require mixed coefficients whose number grows combinatorially with dimension and order. The memory results below do not solve that multivariate problem.

Even ordinary SGD can make this expensive: the third derivative of its state transition may require contracted fourth derivatives of the training loss. Unsupported operations, hidden gradient stops, nonfinite coefficients, and discrete branches that change with the intervention prevent using the recurrence without further work.

Forward tangents and reverse adjoints provide two ways to traverse the temporal computation graph. Their agreement checks the derivative implementation. Finite interventions need a further test against exact replay. The useful signature is how the residual shrinks as ε decreases: a correct first-order prediction should leave approximately quadratic error, a second-order prediction cubic error, and a third-order prediction fourth-order error, until numerical precision takes over.

What survived 8,192 transitions

The long-horizon fixture has 90 trainable parameters: a shared eight-dimensional tanh adapter and two binary heads over frozen PCA features. It uses full-batch SGD with a learning rate of 0.02. This is a separate model from the 13,002-parameter MNIST classifier. The tanh activation is smooth, and fixed full-batch updates remove minibatch sampling variability. These choices make a controlled setting for testing high-order derivatives; they do not guarantee that the approximation stays accurate over a long trajectory.

At one selected update, the intervention attenuates a head's entire gradient contribution. Both heads' subsequent gradients are then recomputed on the changed state. The horizon doubles from 32 through 8,192 transitions. At each horizon, interventions are sampled at the origin, midpoint, and final transition, for each of two event heads and both terminal losses, using deletion scales 0.25, 0.5, 0.75, and 1.0.

The table reports full-deletion results. Third-order relative MAE is the mean absolute residual divided by the mean absolute exact effect across the 12 sampled event/target combinations. Material-sign accuracy includes only effects with magnitude at least 1e-12.

At 8,192 transitions, third-order relative MAE was approximately 1.12e-6, below the declared 1e-4 bound. Deleting the origin update still produced a mean absolute terminal effect of 4.13727e-7. Across deletion scales, origin-stratum residuals above the numerical floor had fitted slopes of approximately 2.003, 3.004, and 4.006 after first-, second-, and third-order prediction, with R² above 0.9999998. There was still an effect to predict, and the residuals shrank at the expected orders.

One qualification affects the pass classifications. The first 512-step run produced residuals too close to arithmetic resolution for the required slope fit. After seeing that result, the rule was changed: full-deletion mean absolute error below 1e-13, ten times the declared 1e-14 numerical floor, was labeled NUMERICALLY_RESOLVED. This is a post-observation correction, so the affected classifications are exploratory. They are not measured slope passes under a rule frozen before the experiment.

At 8,192 transitions, the second- and third-order midpoint and final-position fits use that resolution label. The origin fits pass the slope criterion. No breakpoint was recorded under these partly exploratory rules at the sampled positions. A fresh run with the rules fixed in advance is still needed, as are tests of other models, optimizers, and event positions.

The memory cost of keeping the graph

Retaining the higher-order computation graph became expensive. On September 14, 2026, the complete verification command reran every horizon from 32 through 8,192 on an arm64 Mac. It used Python 3.13.13, NumPy 2.5.1, Matplotlib 3.11.1, and PyTorch 2.13.0, with float64 and one CPU thread.

The command took 2,197.6 seconds, or 36 minutes 37.6 seconds. Maximum resident set was 19.93 GB, peak process footprint was 38.22 GB, and no swaps were recorded. Both the regenerated checkpoint and the canonical result matched their preserved SHA-256 hashes exactly. These measurements cover the complete verification command, not a single 8,192-step trajectory.

Stepwise Taylor-jet materialization offers a different memory tradeoff. After each transition, it materializes the coefficients and releases the previous graph. The recurrence stays the same; the graph no longer needs to remain alive across the whole sequence. At depth one, the measured materialized path is slower and uses more incremental memory. It has little retained history to release while still paying the cost of materializing coefficients. The measurements show this shallow-depth penalty, but they do not separate allocation, dispatch, and evaluation costs.

In the tested MLX implementation:

• Eight balanced routes passed through depth 8, and diagonal and cross-head sentinels passed through depth 32.
• At depth 32, materialization reduced incremental peak memory by approximately 97.7–98.0%, with roughly a 23% runtime penalty relative to retained graphs.
• Batching eight directions was 6.72 times faster than serial materialized execution, 1.96 times faster than retained MLX, and used 42.77 times less incremental peak memory than retained MLX.
• All tested 8 targets × 8 events × 3 orders coefficients matched across target batch widths.

Figure 2. The top panel shows third-order relative MAE across the sampled horizons. The other panels show a separate MLX experiment: overhead was unfavorable at depth one, but by depth 32 incremental peak allocation was 2.05–2.28% of retained-graph allocation, with median runtime at 1.23–1.24 times retained execution. These are separate experiments, not an end-to-end benchmark of an efficient 8,192-step run.

PyTorch float64 replay remained the numerical reference for the MLX tests. Incremental MLX allocation and whole-process resident memory measure different things, so the percentages above should not be applied to the CPU run's 19.93 GB.

This tradeoff matters when memory or accelerator access is the limiting resource. Accepting a longer run may make repeated verification affordable. Whether it helps in practice depends on total replay work, runtime, memory, and the decisions the predictions improve.

What changed on ZeroGPU

A portability check ran horizons 32, 64, and 128 on Hugging Face ZeroGPU using the same PyTorch 2.13 float64 code. It received the exact local CPU checkpoint as a hashed input, so checkpoint generation was held fixed.

The device was an NVIDIA RTX PRO 6000 Blackwell Server Edition MIG 2g.48gb, running Python 3.12.12, PyTorch 2.13.0+cu130, and CUDA 13.0. The run took 24.84 seconds. All three horizons kept their PASS classifications. Stop reasons, sampled positions, residual-order classifications, material-sign accuracy, and the other decision criteria matched the CPU reference.

The numbers were not identical across hardware. Among 2,373 compared numeric values, the largest absolute difference was approximately 7.03e-4, in a third-order log-log slope near the numerical-resolution boundary. Its classification did not change. The largest scaled difference came from a near-zero residual whose absolute difference was only 2.22e-16.

The same three-horizon command took 151.17 seconds in the matched local PyTorch 2.13 environment. Those timings come from different machines and operating conditions, including a shared, leased accelerator; they are not a controlled CPU-versus-GPU benchmark. The check supports decision agreement for those three early horizons. It does not reproduce the full 8,192 horizon on CUDA or provide independent replication.

The CPU and ZeroGPU records and runner source are available at fixed public revisions.

Where this fits in the literature

The ingredients have substantial prior work. Classical influence functions estimate infinitesimal training-point effects around a fitted model (Koh and Liang, 2017). TracIn accumulates first-order influence information across checkpoints (Pruthi et al., 2020). Datamodels learn counterfactual predictors from many completed training runs (Ilyas et al., 2022).

Learning-curve forecasting addresses a related question: what can partial training tell us about later performance or the value of continuing a run? Examples include parametric extrapolation (Domhan, Springenberg, and Hutter, 2015), Freeze-Thaw Bayesian optimization (Swersky, Snoek, and Adams, 2014), Bayesian neural learning-curve prediction (Klein et al., 2017), architecture-performance prediction (Baker et al., 2017), and LC-PFN (Adriaensen et al., 2023). Those methods offer useful baselines for forecast value and resource allocation. SNO's question adds a specified intervention and asks how it changes identified measurements along one trajectory.

There are good reasons to test that prediction carefully. Influence estimates can be fragile across architectures and individual points (Basu, Pope, and Feizi, 2020). A practical influence estimate may target something different from leave-one-out retraining (Bae et al., 2022). Parameter divergence can limit predictive influence even in deterministic training (Schioppa et al., 2023). Approximate unrolling addresses non-converged and multi-stage training while reducing full-unrolling cost (Bae et al., 2024).

Several trajectory methods are closer comparisons. SGD-Influence propagates linearized leave-one-out effects through a recorded SGD trajectory (Hara, Nitanda, and Maehara, 2019). Stagewise attribution finds that influence can peak or change sign around developmental transitions (Lee et al., 2026). Diff-In accumulates changes in data influence using a second-order approximation (Tan et al., 2025).

The closest comparison identified in the publication review is Deng et al. (2026), which predates the SNO experiments. It examines configuration, approximation, and systems errors in trajectory-based attribution, provides an AdamW-correct first-order recurrence, and studies an error proxy and short-look-ahead correction. It also compares estimates with trajectory-specific leave-one-out retraining. Its per-sample removal differs from the long-horizon intervention here, which attenuates a head's entire gradient contribution at one update. That distinction needs a matched experiment before it can support a claim of advantage. Higher Taylor order alone is not a novelty claim. Both Deng et al. and Diff-In were found during the later literature audit, rather than informing the completed experiments.

The differentiation and memory techniques also have an established history: reversible learning (Maclaurin, Duvenaud, and Adams, 2015), adaptive truncation (Aicher, Foti, and Fox, 2020), higher-order forward propagation (Griewank, Utke, and Walther, 2000), JAX's Taylor-mode jet machinery (Bettencourt et al., 2019), and compiler-level graph collapsing (Dangel et al., 2025). The project record shows that this family informed the stepwise implementation.

SNO brings these techniques into an experimental framework that tracks events and measurements, checks derivative order against finite interventions, and measures selection failures as well as prediction error. The experiments here do not establish priority or superiority over the cited methods.

The next tests

The immediate question is how long a fixed-order approximation remains useful as the model, optimizer, intervention size, and trajectory change. Raising the order can improve a local approximation while making derivatives and storage more expensive. It does not by itself solve the scaling problem.

A useful next study would freeze the numerical rules in advance, sample both uniform positions and positions selected by predetermined state-change indicators, and measure sign errors, ranking errors, and replay cost. It would compare full replay, periodic exact re-anchoring, and short-look-ahead correction at matched total cost. The current checkpoint-placement experiment did not test an untouched adaptive policy or a strong change point.

The continuation fixture already uses Adam for exact replay. Extending the high-order horizon result to Adam would require differentiating through its moment estimates as part of the persistent state and checking derivative order and numerical stability. Stochastic batches, larger models, additional observables, and modern language-model training remain open tests. A prospective abstention signal also remains unvalidated: the known learning-rate shift was labeled in advance, and its errors were measured after replay.

The useful evidence so far is specific. Long-range effects remained measurable in the small SGD fixture. Interaction models helped select counterfactuals, yet a strong correlation still hid wrong choices. Exact replay made those mistakes visible and showed when a shortlist repaired them. The next experiments need to establish whether that benefit survives at a cost that makes the method worth using.

Code, evidence, and reproduction

The article uses three public resources:

• The code and evidence repository, pinned to the audited commit.
• The machine-readable evidence Dataset, pinned to the revision used here.
• The read-only evidence explorer, for browsing saved results and checking its pinned Dataset manifest.

The explorer does not run a new training experiment. It uses Dataset revision bc643629900f8beb444eb498c707a6b79b14ad91; the article links revision 223121a27abc9743d77652824a90359e426b02e7, which adds the portability supplement. The 15 corresponding evidence files are byte-identical.

The repository contains reproduction entry points, measurements, figure inputs, a claim-to-evidence index, and a SHA-256 manifest.

On October 25, 2026, the central MNIST fixture was reproduced with all 14 checks passing, including exact archived-array equality, the three top-1 misses, top-two recovery in all 250 pools, and the fixed out-of-domain label. Earlier validation checked ten Taylor-jet records after mapping their pre-migration source paths to byte-identical current source; this review also checked the two sentinel records and the batch record used for the resource results.

The complete CPU horizon reproduction and the three-horizon ZeroGPU control date from September 14, 2026. They were not rerun for this editorial review. The CPU checkpoint and canonical result matched exactly; the ZeroGPU run agreed on decisions with the numerical differences described above. These are project reproductions, not independent external replications.

Reproduction notes

An earlier CPU note reported about 13 minutes, 99% of one CPU, and 3.3 GB resident memory. Its invocation and instrumentation were not preserved, so it cannot be compared with the complete September 14 verification command reported above.

The pinned repository's original Figure 2 contract contains two obsolete receipt hashes. The figure here was rebuilt from the public horizon measurements and current sentinel records, with every plotted value checked directly.

The interaction model was checked against the original runner and replay verifier, whose hashes match the September 13 provenance record. Rebuilding its small checkpoint and rerunning the rank-2 completion reproduced the reported MAE and zero regret. A bounded check also reproduced 32 joint actions and 32 two-step paths at the archived float32 precision. This did not rerun the complete exhaustive suite.

The original interaction routines use different offsets inside the binary-cross-entropy logarithms: 1e-12 for batched one-step scores and 1e-9 for the scalar baseline and two-step scores. The formulas above use those recorded scores. Each error is compared with its own archived evaluator; errors this small should not be interpreted as agreement between the two loss implementations. The 32-action check found differences of up to 5.96e-8 after float32 rounding when substituting the scalar evaluator for the batched one. Those differences concern candidate-state losses, not just the unchanged-state baseline. Although a constant baseline cancels from direct pairwise comparisons, changing it also shifts the residual matrix being fitted; a regularized rank-2 refit is not guaranteed to preserve the same ordering. The zero-regret result applies to the archived scoring conventions. Changing those conventions requires a new check.

References

1. 3Blue1Brown. Neural Networks. Educational topic collection.
2. Pang Wei Koh and Percy Liang. “Understanding Black-box Predictions via Influence Functions.” Proceedings of the 34th International Conference on Machine Learning, PMLR 70, 2017, pp. 1885–1894.
3. Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. “Estimating Training Data Influence by Tracing Gradient Descent.” Advances in Neural Information Processing Systems 33, 2020, pp. 19920–19930.
4. Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. “Datamodels: Understanding Predictions with Data and Data with Predictions.” Proceedings of the 39th International Conference on Machine Learning, PMLR 162, 2022, pp. 9525–9587.
5. Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. “Speeding up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves.” Proceedings of the 24th International Joint Conference on Artificial Intelligence, 2015.
6. Kevin Swersky, Jasper Snoek, and Ryan P. Adams. “Freeze-Thaw Bayesian Optimization.” arXiv:1406.3896, 2014.
7. Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. “Learning Curve Prediction with Bayesian Neural Networks.” International Conference on Learning Representations, 2017.
8. Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. “Accelerating Neural Architecture Search using Performance Prediction.” arXiv:1705.10823, 2017.
9. Steven Adriaensen, Herilalaina Rakotoarison, Samuel Mueller, and Frank Hutter. “Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks.” arXiv:2310.20447, 2023.
10. Samyadeep Basu, Philip Pope, and Soheil Feizi. “Influence Functions in Deep Learning Are Fragile.” arXiv:2006.14651, 2020.
11. Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger Grosse. “If Influence Functions Are the Answer, Then What Is the Question?” Advances in Neural Information Processing Systems 35, 2022, pp. 17953–17967.
12. Andrea Schioppa, Katja Filippova, Ivan Titov, and Polina Zablotskaia. “Theoretical and Practical Perspectives on What Influence Functions Do.” Advances in Neural Information Processing Systems 36, 2023, pp. 27560–27581.
13. Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger Grosse. “Training Data Attribution via Approximate Unrolling.” Advances in Neural Information Processing Systems 37, 2024, pp. 66647–66686.
14. Satoshi Hara, Atsushi Nitanda, and Takanori Maehara. “Data Cleansing for Models Trained with SGD.” Advances in Neural Information Processing Systems 32, 2019.
15. Jin Hwa Lee, Matthew Smith, Maxwell Adam, and Jesse Hoogland. “Influence Dynamics and Stagewise Data Attribution.” International Conference on Learning Representations, 2026.
16. Haoru Tan, Sitong Wu, Xiuzhe Wu, Wang Wang, Bo Zhao, Zeke Xie, Gui-Song Xia, and Xiaojuan Qi. “Understanding Data Influence with Differential Approximation.” arXiv:2508.14648, 2025.
17. Junwei Deng, Pingbang Hu, Suliang Jin, Hao Lu, Jiachen T. Wang, Shichang Zhang, and Jiaqi W. Ma. “How Faithful Is Trajectory-Based Data Attribution? Error Sources, Remedies, and Practical Guidelines.” arXiv:2605.18814, 2026.
18. Dougal Maclaurin, David Duvenaud, and Ryan P. Adams. “Gradient-based Hyperparameter Optimization through Reversible Learning.” Proceedings of the 32nd International Conference on Machine Learning, PMLR 37, 2015, pp. 2113–2122.
19. Christopher Aicher, Nicholas J. Foti, and Emily B. Fox. “Adaptively Truncating Backpropagation Through Time to Control Gradient Bias.” Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence, PMLR 115, 2020, pp. 799–808.
20. Andreas Griewank, Jean Utke, and Andrea Walther. “Evaluating Higher Derivative Tensors by Forward Propagation of Univariate Taylor Series.” Mathematics of Computation 69(231), 2000, pp. 1117–1130.
21. Jesse Bettencourt, Matthew J. Johnson, and David Duvenaud. “Taylor-Mode Automatic Differentiation for Higher-Order Derivatives in JAX.” Program Transformations for Machine Learning Workshop at NeurIPS, 2019.
22. Felix Dangel, Tim Siebert, Marius Zeinhofer, and Andrea Walther. “Collapsing Taylor Mode Automatic Differentiation.” Advances in Neural Information Processing Systems 38, 2025, pp. 147127–147166.