About me

Hi — I’m Amirhossein Davoody. I live in Portland and work at Intel on statistical methods for circuit modeling.

I’ve spent most of my career bouncing between physics, software, and (more recently) machine learning. I started in a cleanroom at the University of Tehran, did a long stretch at UW–Madison studying quantum transport and nanostructures, and have since worked on everything from transistor reliability models to large-scale production code.

Where I’ve worked

Intel — I’m here now, working on circuit modeling. I was also here earlier (2018–2020) doing reliability modeling and building data-analysis tools for logic circuits.

Google — Software engineer, 2020–2023.

UW–Madison — This is where I did my PhD, a postdoc, and built AtomTouch, a molecular-dynamics app for phones and tablets. Research topics included exciton transfer in carbon nanotubes, thermoelectric nanowires, and quantum state evolution.

University of Tehran — Undergrad research: cleanroom work, sputtering, and amorphous-silicon transistors.

School

PhD and two master’s degrees from UW–Madison (ECE and physics). BS in electrical engineering from the University of Tehran.

This site

Mostly my own notes — circuit simulation, matrix stamping, spacetime geometry, things I look up or work through. Messy notebook energy, not a finished book.

Links

• LinkedIn
• GitHub

Modified Nodal Analysis

Modified Nodal Analysis (MNA) is the formulation used by SPICE-family simulators to assemble and solve large, sparse circuit equations. It extends classical nodal analysis by augmenting the unknown vector with branch currents for elements that cannot be expressed solely in terms of node voltages.

Classical nodal analysis

For a circuit with $N$ nodes (excluding the reference ground), nodal analysis writes Kirchhoff’s Current Law (KCL) at each non-reference node:

$$ \sum_k i_k = 0 $$

For resistors, the current from node $i$ to node $j$ is $G_{ij}(v_i - v_j)$ where $G_{ij} = 1/R_{ij}$. Stacking all KCL equations yields a linear system $\mathbf{G}\mathbf{v} = \mathbf{i}_s$ when every element admits a voltage-controlled stamp.

Why “modified”?

Several practical devices break pure nodal analysis:

MNA introduces extra unknowns (typically branch currents or auxiliary variables) and extra equations (branch constitutive relations or KVL constraints). The result is a square system:

$$ \begin{bmatrix} \mathbf{G} & \mathbf{B} \\ \mathbf{C} & \mathbf{D} \end{bmatrix} \begin{bmatrix} \mathbf{v} \\ \mathbf{i}_b \end{bmatrix} = \begin{bmatrix} \mathbf{i}_s \\ \mathbf{e}_s \end{bmatrix} $$

• $\mathbf{v}$ — node voltages (relative to ground)
• $\mathbf{i}_b$ — branch currents for voltage-defined elements
• $\mathbf{B}, \mathbf{C}$ — incidence / constraint coupling blocks
• $\mathbf{D}$ — usually zero for standard SPICE elements

MNA unknown vector

For a circuit with $n$ nodes (excluding ground) and $b_v$ voltage-defined branches:

$$ \mathbf{x} = \begin{bmatrix} v_1 \ v_2 \ \vdots \ v_n \ i_{b_1} \ \vdots \ i_{b_{b_v}} \end{bmatrix} \in \mathbb{R}^{n + b_v} $$

Ground is fixed at $v_0 = 0$ and is not an unknown.

Example: voltage source

An independent voltage source $V_s$ between nodes $p$ and $n$ enforces

$$ v_p - v_n = V_s $$

and introduces branch current $i_s$ as an unknown. The stamp adds one row and one column:

$$ \begin{bmatrix} \cdots & +1 & \cdots & -1 & \cdots & 0 \\ \vdots & & & & & \vdots \\ +1 & & & & & 0 \\ -1 & & & & & 0 \\ \vdots & & & & & \vdots \\ 0 & 0 & \cdots & 0 & 0 & 0 \end{bmatrix} \begin{bmatrix} \vdots \ v_p \ v_n \ \vdots \ i_s \ \vdots \end{bmatrix} = \begin{bmatrix} \vdots \ V_s \ \vdots \end{bmatrix} $$

The row is KVL ($v_p - v_n = V_s$); the columns in the $i_s$ row/column implement KCL at nodes $p$ and $n$.

Nonlinear DC analysis

For nonlinear devices (diodes, BJTs, MOSFETs), element currents become functions of local voltages: $i = f(\mathbf{v})$. SPICE linearizes around the current iterate $\mathbf{v}^{(k)}$:

$$ i(\mathbf{v}) \approx i(\mathbf{v}^{(k)}) + \mathbf{G}_\text{equiv}^{(k)} \left(\mathbf{v} - \mathbf{v}^{(k)}\right) $$

Each Newton iteration solves an MNA system with equivalent conductance stamps from the Jacobian $\partial i / \partial v$. Convergence is declared when

$$ |\mathbf{v}^{(k+1)} - \mathbf{v}^{(k)}|_\infty < \text{abstol} + \text{reltol} \cdot \max(|v_i^{(k+1)}|, |v_i^{(k)}|) $$

Dynamic (transient) analysis

Capacitors and inductors contribute companion models after discretization. With backward Euler, a capacitor $C$ between nodes $p$ and $n$ becomes a conductance $G_C = C/\Delta t$ in parallel with a history current source:

$$ i_C^{(n)} = \frac{C}{\Delta t}\left(v_p^{(n)} - v_n^{(n)}\right) + I_\text{eq} $$

where $I_\text{eq}$ depends on the previous time step. The MNA structure is unchanged — only stamps differ per time point.

MNA assembly pipeline

flowchart TD
    N[Parse netlist] --> T[Build node table]
    T --> S[Initialize sparse MNA matrix]
    S --> L{For each element}
    L --> ST[Stamp into G,B,C,D blocks]
    ST --> L
    L --> NR[Newton-Raphson loop]
    NR --> SOLVE[Sparse LU / iterative solve]
    SOLVE --> CHECK{Converged?}
    CHECK -->|No| ST
    CHECK -->|Yes| OUT[Extract v, i_b]

Advantages of MNA

1. Uniform framework — resistors, sources, controlled sources, and many nonlinear models share one matrix pattern.
2. Sparsity — each element touches only a handful of rows/columns; sparse solvers scale to millions of nodes.
3. Differentiability — linearization for Newton–Raphson is natural: stamps are Jacobians of branch relations.

See Element Stamping for how individual components map into the MNA blocks.

Element Stamping

Stamping is the process of adding an element’s contribution to the global Modified Nodal Analysis (MNA) matrix and right-hand side. Each device type has a small template that scatters values into row/column indices determined by its connecting nodes and any auxiliary branch current unknowns.

Indexing convention

Label non-ground nodes $1, \ldots, n$. For a branch current unknown $i_b$ associated with element $k$, assign it index $n + k$ in the augmented unknown vector $\mathbf{x}$.

The MNA system is $\mathbf{A}\mathbf{x} = \mathbf{b}$ with

$$ \mathbf{A} = \begin{bmatrix} \mathbf{G} & \mathbf{B} \\ \mathbf{C} & \mathbf{D} \end{bmatrix}, \quad \mathbf{x} = \begin{bmatrix} \mathbf{v} \ \mathbf{i}_b \end{bmatrix} $$

Stamping never rebuilds the matrix from scratch each iteration — it accumulates into preallocated sparse structures.

Resistor

A resistor $R$ between nodes $p$ and $n$ has conductance $g = 1/R$.

Stamp into $\mathbf{G}$:

No RHS contribution unless paired with a source network.

Independent current source

A source $I_s$ from node $p$ toward node $n$ adds to the KCL RHS:

$$ b_p \mathrel{+}= I_s, \quad b_n \mathrel{-}= I_s $$

Independent voltage source

Between nodes $p$ (positive) and $n$ (negative), with branch current $i_s$:

KVL row (new row index $r$):

$$ A_{r,p} = +1,\quad A_{r,n} = -1,\quad b_r = V_s $$

KCL columns for $i_s$:

$$ A_{p,r} = +1,\quad A_{n,r} = -1 $$

This symmetric $2\times2$ coupling pattern is the hallmark of voltage-source stamping.

Voltage-controlled current source (VCCS)

Transconductance $g_m$, controlling nodes $c$ and $d$, output nodes $p$ and $n$:

$$ i_\text{out} = g_m (v_c - v_d) $$

Stamp:

No extra unknown — pure nodal stamp.

Capacitor (companion model, backward Euler)

For transient analysis with timestep $\Delta t$, replace $C$ with:

• equivalent conductance $g_C = C / \Delta t$
• history current source $I_\text{eq}$

Between $p$ and $n$:

$$ G_{pp} \mathrel{+}= g_C,\quad G_{nn} \mathrel{+}= g_C,\quad G_{pn} \mathrel{-}= g_C,\quad G_{np} \mathrel{-}= g_C $$

$$ b_p \mathrel{+}= I_\text{eq},\quad b_n \mathrel{-}= I_\text{eq} $$

where $I_\text{eq} = g_C \cdot v_{pn}^{(n-1)}$ for the trapezoidal or backward-Euler companion (exact form depends on integration rule).

Diode (Newton linearization)

Shockley equation:

$$ i_D = I_S \left(e^{v_D / (n V_T)} - 1\right) $$

At iteration $k$, with $v_D^{(k)} = v_p^{(k)} - v_n^{(k)}$:

$$ g_d = \frac{di_D}{dv_D}\bigg|_{v_D^{(k)}} = \frac{I_S}{n V_T} e^{v_D^{(k)} / (n V_T)} $$

$$ I_\text{eq} = i_D(v_D^{(k)}) - g_d \thinspace v_D^{(k)} $$

Stamp $g_d$ as a resistor between $p$ and $n$, and add $I_\text{eq}$ to the RHS (positive into node $p$).

Stamp accumulation diagram

flowchart LR
    subgraph element["Resistor R: nodes 2—5"]
        R["g = 1/R"]
    end
    subgraph matrix["MNA matrix G"]
        M["G(2,2)+=g  G(2,5)-=g
             G(5,2)-=g  G(5,5)+=g"]
    end
    R --> M

Sparse matrix considerations

Real SPICE netlists may contain $10^6+$ elements. Stamping must be O(1) per element:

1. Precompute node-to-row maps during netlist parse.
2. Use compressed sparse column (CSC) or coordinate (COO) format; COO is often assembled then converted once per Newton iteration.
3. Reuse symbolic factorization when sparsity pattern is fixed (DC operating point); partial refactor for topology changes only.

Stamp table summary

Mastering stamping is the bridge between circuit theory on paper and the numerical kernel of any SPICE implementation.

Minkowski Space

Minkowski space is the flat four-dimensional spacetime manifold of special relativity. It replaces the Galilean separation of absolute time and Euclidean space with a single geometric structure where temporal and spatial separations enter on equal footing — but with opposite sign.

The metric

In coordinates $(t, x, y, z)$ with $c = 1$, the Minkowski metric is

$$ \eta_{\mu\nu} = \text{diag}(-1, +1, +1, +1) $$

The line element (invariant interval) between two events is

$$ ds^2 = \eta_{\mu\nu}\thinspace dx^\mu dx^\nu = -dt^2 + dx^2 + dy^2 + dz^2 $$

Einstein summation convention applies. The signature $(-+++)$ is common in particle physics; $(+—)$ appears in some GR texts — physics is unchanged up to an overall sign convention.

Event classification

For a displacement $\Delta x^\mu$ from one event to another:

The light cone at an event $P$ divides spacetime into future, past, and elsewhere:

flowchart TB
    P((Event P))
    P --> F[Future: ds² < 0, Δt > 0]
    P --> PA[Past: ds² < 0, Δt < 0]
    P --> E[Elsewhere: ds² > 0]
    P --> L[Light cone: ds² = 0]

Lorentz transformations

Transformations preserving $\eta_{\mu\nu}$ form the Lorentz group $O(1,3)$. A boost with velocity $v$ along $x$:

$$ \Lambda^\mu_{\ \nu} = \begin{pmatrix} \gamma & -\beta\gamma & 0 & 0 \\ -\beta\gamma & \gamma & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{pmatrix}, \quad \beta = \frac{v}{c},\ \gamma = \frac{1}{\sqrt{1-\beta^2}} $$

Proper Lorentz transformations ($\det\Lambda = +1$) include continuous boosts and rotations; discrete parity and time reversal are in $O(1,3)$ but not the connected component.

Four-vectors

Quantities transforming as $V’^\mu = \Lambda^\mu_{\ \nu} V^\nu$ include:

Four-position: $x^\mu = (t, \mathbf{x})$

Four-velocity: $u^\mu = \frac{dx^\mu}{d\tau}$ where $\tau$ is proper time. Normalization: $u^\mu u_\mu = -1$.

Four-momentum: $p^\mu = m u^\mu = (E, \mathbf{p})$ with

$$ p^\mu p_\mu = -m^2 \quad \Leftrightarrow \quad E^2 = |\mathbf{p}|^2 + m^2 $$

Four-gradient: $\partial_\mu = \frac{\partial}{\partial x^\mu}$

Raising and lowering indices uses $\eta_{\mu\nu}$: $v_\mu = \eta_{\mu\nu} v^\nu$.

Relativistic mechanics in geometric form

Newton’s second law becomes the geodesic equation in flat spacetime (trivial Christoffel symbols):

$$ \frac{d u^\mu}{d\tau} = \frac{q}{m} F^{\mu\nu} u_\nu $$

where $F^{\mu\nu}$ is the electromagnetic field tensor. Energy-momentum conservation for a closed system:

$$ \partial_\mu T^{\mu\nu} = 0 $$

The stress-energy tensor $T^{\mu\nu}$ encodes energy density, momentum density, and stress in a single symmetric (for perfect fluids and EM) rank-2 tensor.

Maxwell equations

In natural units ($\varepsilon_0 = \mu_0 = c = 1$), Maxwell’s equations unify as

$$ \partial_\mu F^{\mu\nu} = J^\nu, \quad \partial_{[\lambda} F_{\mu\nu]} = 0 $$

where $F_{\mu\nu} = \partial_\mu A_\nu - \partial_\nu A_\mu$ is the field strength and $J^\mu = (\rho, \mathbf{j})$ is the four-current. The second equation is the Bianchi identity (homogeneous Maxwell equations).

Why Minkowski space matters

1. Covariance — physical laws take the same form in all inertial frames without ad hoc length contraction or time dilation factors.
2. Causality — the metric defines light cones and forbids superluminal influence.
3. Bridge to GR — Minkowski space is the tangent space at any point of a curved Lorentzian manifold; special relativity is local flatness.

The transition from Minkowski space to curved spacetime is developed in Curved Spacetime.

Curved Spacetime

General relativity replaces the fixed Minkowski metric $\eta_{\mu\nu}$ with a position-dependent Lorentzian metric $g_{\mu\nu}(x)$ on a differentiable manifold. Gravity is not a force in the Newtonian sense — it is the geometry of spacetime, encoded in $g_{\mu\nu}$ and its derivatives.

From flat to curved

In special relativity, free particles follow straight worldlines in Minkowski space. In GR, free particles follow geodesics of $(\mathcal{M}, g)$:

$$ \frac{d^2 x^\mu}{d\tau^2} + \Gamma^\mu_{\alpha\beta} \frac{dx^\alpha}{d\tau}\frac{dx^\beta}{d\tau} = 0 $$

The Christoffel symbols (connection coefficients) are

$$ \Gamma^\mu_{\alpha\beta} = \frac{1}{2} g^{\mu\nu}\left(\partial_\alpha g_{\nu\beta} + \partial_\beta g_{\nu\alpha} - \partial_\nu g_{\alpha\beta}\right) $$

In flat spacetime with inertial coordinates, $g_{\mu\nu} = \eta_{\mu\nu}$ and $\Gamma^\mu_{\alpha\beta} = 0$ — geodesics reduce to straight lines.

The Einstein field equations

Matter and energy curve spacetime via

$$ G_{\mu\nu} + \Lambda g_{\mu\nu} = 8\pi G\thinspace T_{\mu\nu} $$

where

$$ G_{\mu\nu} = R_{\mu\nu} - \frac{1}{2} R\thinspace g_{\mu\nu} $$

is the Einstein tensor, $R_{\mu\nu}$ the Ricci tensor, $R$ the Ricci scalar, $\Lambda$ the cosmological constant, and $T_{\mu\nu}$ the stress-energy tensor.

In vacuum ($T_{\mu\nu} = 0$), the field equations become $R_{\mu\nu} = 0$ (when $\Lambda = 0$) — spacetime can still be curved (Schwarzschild solution).

Riemann curvature

The Riemann tensor measures intrinsic curvature:

$$ R^\rho_{\ \sigma\mu\nu} = \partial_\mu \Gamma^\rho_{\nu\sigma} - \partial_\nu \Gamma^\rho_{\mu\sigma} + \Gamma^\rho_{\mu\lambda}\Gamma^\lambda_{\nu\sigma} - \Gamma^\rho_{\nu\lambda}\Gamma^\lambda_{\mu\sigma} $$

Key symmetries: $R_{\rho\sigma\mu\nu} = -R_{\sigma\rho\mu\nu}$, $R_{\rho\sigma\mu\nu} = R_{\mu\nu\rho\sigma}$, and the Bianchi identity $R_{\rho[\sigma\mu\nu]} = 0$.

Flat spacetime has $R^\rho_{\ \sigma\mu\nu} = 0$ globally; locally, any smooth Lorentzian manifold is approximately Minkowski in a small enough neighborhood (equivalence principle).

Relation to Minkowski space

Every Lorentzian manifold has, at each event $p$, a tangent space $T_p\mathcal{M}$ equipped with Minkowski inner product $g_{\mu\nu}(p)$. Special relativity governs physics in that infinitesimal neighborhood; global curvature accumulates over finite distances.

Schwarzschild metric (spherical symmetry)

Outside a spherically symmetric mass $M$ (in geometric units $G = c = 1$):

$$ ds^2 = -\left(1 - \frac{2M}{r}\right) dt^2 + \left(1 - \frac{2M}{r}\right)^{-1} dr^2 + r^2 d\Omega^2 $$

where $d\Omega^2 = d\theta^2 + \sin^2\theta\thinspace d\phi^2$.

• Event horizon at $r = 2M$
• Gravitational redshift — clocks run slower in stronger gravitational fields
• Perihelion precession — Mercury’s orbit precesses because $R \neq 0$

Geodesic deviation (tidal forces)

Nearby geodesics do not remain parallel in curved spacetime. The separation vector $V^\mu$ between two geodesics satisfies

$$ \frac{D^2 V^\mu}{d\tau^2} = -R^\mu_{\ \nu\rho\sigma} u^\nu V^\rho u^\sigma $$

This is the relativistic origin of tidal gravity — the Riemann tensor directly measures how a gravitational field differs from place to place.

Curvature and physics pipeline

flowchart TD
    T[Tμν stress-energy] --> EFE[Einstein field equations]
    EFE --> G[gμν metric]
    G --> CHR[Christoffel symbols Γ]
    CHR --> GEO[Geodesic equation]
    G --> RIE[Riemann tensor]
    RIE --> TID[Tidal forces / deviation]
    GEO --> OBS[Observable orbits, redshift, lensing]
    TID --> OBS

Weak-field limit

For weak, static fields ($|\Phi| \ll 1$, slow motion), write

$$ g_{00} \approx -(1 + 2\Phi), \quad g_{ij} \approx (1 - 2\Phi)\delta_{ij} $$

where $\Phi$ is the Newtonian gravitational potential. The geodesic equation reproduces $\ddot{\mathbf{x}} = -\nabla\Phi$ — Newtonian gravity emerges as a limit.

Why curved spaces appear in physics

1. Equivalence principle — gravity is locally indistinguishable from acceleration; globally it requires curved geometry.
2. Coordinate independence — tensor equations on manifolds express laws valid in any smooth coordinate chart.
3. Cosmology — the universe’s large-scale structure is modeled as a time-dependent curved (often Friedmann–Lemaître–Robertson–Walker) spacetime.
4. Black holes & waves — regions where $g_{\mu\nu}$ deviates radically from $\eta_{\mu\nu}$ are observational targets (LIGO, EHT).

Minkowski space is the linearized, zero-curvature backbone; curved Lorentzian geometry is the dynamical stage on which matter writes its gravitational history.

BCS Superconductivity Theory

The Bardeen–Cooper–Schrieffer (BCS) theory explains conventional (low-$T_c$) superconductivity as a Cooper-pair condensate of electrons near the Fermi surface, stabilized by an effective attractive interaction mediated by phonons. It is mean-field many-body physics: one variational ground state captures the essential gap, thermodynamics, and electrodynamics.

Prerequisites: Fermi liquid / free-electron gas, second quantization, basic statistical mechanics at $T=0$ and finite $T$. Scope: BCS weak-coupling mean field and its measurable consequences. Eliashberg strong-coupling numerics, unconventional pairing symmetries, and high-$T_c$ cuprates are out of scope here.

Phenomenology BCS must explain

The order parameter is a complex gap $\Delta(\mathbf{r})$ (or $\Delta_{\mathbf{k}}$ in momentum space): the amplitude of Cooper-pair coherence. In s-wave BCS, $\Delta$ is uniform in $\mathbf{k}$ on the Fermi surface.

Cooper instability

Two electrons above a filled Fermi sea with opposite momenta and opposite spins ($\mathbf{k}\uparrow$, $-\mathbf{k}\downarrow$) can form a bound state if the interaction is net attractive in that channel — even when the bare Coulomb repulsion is large, phonon exchange can win at energies $\hbar\omega_D \ll E_F$ (Debye scale $\ll$ Fermi energy).

Cooper’s variational argument (1956): with an attractive square-well potential of width $2\hbar\omega_D$ around $E_F$, a bound pair exists for arbitrarily weak attraction in 3D. That instability at $T=0$ is the seed of BCS.

flowchart LR
    A[Two electrons near E_F] --> B{Net attraction in pair channel?}
    B -->|Yes| C[Cooper bound state]
    C --> D[Macroscopic pair condensate]
    D --> E[Gap Δ and superfluid response]
    B -->|No| F[Remain normal Fermi liquid]

BCS Hamiltonian (reduced form)

In momentum space, restricting to the pairing channel:

$$ \begin{aligned} \mathcal{H} ={}& \sum_{\mathbf{k},\sigma} \xi_{\mathbf{k}}\thinspace c_{\mathbf{k}\sigma}^\dagger c_{\mathbf{k}\sigma} \\ &\quad + \sum_{\mathbf{k}} \left( \Delta\thinspace c_{\mathbf{k}\uparrow}^\dagger c_{-\mathbf{k}\downarrow}^\dagger + \Delta^* c_{-\mathbf{k}\downarrow} c_{\mathbf{k}\uparrow} \right) \\ &\quad + \frac{|\Delta|^2}{g} \end{aligned} $$

Here $\xi_{\mathbf{k}} = \epsilon_{\mathbf{k}} - \mu$ is measured from the chemical potential (vanishes on the Fermi surface). $\Delta$ is the pair potential (order parameter), determined self-consistently. $g > 0$ is the effective pairing coupling (phonon-mediated attraction in the reduced model).

The quartic interaction that generates $\Delta$ is often written schematically as

$$ \begin{aligned} \mathcal{H}_{\text{int}} &= -\frac{g}{V} \sum_{\mathbf{k}} c_{\mathbf{k}\uparrow}^\dagger c_{-\mathbf{k}\downarrow}^\dagger \\ &\quad c_{-\mathbf{k}\downarrow} c_{\mathbf{k}\uparrow} \end{aligned} $$

with $g$ nonzero only for $|\xi_{\mathbf{k}}| < \hbar\omega_D$ (Debye cutoff). Mean-field decoupling replaces the four-operator term by $\Delta c^\dagger c^\dagger + \text{h.c.}$ plus $|\Delta|^2/g$.

Bogoliubov quasiparticles

Diagonalize $\mathcal{H}$ via the Bogoliubov transformation:

$$ \gamma_{\mathbf{k}\uparrow} = u_{\mathbf{k}} c_{\mathbf{k}\uparrow} - v_{\mathbf{k}} c_{-\mathbf{k}\downarrow}^\dagger, \quad \gamma_{-\mathbf{k}\downarrow}^\dagger = u_{\mathbf{k}} c_{-\mathbf{k}\downarrow}^\dagger + v_{\mathbf{k}} c_{\mathbf{k}\uparrow} $$

with $u_{\mathbf{k}}^2 + v_{\mathbf{k}}^2 = 1$ and $u_{\mathbf{k}} v_{\mathbf{k}} = \Delta / 2E_{\mathbf{k}}$. The Hamiltonian becomes

$$ \mathcal{H} = \sum_{\mathbf{k}} E_{\mathbf{k}} \left( \gamma_{\mathbf{k}\uparrow}^\dagger \gamma_{\mathbf{k}\uparrow} + \gamma_{-\mathbf{k}\downarrow}^\dagger \gamma_{-\mathbf{k}\downarrow} \right) + E_0 $$

Bogoliubov dispersion:

$$ E_{\mathbf{k}} = \sqrt{\xi_{\mathbf{k}}^2 + |\Delta|^2} $$

• $E_{\mathbf{k}} \geq |\Delta|$: the superconducting gap is the minimum excitation energy.
• Creating a real electron at $\mathbf{k}$ costs at least $\Delta$ if $\xi_{\mathbf{k}} = 0$ — scattering that would degrade coherence is suppressed at low $T$, hence zero DC resistance.

For a s-wave order parameter, $|\Delta_{\mathbf{k}}| = \Delta$ is constant on the Fermi surface; angle-dependent gaps appear in anisotropic or unconventional superconductors.

Gap equation (self-consistency)

At $T = 0$, the self-consistency condition for $\Delta$ is

$$ 1 = g \sum_{\mathbf{k}} \frac{1}{2E_{\mathbf{k}}} \quad \Rightarrow \quad 1 = N(0)\thinspace g \int_0^{\hbar\omega_D} \frac{d\xi}{\sqrt{\xi^2 + \Delta^0}} $$

with $N(0)$ the normal-state density of states at the Fermi level. Evaluating the integral gives the BCS gap equation:

$$ \Delta^0 = 2\hbar\omega_D \exp\left(-\frac{1}{N(0)\thinspace g}\right) $$

At finite $T$, thermal quasiparticle occupancy smears the gap:

$$ 1 = N(0)\thinspace g \int_0^{\hbar\omega_D} d\xi\thinspace \frac{\tanh(E/2k_B T)}{\sqrt{\xi^2 + \Delta^2(T)}} $$

The gap vanishes at $T_c$ where the linearized equation yields

$$ k_B T_c = \frac{2 e^\gamma}{\pi}\thinspace \hbar\omega_D \exp\left(-\frac{1}{N(0)\thinspace g}\right), \quad \gamma \approx 0.5772\quad \text{(Euler–Mascheroni)} $$

Weak-coupling BCS ratios (useful sanity checks):

$$ \frac{\Delta^0}{k_B T_c} \approx 1.764, \qquad \frac{C_s - C_n}{C_n}\bigg|_{T_c} \approx 1.43 $$

The isotope effect $T_c \propto M^{-1/2}$ follows because $\hbar\omega_D \propto M^{-1/2}$ when phonons mediate pairing.

Ginzburg–Landau and electrodynamics

Near $T_c$, a Ginzburg–Landau (GL) expansion in $|\psi|^2$ captures the same order parameter with a coherence length $\xi \sim v_F / \Delta$ and penetration depth $\lambda_L$. BCS microscopically fixes the GL coefficients.

London equation (local limit, $T \ll T_c$):

$$ \nabla \times \mathbf{j}_s = -\frac{n_s e^2}{m}\thinspace \mathbf{B} $$

Persistent supercurrents screen magnetic fields over $\lambda_L$ — the Meissner effect. A superconductor in an applied field is not merely a zero-resistance conductor; it is a perfect diamagnet (up to flux quantization and vortex physics in Type II materials).

Flux quantization: magnetic flux through a superconducting loop is quantized in units of $\Phi_0 = h/2e$, reflecting the $2e$ charge of Cooper pairs.

Josephson effect (device-relevant)

Two superconductors separated by a thin insulator form a Josephson junction. The DC Josephson relation:

$$ I = I_c \sin(\phi) $$

where $\phi$ is the difference of the superconducting phases across the barrier and $I_c$ depends on $\Delta$ and tunneling. The AC Josephson relation $\dot\phi = 2eV/\hbar$ links phase evolution to voltage — the basis of SQUIDs, voltage standards, and superconducting qubits.

For someone coming from nanostructures and transport, the same pairing physics appears when a normal metal or semiconductor is proximitized by a superconductor: induced gaps, Andreev reflection, and subgap conductance are mesoscopic signatures of the BCS order parameter.

Density of states

The quasiparticle DOS (per spin) is

$$ N_s(E) = N(0)\thinspace \frac{|E|}{\sqrt{E^2 - \Delta^2}}, \qquad |E| > \Delta $$

with a square-root van Hove singularity at $|E| = \Delta$. Tunneling spectroscopy (STM on superconductors, or planar junction $dI/dV$) measures this directly — a clean experimental handle on $\Delta$ and, with strong coupling, phonon structure (Eliashberg regime).

Limitations and extensions

BCS is the reference frame for conventional superconductivity: Cooper pairing, broken U(1) symmetry, gapped quasiparticles, and macroscopic phase coherence. Modern circuit QED and superconducting qubits still live in this picture, with junction nonlinearity and charge noise layered on top.

Related notes

• Quantum transport and mesoscopic signatures: pair tunneling, Andreev reflection (not yet a dedicated page here).
• Minkowski Space — unrelated physically, but the same “state a convention, then compute” style applies.

References (standard): Bardeen, Cooper & Schrieffer, Phys. Rev. 108, 1175 (1957); de Gennes, Superconductivity of Metals and Alloys; Tinkham, Introduction to Superconductivity.

Reinforcement Learning for Generative Models

Supervised training of generative networks maximizes likelihood of observed samples. That objective is misaligned with many deployment goals — preference, safety, task success, or sample quality under a non-differentiable metric. Reinforcement learning (RL) treats the generator as a policy and optimizes expected reward, which is how modern LLM alignment (RLHF) and several image/video fine-tuning pipelines work.

Prerequisites: policy gradients / REINFORCE at a first-course level; autoregressive or diffusion generative models. Scope: how RL is used to train generative NNs (especially language models), not a full RL textbook. Actor–critic theory, offline RL, and multi-agent RL are out of scope except where they appear in practice.

Why leave maximum likelihood?

Maximum likelihood (MLE / cross-entropy) for a generative model $p_{\theta}$ fits the training distribution:

$$ \theta^\star = \arg\max_{\theta}\thinspace \mathbb{E}_{x \sim \mathcal{D}}\bigl[\log p_{\theta}(x)\bigr] $$

Problems that show up in practice:

RL reframes generation as sequential decision-making and optimizes $\mathbb{E}[R]$ under a reward that can be sparse, learned, or black-box.

Generation as an MDP

For an autoregressive model (LLM, pixelCNN-style decoder), one natural MDP is:

A trajectory is a full completion $y = (y_{1},\ldots,y_{T})$ given prompt $x$:

$$ \pi_{\theta}(y \mid x) = \prod_{t=1}^{T} \pi_{\theta}(y_{t} \mid x, y_{<t}) $$

The RL objective is expected reward (often with a KL penalty back to a reference policy $\pi_{\mathrm{ref}}$ — see below):

$$ J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\thinspace y \sim \pi_{\theta}(\cdot\mid x)}\bigl[R(x,y)\bigr] $$

Diffusion / flow models can be cast similarly (denoising steps as actions), but the dominant industrial use of RL for generative models is still token-level policies with sequence-level rewards. The rest of this note focuses on that case.

flowchart LR
    P["Prompt x"] --> S0["State: prefix"]
    S0 --> A["Sample token"]
    A --> S1["Append"]
    S1 --> A
    S1 --> Done{"EOS or max len?"}
    Done -->|No| A
    Done -->|Yes| R["Reward R(x,y)"]
    R --> Upd["Policy update"]

Policy gradient backbone

The score-function (REINFORCE) identity gives an unbiased gradient without differentiating through $R$:

$$ \nabla_{\theta} J(\theta) = \mathbb{E}_{y \sim \pi_{\theta}}\Bigl[ R(x,y)\thinspace \nabla_{\theta} \log \pi_{\theta}(y \mid x) \Bigr] $$

For autoregressive policies,

$$ \nabla_{\theta} \log \pi_{\theta}(y \mid x) = \sum_{t=1}^{T} \nabla_{\theta} \log \pi_{\theta}(y_{t} \mid x, y_{<t}) $$

so credit from a sequence-level reward is smeared across all tokens — high variance. Practical systems reduce variance with:

• Baselines $b(x)$: replace $R$ by $R - b$ (advantage-like)
• Learned critics $V_{\phi}(s_{t})$ (actor–critic / GAE as in PPO)
• Group comparisons of several samples for the same prompt (GRPO-style)

Importance sampling and clipping (PPO) keep updates from leaving the trust region of the data collected under an older policy $\pi_{\theta_{\mathrm{old}}}$.

The RLHF stack (language models)

RLHF (reinforcement learning from human feedback) is the canonical pipeline for aligning generative LLMs after pretraining.

flowchart TB
    PT["Pretrained LM"] --> SFT["SFT on demos"]
    SFT --> RM["Train reward model on preferences"]
    SFT --> RL["RL fine-tune policy"]
    RM --> RL
    RL --> Out["Aligned policy"]

1. Supervised fine-tuning (SFT)

Start from a pretrained LM; fine-tune on high-quality (prompt, response) demos. This yields $\pi_{\mathrm{SFT}}$ — a competent but not preference-optimized policy. It also becomes the usual reference $\pi_{\mathrm{ref}}$ for the KL term.

2. Reward model from preferences

Collect pairwise (or ranked) human comparisons: for prompt $x$, prefer $y_{w}$ over $y_{l}$. Fit a scalar reward model $r_{\phi}(x,y)$ under a Bradley–Terry likelihood:

$$ p(y_{w} \succ y_{l} \mid x) = \sigma\bigl(r_{\phi}(x,y_{w}) - r_{\phi}(x,y_{l})\bigr) $$

$r_{\phi}$ is typically another LM with a scalar head. Once trained, it stands in for the human as a dense-enough (still usually terminal) reward for RL.

3. RL fine-tuning with KL regularization

Optimize

$$ \begin{aligned} J(\theta) ={}& \mathbb{E}_{x,y \sim \pi_{\theta}}\bigl[r_{\phi}(x,y)\bigr] \\ &\quad - \beta\thinspace \mathbb{E}_{x}\bigl[D_{\mathrm{KL}}\bigl(\pi_{\theta}(\cdot\mid x)\Vert \pi_{\mathrm{ref}}(\cdot\mid x)\bigr)\bigr] \end{aligned} $$

The KL term is not optional decoration: without it the policy hacks $r_{\phi}$ (gibberish that scores high, length gaming, reward-model exploits) and drifts from fluent language. Equivalently one maximizes expected regularized reward

$$ R_{\mathrm{eff}}(x,y) = r_{\phi}(x,y) - \beta \log\frac{\pi_{\theta}(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)} $$

PPO has been the workhorse optimizer (clipped surrogate + value head). Alternatives that avoid an explicit online RL loop are discussed next.

Preference optimization without an RL loop

If the goal is “match a preference distribution under KL to $\pi_{\mathrm{ref}}$,” the optimal policy has a closed form in terms of the reward. Direct Preference Optimization (DPO) and relatives rearrange that so you never train $r_{\phi}$ or run PPO: you optimize a classification-style loss on preference pairs directly in policy space.

DPO is often enough for chat alignment; online RL (PPO/GRPO) still matters when the reward is verifiable (unit tests, math checkers, compilers) and you want the policy to explore beyond the preference dataset.

Verifiable rewards and “RLVR”

A growing pattern for reasoning / code models: skip the learned reward model and use ground-truth checkers — pass/fail on unit tests, exact match on math, format constraints. Call this RL with verifiable rewards (RLVR) informally.

• Reward is sparse and objective → less reward hacking of a learned $r_{\phi}$, but still hacking of the checker (hardcoded answers, trivial programs).
• Group-relative methods (sample $K$ completions, normalize advantages within the group) fit well: no value network required.
• Process rewards (step-level) vs outcome rewards (final answer) trade annotation cost against credit assignment.

This is closer to classical RL (environment gives $R$) than to preference modeling, but the policy is still a generative NN.

Image and other modalities (brief)

Same idea, different action space:

• RL fine-tuning of text-to-image: reward from human prefs, CLIP/aesthetic scorers, or downstream task metrics; policy may be the denoiser or a prompt/adapter.
• Discrete latent policies (VQ tokens) look like language-model RL again.
• Diffusion RL needs care: the generative process is a long denoising chain; many methods use truncated / single-step surrogates or optimize only late steps.

The conceptual map is unchanged: define $\pi_{\theta}$, define $R$, control divergence from a reference generator.

Failure modes worth remembering

Practical mental model

1. Pretrain / SFT → fluent generative prior $\pi_{\mathrm{ref}}$.
2. Define success → human prefs, learned $r_{\phi}$, or verifiable checker.
3. Optimize expected reward under a trust region (KL / PPO clip / DPO implicit constraint).
4. Evaluate on held-out preferences and real tasks — the training reward is a proxy.

RL does not replace representation learning in the base model; it steers an already-capable generator toward objectives that likelihood training cannot express.

Related notes

• RL for Neural Posterior Estimation — same RL toolkit on the outer loop of simulation-based inference (proposals, active simulation, calibration rewards), not chat alignment.
• Statistical estimation and model misspecification show up the same way in circuit modeling proxies — optimize the metric you mean, then distrust it (no dedicated page yet).
• Modified Nodal Analysis — unrelated domain, same habit: state the unknowns and the objective before choosing an optimizer.

References (standard): Sutton & Barto, Reinforcement Learning; Schulman et al., PPO; Christiano et al. / InstructGPT (RLHF); Rafailov et al., DPO.

RL for Neural Posterior Estimation

Simulation-based inference (SBI) approximates posteriors when the likelihood is intractable but a simulator can generate data. Neural Posterior Estimation (NPE) does that with a conditional generative model $q_{\phi}(\theta \mid x)$. Reinforcement learning enters when the scarce resource is not likelihood evaluations but simulations: which parameters to run next, how to adapt proposals across rounds, and how to push the generative estimator toward calibrated posteriors rather than a pure likelihood fit on prior-weighted data.

This note sits next to RL for generative models. That page is about RLHF-style fine-tuning of a generator (chat, images). Here the generator is the posterior approximator, and RL mostly acts on the outer loop — budget, proposals, active selection — not token-level preference.

Prerequisites: Bayes rule, basic importance sampling; familiarity with normalizing flows or another conditional density estimator helps. Scope: how RL-style ideas interface with NPE / SNPE pipelines. A full SBI survey, ABC, and neural likelihood estimation (NLE) are out of scope except as contrasts.

Why likelihood-free inference shows up

Bayes’ rule is not the bottleneck. For observed data $x_{o}$ and parameters $\theta$,

$$ p(\theta \mid x_{o}) \propto p(x_{o} \mid \theta)\thinspace p(\theta) $$

is fine when the likelihood $p(x_{o} \mid \theta)$ can be evaluated (or its gradient used in HMC). In a large class of scientific and engineering models it cannot. The map $\theta \mapsto x$ is defined by a simulator $g$: integrate ODEs/PDEs, run a SPICE transient, push particles through a transport code, ray-trace a detector. The simulator returns a sample $x \sim g(\theta)$ (possibly stochastic), not a density value $p(x \mid \theta)$.

That is the SBI setting: the likelihood is unavailable or too expensive to evaluate, but forward simulation is possible. The goal is still the posterior for one or many observations $x_{o}$.

Circuit and device work lives here constantly. Compact-model calibration, TCAD process corners, reliability aging models, and Monte Carlo over SPICE decks all produce $x$ from $\theta$ without a tractable $p(x \mid \theta)$. Molecular dynamics and mesoscopic transport codes are the same pattern: forward is natural, likelihood is not.

flowchart LR
    Prior["Prior p(theta)"] --> Sim["Simulator g(theta)"]
    Sim --> Pairs["Pairs (theta, x)"]
    Pairs --> NPE["Train q_phi(theta | x)"]
    Xo["Observed x_o"] --> NPE
    NPE --> Post["Approximate posterior"]

Classical workarounds — Approximate Bayesian Computation (ABC), synthetic likelihoods, hand-crafted summary statistics — struggle in moderate-to-high dimension and require careful tuning. Neural density estimation shifted the default: learn a flexible surrogate from simulated pairs $(\theta_{i}, x_{i})$ and amortize.

Neural Posterior Estimation

NPE trains a conditional generative model

$$ q_{\phi}(\theta \mid x) \approx p(\theta \mid x) $$

on pairs drawn by $\theta_{i} \sim p(\theta)$, $x_{i} \sim g(\theta_{i})$. The usual training objective is conditional maximum likelihood (or a flow / diffusion equivalent):

$$ \phi^\star = \arg\max_{\phi}\thinspace \mathbb{E}_{\theta \sim p(\theta),\thinspace x \sim g(\theta)}\bigl[\log q_{\phi}(\theta \mid x)\bigr] $$

The network behind $q_{\phi}$ has to represent a flexible family of conditional densities on $\theta$ given $x$. In practice that means a normalizing flow (coupling layers such as RealNVP, or autoregressive flows such as MAF / NSF), or a more recent generative backbone such as flow matching or a diffusion / score model trained to sample $\theta \mid x$. Flows give exact densities and were the NPE default for a long time; diffusion-style models trade that for expressivity in higher-dimensional $\theta$. The outer-loop RL discussion below does not depend on which backbone you pick — only that you can sample from $q_{\phi}(\cdot \mid x)$ (and, for some utilities, evaluate or approximate its density / entropy).

Once trained, inference for a new $x_{o}$ is a forward sampling (or density) call — amortized inference. Pay the simulation cost once; reuse $q_{\phi}$ across many observations.

NPE is attractive when many $x_{o}$ will appear (many dies, many devices, many events). NLE / NRE can be preferable when amortization is unnecessary and one wants standard MCMC diagnostics on a single observation. The RL discussion below focuses on NPE because the generative posterior model is explicit — but active simulation and proposal adaptation transfer to NLE/NRE with minor changes.

What “amortized” buys — and what it costs

Amortization means $q_{\phi}(\cdot \mid x)$ must be accurate across a distribution of $x$, not only at one $x_{o}$. Training therefore spreads simulations over the prior predictive. That is also the failure mode: if the prior is broad and the posterior for a given $x_{o}$ is a thin ridge, most prior draws never visit the relevant $\theta$ region. The density estimator sees almost no training signal where it will later be queried. Sequential methods exist to fix that; RL is one way to systematize the fix.

Sequential NPE and the proposal problem

Naive NPE samples $\theta$ from the prior. Sequential NPE (SNPE) runs in rounds. After round $r$, form a proposal $\tilde{p}_{r}(\theta)$ from the current approximate posterior (truncation to high-density regions, mixture proposals, etc.), simulate under that proposal, and update $q_{\phi}$. Simulations concentrate where they matter for the observation(s) of interest.

flowchart TB
    R0["Round 0: sample from prior"] --> Train0["Fit / update q_phi"]
    Train0 --> Prop["Build proposal from current posterior"]
    Prop --> R1["Simulate under proposal"]
    R1 --> Train0
    Train0 --> Done["Stop: budget or convergence"]

There is a catch. If you train $q_{\phi}$ with ordinary conditional MLE on samples from $\tilde{p}_{r}$ instead of $p(\theta)$, you are no longer targeting $p(\theta \mid x)$ — you are targeting a posterior under the wrong prior. SNPE algorithms correct for that with importance weights, atomic proposals, or carefully chosen losses so that the fixed point remains the true posterior. The details differ across SNPE-A/B/C; the design tension is shared:

• Propose narrowly enough to waste fewer simulations.
• Correct carefully enough that $q_{\phi}$ does not converge to the wrong conditional.

SNPE already feels like closed-loop experimental design: the next batch of $\theta$ depends on what you have learned. Heuristics (sample from current $q_{\phi}(\theta \mid x_{o})$, truncate the prior) work surprisingly well. They also leave an obvious opening: replace the heuristic with a learned policy that maximizes a utility tied to posterior quality per simulation.

Where the budget actually breaks

Before attaching RL, name the failure modes that motivate it.

Expensive simulators. Each $(\theta, x)$ pair may cost seconds to hours. A budget of $10^{3}$–$10^{5}$ simulations is not academic — it is the entire experiment. Methods that cut the required $N$ by a small factor are worth real engineering.

Prior-weighted waste. Under a diffuse prior, the fraction of draws that land near the posterior for a typical $x_{o}$ can be tiny. Conditional MLE then spends capacity modeling irrelevant regions of $\theta$-space. Sequential proposals help; poorly tuned proposals either stay too wide (waste) or collapse too early (miss modes, bad coverage).

Mis-calibration. $q_{\phi}$ can achieve low training NLL and still be overconfident: credible intervals that do not cover at the nominal rate. Simulation-based calibration (SBC), coverage tests, and classifier two-sample tests (C2ST) against a reference posterior catch this. Likelihood on the training pairs does not directly optimize those diagnostics.

High-dimensional $\theta$. Both proposal design and density estimation degrade. Local active selection (choose the next $\theta$ carefully) becomes more important than drawing large i.i.d. batches from a crude proposal.

RL does not replace the density estimator. It allocates simulations and sometimes retargets the training signal toward metrics that conditional MLE on the training set does not see.

Casting the outer loop as an MDP

Treat proposal / active selection as a Markov decision process. One useful reading:

A policy $\pi(a \mid s)$ that maximizes expected cumulative reward is an adaptive design for the simulation budget. Formally this is the same object as RL for Bayesian experimental design or Bayesian optimization, with a generative posterior model inside the loop instead of a Gaussian process.

When the action is a continuous proposal parameter and the utility is differentiable, pathwise gradients may suffice and one never needs a full RL stack. RL becomes natural when:

• the utility is non-differentiable (coverage tests, C2ST, downstream task loss through a discrete decision);
• the horizon is multi-step (several rounds of simulate → update → propose) with delayed reward;
• the action is a discrete choice among candidate $\theta$ or among simulator fidelities.

Score-function (REINFORCE) estimators — the same identity used in RL for generative models — already appear inside variational and adversarial SBI when a proposal or discriminator objective does not admit a pathwise gradient. The branding is not always “RL,” but the estimator is.

Active and sequential simulation selection

This is the closest match to “real RL” in current SBI practice, and the right place to start.

SNPE already adapts the proposal over rounds. Active Sequential NPE (ASNPE) and related methods go further: instead of only sampling from the current posterior approximation, they score candidate $\theta$ values by a utility — typically where an additional simulation is expected to reduce uncertainty the most — and prefer high-utility candidates. That is classical Bayesian optimal design, with $q_{\phi}$ (or an ensemble of NPEs) supplying the uncertainty model.

An RL-shaped version makes the utility maximization online and sequential. The state encodes the current posterior approximation; the action selects the next $\theta$ (or a mini-batch); the reward is the realized improvement after the simulation and update. Useful reward proxies:

Expected information gain. In experimental design one often maximizes mutual information between parameters and the future observation. A common one-step utility is the expected reduction in posterior entropy (or a KL between prior-for-the-step and posterior-for-the-step). With an NPE ensemble, disagreement across members is a cheap uncertainty proxy when analytic entropy is awkward.

Validation / held-out loss. After adding a batch, measure $-\mathcal{L}$ on held-out simulated pairs near the region of interest. Noisy, but directly tied to density-estimation quality.

Calibration and coverage. SBC-style scores or empirical coverage of credible intervals. Slow to estimate and high variance — treat as an occasional reward or a filter, not necessarily the every-step signal.

Distance to a reference posterior. When a trustworthy reference is available on benchmark tasks (reject/accept ABC with huge budget, or an analytic posterior), C2ST accuracy or MMD between $q_{\phi}(\cdot \mid x_{o})$ and the reference is a sharp utility. On real problems the reference is missing; use this for method development, not deployment.

The policy can be a simple parametric proposal $q_{\psi}(\theta \mid s)$ trained with policy gradients, or a bandit / Bayesian optimization layer over a finite candidate set. For expensive simulators, even a myopic one-step lookahead (pick $\arg\max$ utility, no long-horizon credit assignment) already beats prior sampling; multi-step RL is justified when early simulations should explore and later ones exploit, with a budget horizon baked into the state.

RL-style objectives on the generative posterior itself

A different lever: leave the proposal heuristic alone and change what $q_{\phi}$ is trained to optimize.

Conditional MLE on simulated pairs is the right fixed point for “match $p(\theta \mid x)$ under the training distribution.” It is the wrong objective when the deployed failure mode is mis-calibration or bad downstream decisions. One can fine-tune or regularize $q_{\phi}$ with rewards / penalties from:

• Posterior predictive checks — simulate $x’ \sim g(\theta’)$ for $\theta’ \sim q_{\phi}(\theta \mid x_{o})$ and score whether $x’$ looks like $x_{o}$ under chosen summaries.
• Simulation-based calibration — ranks of true $\theta$ under the inferred posterior should be uniform; deviations become a loss.
• Task loss — if $\theta$ feeds a controller, optimizer, or yield decision, differentiate (or REINFORCE) through that loss.

This is closer in spirit to RLHF on generative models: the base model is trained by likelihood, then steered by a reward that likelihood does not encode. The analogy is imperfect — here there is often a ground-truth notion of calibration — but the machinery overlaps. Adversarial and variational SBI methods that use score-function gradients sit in the same neighborhood.

Flow-matching and diffusion-based posterior estimators also sit next to diffusion / flow policies in offline RL. Same generative toolkit; different reward. That does not by itself give a method, but it explains why ideas cross-pollinate quickly.

Use this lever when the proposal is already decent and the density still looks sharp and wrong. More prior-weighted simulations will not fix a calibration bug that the training loss does not see.

Adaptive proposals and importance weights

Even without a full active-learning policy, the proposal $q(\theta)$ (or $q(\theta \mid x_{o})$) can be adapted so that importance-weighted NPE updates keep a healthy effective sample size. When ESS collapses, the sequential scheme is lying to itself: a few particles dominate, and the “posterior” is an artifact of weight degeneracy.

Adaptation rules range from simple (inflate proposal covariance, mixture with the prior) to learned (parametric $q_{\psi}$ updated from recent ESS / variance of weights). Bandit or RL updates are optional; what matters is closing the loop on a diagnostic (ESS, weight entropy) rather than only on training NLL.

A related thread is domain randomization and sim-to-real: treat simulator nuisance parameters as part of an outer adaptation loop so the amortized posterior transfers to real $x_{o}$. Neural posterior domain randomization is one named pattern; the RL connection is again the outer-loop policy over nuisances or fidelities.

Hybrid pipelines: NPE inside RL, RL around the simulator

Two composition directions are easy to confuse and worth separating.

NPE inside an RL agent. The agent must act under parameter uncertainty (robust control, Bayes-adaptive MDPs). Each episode or time step, observations update a belief; $q_{\phi}(\theta \mid x)$ supplies that belief in amortized form. Here SBI is a subroutine for the belief state. The RL problem is the original control problem; NPE just replaces an extended Kalman filter or a particle filter when the observation model is a heavy simulator.

RL around the simulator / SBI loop. The MDP is the inference pipeline: actions choose $\theta$, fidelities, or hyperparameters; rewards are posterior quality. This is the outer-loop setting emphasized above.

For circuit and reliability modeling the first pattern is natural: infer device or process parameters from measurements, then make a decision (binning, guard-banding, stress conditions) under the posterior. The second pattern is natural when the simulator is so expensive that designing the simulation campaign is itself an optimization problem.

Multi-fidelity variants blur the line: an action may choose a cheap approximate simulator versus a gold-standard one, with a reward that trades information against cost. That is standard in BO; embedding it around NPE is still under-explored relative to how often real pipelines have cheap/approximate modes.

Implementation sketch

A minimal outer-loop experiment looks like this.

The environment is the simulator $g$ plus the current NPE $q_{\phi}$. The state is a featurization of $q_{\phi}$ — moments or a particle set under $x_{o}$, predictive entropy, ensemble spread if you train several flows, and the remaining budget. The action is either a $\theta$ drawn from a parameterized proposal or a selection among candidates scored by a utility network. The reward should be chosen to match the scientific claim you want to make:

Stack. Use sbi (or an equivalent) for NPE/SNPE baselines so the density-estimation half is not a moving target. Put the outer policy in a standard RL library (Stable-Baselines3, RLlib) or, for myopic design, in a BO package. If $g$ is differentiable, try pathwise gradients through proposal parameters before REINFORCE — variance matters when each reward costs a simulation.

Benchmarking habit. On toy tasks (Gaussian linear, SLCP, two moons) one can afford a reference posterior and plot posterior quality against simulation count for: prior NPE, SNPE, ASNPE-style active selection, and any RL policy. Claim wins only on the budget–quality curve, not on a single-$N$ snapshot. Then move to a simulator you already trust in your domain (even a reduced SPICE or analytic surrogate) before touching production TCAD.

What is mature vs what is open

Active and sequential simulation for NPE is the mature-ish core: SNPE variants, ASNPE, round-free and pruning schemes that discard uninformative simulations. REINFORCE-style gradients appear inside variational and adversarial SBI as a technique, often without the RL label. End-to-end RL policies over NPE budgets — learned $\pi(a \mid s)$ with long-horizon credit assignment — are emerging in adjacent communities (physics, cosmology, policy learning with SBI beliefs) but are not a single standard recipe.

If it works, the payoff is concrete: fewer simulations for comparable posterior quality; better concentration on relevant $\theta$; room to optimize calibration and task loss, not only training NLL.

The hard parts are also concrete. Rewards that depend on posterior quality are noisy and often delayed across many simulator calls. The MDP is non-stationary because $q_{\phi}$ moves as data arrive. Density-estimation pathologies (mode collapse, overconfidence) interact with the outer loop: a collapsed $q_{\phi}$ produces overconfident states and a policy that stops exploring. Same Goodhart risk as other RL-for-design systems — optimize the proxy hard enough and the real posterior diagnostics get worse.

Mental model

The simulator is the oracle; the likelihood is unavailable. NPE amortizes $p(\theta \mid x)$ with a conditional generator trained on simulated pairs. SNPE and active learning already close the loop on where to simulate next; RL is the general language for that loop when the utility is sequential, non-differentiable, or multi-objective. Keep the density estimator honest with calibration checks, and do not let the outer reward redefine “good posterior” into “good at gaming the proxy.”

Relative to RL for generative models: there the policy is the generator and the reward is preference or task success; here the policy usually feeds the generator with better training data (or a better training objective), and the reward is inference quality per simulation.

Related notes

• Reinforcement Learning for Generative Models — policy gradients, KL trust regions, and rewards that are not likelihood (LLM / alignment setting).
• Modified Nodal Analysis — the kind of simulator stack where likelihood-free inference shows up in practice.

Pointers: Cranmer, Brehmer & Louppe on the SBI landscape; Papamakarios et al. on neural density estimation and NPE; Greenberg, Nonnenmacher & Macke on automatic posterior transformation (SNPE); the sbi package documentation; ASNPE and related active-simulation papers. Direct “RL + NPE” as a named standard method is still thin — treat this as a design pattern sitting on top of SNPE/active learning, not as a drop-in algorithm.