# Stochastic Gradient Langevin Dynamics

> Stochastic Gradient Langevin Dynamics is the tool Singular Learning Theory uses to measure how "simple" a trained network is (the Local Learning Coefficient, LLC)

- Published: 2026-10-21
- Topics: mechanistic interpretability, singular learning theory, langevin dynamics
- Web: https://sadiq.tech/blog/sgld

Stochastic Gradient Langevin Dynamics is the tool Singular Learning Theory uses to measure how "simple" a trained network is (the Local Learning Coefficient, LLC) 
In one line it is: gradient descent (walk downhill on the loss) + random shaking (so the walker explores instead of stopping) + minibatches (use a small random slice of the data for each step). These notes answer two questions:
- Does SGLD actually do its job when the model is singular (has flat, degenerate minima, like neural networks)?
- The textbook theory says "not guaranteed", so why does it work in practice?

TLDR; answer: the idealized smooth version works fine. The trouble starts when a computer takes finite steps. Singular losses are very flat at the bottom but very steep far away, and on steep walls a fixed-size stepping rule overshoots and can fly off to infinity. The usual proofs assume walls are never that steep, so they don't cover the models SLT cares about
##### The Goal: Sampling
We have a probability distribution $\pi(x)$ (a rule saying how likely each point $x$ is; picture a landscape where tall places are likely and low places are unlikely) and we want to sample from it (pick random points so that, after many picks, the pile of points has the same shape as the landscape). For a neural network a "point" $x$ is one full set of weights, so the landscape lives in a space with millions of directions

_Score_: Every method here only needs $\nabla \log \pi(x)$, called the score (an arrow at each point that points uphill on the landscape, longer where the landscape is steeper). Think of it as "which way is more likely, and how strongly"

**Q:** Why use the uphill arrow instead of the landscape height itself ?  
**A:** Because we usually don't know the true height. We only know the landscape up to an unknown scaling number (the normalizing constant, which makes all probabilities add up to 1, and is impossible to compute for big models). Scaling a landscape doesn't change which way is uphill, so the arrow can be computed exactly even though the height can't. That's the main reason these methods are popular

_Tempered posterior_: The landscape SLT wants to sample is built from the training loss: $$\pi(w) \propto \varphi(w), e^{-\beta n L_n(w)}$$
- $w$ = the weights
- $L_n(w)$ = the training loss (average error on the $n$ data points)
- $e^{-\beta n L_n(w)}$ = turns loss into likelihood: low loss gives a tall landscape, high loss a low one
- $\varphi(w)$ = prior (a mild preference about the weights before seeing any data)
- $\beta$ = temperature knob; smaller $\beta$ flattens the landscape (more wandering), larger $\beta$ sharpens it (sticks close to the best weights)
- $\propto$ = "equal up to that unknown scaling number"

The uphill arrow of this landscape is just the negative loss gradient (scaled by $\beta n$) plus a small pull from the prior. So "walk uphill on the landscape" is the same thing as "walk downhill on the loss"

_Why SLT needs samples_: To measure the LLC at a trained set of weights $w^{\star}$, you let the weights wander randomly around $w^{\star}$ and ask how much the loss goes up on average:
$$
\hat{\lambda}(w^{\star}) = n\beta \left( \text{average loss while wandering} - L_n(w^{\star}) \right)
$$
- If the minimum is flat (singular), wandering barely raises the loss, so the LLC is small, meaning the solution is "simpler"
- A leash (called localization, strength $\gamma$) keeps the wandering near $w^*$ so we measure this valley and not some other one
- The wandering is done with SGLD. If SGLD doesn't sample correctly, the LLC number can't be trusted. That's why this question matters
##### Langevin Dynamics: A Marble That Rolls and Shakes
Picture the loss as a hilly surface and a marble on it. Two things act on the marble:
- Rolling: it slides downhill on the loss (uphill on the probability landscape)
- Shaking: the surface is constantly jiggled at random

Rolling alone, the marble gets stuck at the bottom of one valley (that's plain optimization). Shaking alone, it drifts away forever. With both, it keeps moving, but spends more time in low-loss places and less time in high-loss places, in exactly the right proportions. Recording where it goes gives you samples of $\pi$. This rolling-plus-shaking motion is called Langevin dynamics

_Brownian motion_: The mathematical name for the random shaking (like a speck of dust kicked around by air molecules). One rule about it matters for everything later: over a short time $t$, the random kick has typical size $\sqrt{t}$, not $t$. For small times, $\sqrt{t}$ is much bigger than $t$ (e.g. $t = 0.01$ gives $\sqrt{t} = 0.1$), so over short times the noise is much bigger than the rolling

_The Langevin equation_: Written as an update over a tiny time $dt$: $$dX_t = \tfrac{1}{2}\nabla \log \pi(X_t), dt + dW_t$$
- $X_t$ = position of the marble at time $t$
- $\frac{1}{2}\nabla \log \pi, dt$ = the roll (move along the uphill arrow, scaled by the time step)
- $dW_t$ = the shake (a random kick of size about $\sqrt{dt}$)
- The $\frac{1}{2}$ is the exact amount of rolling that balances this amount of shaking. More rolling and the marble would hug the peaks too tightly; less and it would spread out too much

Releasing many marbles at the same spot shows what "sampling" means: each marble keeps moving forever, but the shape of the crowd settles into the target landscape and then stops changing
```manim
from manim import *
import numpy as np

class MarblesSettle(Scene):
    def construct(self):
        rng = np.random.default_rng(0)
        log_pi = lambda x: -(x**2 - 1) ** 2
        slope = lambda x: -4 * x * (x**2 - 1)
        grid = np.linspace(-3, 3, 2001)
        Z = np.sum(np.exp(log_pi(grid))) * (grid[1] - grid[0])

        axes = Axes(x_range=[-2.5, 2.5, 0.5], y_range=[0, 1.2, 0.2],
                    x_length=11, y_length=4.6, tips=False).to_edge(DOWN, buff=0.5)
        target = axes.plot(lambda x: np.exp(log_pi(x)) / Z, x_range=[-2.3, 2.3], color=YELLOW)
        title = Text("600 marbles start on the far right, then roll + shake", font_size=30).to_edge(UP)
        legend = VGroup(
            Text("yellow curve = target distribution", font_size=22, color=YELLOW),
            Text("blue bars = where the marbles are now", font_size=22, color=BLUE),
        ).arrange(DOWN, aligned_edge=LEFT).next_to(title, DOWN)

        N, eps = 600, 0.01
        X = rng.uniform(1.8, 2.2, N)

        def bars_for(X):
            counts, edges = np.histogram(X, bins=40, range=(-2.5, 2.5), density=True)
            g = VGroup()
            for c, a, b in zip(counts, edges[:-1], edges[1:]):
                if c <= 1e-6:
                    continue
                c = min(c, 1.2)
                p0, p1 = axes.c2p(a, 0), axes.c2p(b, c)
                g.add(Rectangle(width=p1[0] - p0[0], height=p1[1] - p0[1],
                                fill_color=BLUE, fill_opacity=0.6, stroke_width=0.5)
                      .move_to((p0 + p1) / 2))
            return g

        bars = bars_for(X)
        clock = Text("time 0.0", font_size=26).to_corner(UR).shift(DOWN * 1.6)
        self.add(axes, title, legend, target, bars, clock)
        self.wait(0.5)
        t = 0.0
        for _ in range(60):
            for _ in range(20):
                X = X + 0.5 * eps * slope(X) + np.sqrt(eps) * rng.standard_normal(N)
            t += 20 * eps
            self.play(Transform(bars, bars_for(X)),
                      Transform(clock, Text(f"time {t:.1f}", font_size=26).move_to(clock)),
                      run_time=0.12, rate_func=linear)
        self.wait(1)
```
_Stationary distribution_: The shape the crowd settles into and keeps forever (individual marbles move, the overall shape doesn't). Sampling methods like this one (called MCMC, Markov chain Monte Carlo) work by designing a random motion whose stationary distribution is the target, then running it for a long time

_Markov_: The marble's next move depends only on where it is now, not on how it got there (no memory). This is what makes the math manageable

**Q:** Does the smooth, continuous version have any trouble with singular landscapes ?  
**A:** No. As long as the landscape doesn't push marbles outward in some crazy way, the continuous motion always settles into the target, even for very flat-bottomed, steep-walled landscapes. Steep walls that push inward are fine. All the problems below come from the computer taking finite steps
##### Taking Steps on a Computer (ULA)
A computer can't move the marble continuously; it jumps in steps. Each step is a small roll plus a random kick: $$x_{k+1} = x_k + \frac{\epsilon}{2}\nabla \log \pi(x_k) + \sqrt{\epsilon},\eta_k$$
- $x_k$ = position after $k$ steps
- $\epsilon$ = step size (how much time each jump represents)
- $\frac{\epsilon}{2}\nabla \log \pi(x_k)$ = the roll: move along the uphill arrow, proportional to how steep it is here
- $\eta_k$ = a fresh random number from a bell curve (normal distribution) each step
- $\sqrt{\epsilon},\eta_k$ = the kick, sized by the square-root rule

This is called the Unadjusted Langevin Algorithm (ULA) ("unadjusted" because nothing checks whether a step was sensible). SGLD is exactly this, except the uphill arrow is estimated from a small batch of data

Stepping introduces a small error: even when things go well, the crowd settles into a shape slightly different from the target. Smaller steps mean less error but slower exploration

**Q:** Why would stepping break something that works when moving smoothly ?  
**A:** Smooth motion reacts to the slope continuously, so it can't overshoot. A step looks at the slope once and then commits to a jump of that size. If the slope is huge, the jump is huge, and it can land far past where it should
##### Where Stepping Breaks: Steep Walls
Imagine standing on a wall of a valley with the rule "jump toward the bottom, by an amount proportional to how steep it is here". On a gentle bowl this works. On a very steep wall, the jump is so big that you sail over the bottom and land on the opposite wall, higher than where you started. That wall is even steeper, so the next jump is even bigger. You bounce higher and higher and fly off to infinity

_Bowl vs steep walls_:
- Bowl-shaped loss ($x^2$, a bell-curve landscape): steepness grows in proportion to distance. Jump size grows at the same rate as the distance, so with a small enough step size the jumps never overshoot. Safe
- Steep-walled loss ($x^4$): steepness grows like distance cubed. Far enough out, the jump is always bigger than the distance to the bottom, no matter how small the step size is. Not safe

_Worked example_: Loss $x^4$ (landscape $e^{-x^4}$), step size $\epsilon = 0.05$. Ignoring the random kick, one step is $x \to x - 2\epsilon x^3 = x - 0.1x^3$. Beyond $|x| = 1/\sqrt{\epsilon} \approx 4.47$ (the danger zone), every step overshoots to a bigger number on the other side
- Start at 4.0 (just inside): $4 \to -2.4 \to -1.02 \to -0.91 \to \dots$, settles toward the bottom
- Start at 4.6 (just outside): $4.6 \to -5.13 \to 8.40 \to -50.8 \to 13{,}000 \to \dots$, explodes
- With the random kicks switched back on, a chain sitting near the bottom will sooner or later get kicked into the danger zone, and then it explodes

The two starting points side by side:
```manim
from manim import *
import numpy as np

class OvershootOnSteepWalls(Scene):
    def construct(self):
        eps, R = 0.05, 10
        step = lambda x: x - 2 * eps * x**3
        danger = 1 / np.sqrt(eps)

        line = NumberLine(x_range=[-R, R, 2], length=12, include_numbers=False).shift(DOWN * 0.5)
        ticks = VGroup(*[Text(str(v), font_size=20).next_to(line.n2p(v), DOWN)
                         for v in range(-R, R + 1, 2)])
        zones = VGroup(
            Rectangle(width=line.n2p(R)[0] - line.n2p(danger)[0], height=0.6,
                      fill_color=RED, fill_opacity=0.25, stroke_width=0)
            .move_to((line.n2p(danger) + line.n2p(R)) / 2),
            Rectangle(width=line.n2p(-danger)[0] - line.n2p(-R)[0], height=0.6,
                      fill_color=RED, fill_opacity=0.25, stroke_width=0)
            .move_to((line.n2p(-danger) + line.n2p(-R)) / 2),
        )
        title = Text("Landscape exp(-x^4), step size 0.05", font_size=30).to_edge(UP)
        zone_lab = Text("red zone: past 4.47 every step overshoots", font_size=24, color=RED)\
            .next_to(title, DOWN)
        self.play(FadeIn(title), Create(line), FadeIn(ticks), FadeIn(zones), FadeIn(zone_lab))

        def run(x0, color, y_shift, name):
            dot = Dot(line.n2p(x0), color=color, radius=0.12).shift(UP * y_shift)
            lab = Text(f"{name}: {x0:.2f}", font_size=24, color=color)\
                .to_edge(LEFT).shift(UP * (y_shift + 1.2))
            self.play(FadeIn(dot), FadeIn(lab))
            x = x0
            for _ in range(5):
                x = step(x)
                new_lab = Text(f"{name}: {x:.2f}", font_size=24, color=color).move_to(lab, aligned_edge=LEFT)
                if abs(x) > R:
                    target = line.n2p(np.sign(x) * R) + UP * y_shift
                    self.play(dot.animate.move_to(target), Transform(lab, new_lab), run_time=0.8)
                    self.play(FadeIn(Text("flies off the line: next steps 13,000, then -221 billion",
                                          font_size=24, color=color).next_to(line, DOWN, buff=1.0)))
                    break
                self.play(dot.animate.move_to(line.n2p(x) + UP * y_shift),
                          Transform(lab, new_lab), run_time=0.8)

        run(4.0, GREEN, 0.6, "start 4.0")
        run(4.6, ORANGE, -0.0, "start 4.6")
        self.wait(2)
```
_Why this is about singular models_: A singular minimum is one where the bottom is extra flat, like $x^4$ (whose curvature at the bottom is zero) instead of $x^2$. The same high power that makes the bottom flat also makes the walls far away extremely steep. In SLT terms, $x^4$ has learning coefficient $\frac{1}{4}$, compared with $\frac{1}{2}$ for an ordinary $x^2$ minimum

**Q:** So is the flat bottom the problem ?  
**A:** No, the steep walls are. The flat bottom is harmless for sampling. It's just that for these losses, flat bottoms and steep walls come as a package. In general, whenever the steepness grows faster than the distance, plain step-by-step simulation is known to blow up (a standard result about simulating random motion on a computer)
- Fix 1, check each step (MALA): Treat each jump as a proposal and accept it only if it lands somewhere reasonably likely; otherwise stay put. Crazy overshoots get rejected, so nothing explodes. The catch: checking needs the loss on the entire dataset every step, which is too slow for big models
- Fix 2, cap the jump (taming): Shrink the roll whenever it would be huge, so a single step can never be longer than about 1. Small rolls are almost unchanged; giant ones are limited, so the bouncing can't build up. Tamed versions of these samplers come with guarantees even for steep-walled landscapes
##### SGLD: The Same Steps With a Cheap Slope
Computing the uphill arrow exactly needs the gradient on all $n$ data points, every step. SGLD instead uses a small random batch of $m$ points and scales it up: $$g(w) = \nabla \log \varphi(w) + \beta \frac{n}{m}\sum_{\text{batch}} \nabla \log p(\text{data point} \mid w)$$
- The sum over the batch is like the full sum but over $m$ points instead of $n$, so multiplying by $\frac{n}{m}$ brings it back to full size
- Each batch gives a slightly wrong arrow, but on average it is exactly right (unbiased)
- The step is then the ULA step with $g$ in place of the true arrow

**Q:** Now there are two kinds of noise: the batch error and the random kick. Doesn't the batch error mess things up ?  
**A:** Not much, if steps are small. The batch error gets multiplied by the step size $\epsilon$, while the kick has size $\sqrt{\epsilon}$. For a small step like $\epsilon = 0.0001$, that's $0.0001$ versus $0.01$, so the deliberate kick is about 100 times bigger and the batch error barely matters

_Step size_:
- In theory: shrink the step size over time (so the stepping error disappears), but slowly enough that the total amount of simulated time still grows without limit (so the chain keeps exploring)
- In practice: one fixed small step for a few hundred to a few thousand steps. The result is slightly off from the true target, but close
- Smarter variants give each weight its own step size based on how big its gradients have been, the same idea as the Adam optimizer. They are much less fussy about choosing the step size
##### What the Theory Guarantees and Why It Doesn't Cover Singular Models
The main proof that SGLD works needs the landscape to pass a "bowl test": the steepness may grow at most as fast as in a simple bowl (in proportion to distance). Put differently, the landscape's tails must fade out no faster than a bell curve's
- Bell-curve landscape ($x^2$ loss): passes
- $x^4$ loss: steepness grows like distance cubed, fails
- Raising the allowed limit doesn't help: it only pushes the point of failure further out

```manim
from manim import *
import numpy as np

class SteepnessVsAllowed(Scene):
    def construct(self):
        axes = Axes(x_range=[0, 3, 0.5], y_range=[0, 150, 30], x_length=8, y_length=5,
                    tips=False).to_edge(LEFT, buff=1.0).shift(DOWN * 0.3)
        xl = Text("distance from the centre", font_size=22).next_to(axes.x_axis, DOWN)
        yl = Text("steepness", font_size=22).rotate(PI / 2).next_to(axes.y_axis, LEFT)
        title = Text("The proof only allows steepness to grow like a bowl", font_size=30).to_edge(UP)
        bell = axes.plot(lambda x: 4 * x**2, x_range=[0, 3], color=GREEN)
        quart = axes.plot(lambda x: 16 * x**6, x_range=[0, 1.452], color=ORANGE)
        C = ValueTracker(8)
        limit = always_redraw(lambda: axes.plot(
            lambda x: C.get_value() * (1 + x**2),
            x_range=[0, min(3, np.sqrt(150 / C.get_value() - 1))], color=BLUE))

        def crossing():
            c = C.get_value()
            r = np.roots([16, 0, -c, -c])
            u = max(z.real for z in r if abs(z.imag) < 1e-9 and z.real > 0)
            return np.sqrt(u)

        mark = always_redraw(lambda: DashedLine(axes.c2p(crossing(), 0),
                                                axes.c2p(crossing(), 150), color=RED))
        legend = VGroup(
            Text("bell-curve landscape", font_size=24, color=GREEN),
            Text("x^4 landscape (singular)", font_size=24, color=ORANGE),
            Text("allowed limit", font_size=24, color=BLUE),
            Text("red: limit broken from here", font_size=24, color=RED),
            Text("raising the limit only moves", font_size=22),
            Text("the red line, never removes it", font_size=22),
        ).arrange(DOWN, aligned_edge=LEFT).to_edge(RIGHT)

        self.play(FadeIn(title), Create(axes), FadeIn(xl), FadeIn(yl))
        self.play(Create(bell), FadeIn(legend[0]))
        self.play(Create(limit), FadeIn(legend[2]))
        self.play(Create(quart), FadeIn(legend[1]))
        self.play(Create(mark), FadeIn(legend[3]))
        self.play(C.animate.set_value(25), run_time=3)
        self.play(C.animate.set_value(40), run_time=3)
        self.play(FadeIn(legend[4]), FadeIn(legend[5]))
        self.wait(2)
```
_Deep linear networks fail the test too_: A deep linear network is a stack of weight matrices multiplied together with no activation functions in between. It's the standard test model for LLC estimation because its true LLC is known exactly. With $L$ layers, the output multiplies $L$ weights together, so the loss behaves like a power of the weights of degree $2L$ (degree 4 for 2 layers, 6 for 3 layers), and its steepness grows like distance to the power $2L - 1$. Only $L = 1$ (plain linear regression, steepness proportional to distance) passes the bowl test

**Q:** What does the smallest example look like ?  
**A:** Two layers with one weight each: loss $(w_1 w_2 - c)^2$. Every pair with $w_1 w_2 = c$ is a perfect solution, so the minimum is a whole curved valley rather than a single point (that's the singular part). But walk out along $w_1 = w_2 = t$ and the steepness grows like $t^3$. Flat along the valley floor, very steep far away
##### So Why Does It Work in Practice?
LLC estimates made with SGLD on deep linear networks match the known true values, even for networks with around 100 million weights. They are not perfect: in large studies some runs do fail because chains blow up. Three possible explanations:
1. The walls aren't really that steep: Unlikely. For deep linear networks the loss is exactly a high-degree polynomial, so the walls definitely are steep
2. Batch noise acts as a brake: Maybe, out on the steep walls, the batch estimate of the slope tends to come out smaller than the true slope, which would shrink exactly the jumps that cause overshoot (like automatic taming). Gradient noise in real networks is known to have occasional extreme values, but whether it behaves like this brake hasn't been shown. Still an open idea
3. Chains do blow up, but only after a very long time: Getting kicked into the danger zone can be extremely rare. If it takes far longer than the few thousand steps used for an LLC estimate, the estimate is fine even though the chain would eventually explode. Running the $x^4$ example from the bottom, 10 times each:
	- Step size 0.25: blows up after about 760 steps (typical)
	- Step size 0.20: about 24,000 steps
	- Step size 0.15: most chains still fine after 1,000,000 steps

A small decrease in step size makes the blow-up enormously rarer, because the danger zone moves out to places the chain almost never visits. A single chain shows the pattern: calm, normal-looking sampling for hundreds of steps, then one unlucky kick and it's gone
```manim
from manim import *
import numpy as np

class LongCalmThenBlowUp(Scene):
    def construct(self):
        eps = 0.25
        rng = np.random.default_rng(3)
        xs = [0.0]
        while abs(xs[-1]) < 1e6 and len(xs) < 20000:
            x = xs[-1]
            xs.append(x - 2 * eps * x**3 + np.sqrt(eps) * rng.standard_normal())
        blow = len(xs) - 1
        cut = next(i for i, v in enumerate(xs) if abs(v) > 3)

        axes = Axes(x_range=[0, cut + 40, 100], y_range=[-3, 3, 1], x_length=11, y_length=4.2,
                    tips=False).shift(DOWN * 0.7)
        xl = Text("step number", font_size=22).next_to(axes, DOWN, buff=0.15)
        title = Text("One chain on exp(-x^4), step size 0.25", font_size=30).to_edge(UP)
        danger = 1 / np.sqrt(eps)
        lines = VGroup(*[DashedLine(axes.c2p(0, s * danger), axes.c2p(cut + 40, s * danger), color=RED_C)
                         for s in (-1, 1)])
        lab = Text("danger lines", font_size=22, color=RED_C).next_to(lines[1], UP, buff=0.05).align_to(lines[1], LEFT)

        pts = [axes.c2p(i, xs[i]) for i in range(cut)]
        pts.append(axes.c2p(cut, float(np.sign(xs[cut]) * 3)))
        path = VMobject(color=BLUE, stroke_width=1.6).set_points_as_corners(pts)

        self.play(FadeIn(title), Create(axes), FadeIn(xl))
        self.play(Create(lines), FadeIn(lab))
        self.play(Create(path), run_time=7, rate_func=linear)
        sgn = float(np.sign(xs[cut]))
        boom = Arrow(axes.c2p(cut, sgn * 2.2), axes.c2p(cut, sgn * 3.4), buff=0, color=ORANGE)
        boom_lab = Text(f"blows up at step {blow}", font_size=26, color=ORANGE)\
            .next_to(boom, LEFT)
        self.play(GrowArrow(boom), FadeIn(boom_lab))
        note = Text("calm for hundreds of steps, then one unlucky kick and it is gone",
                    font_size=24).next_to(title, DOWN)
        self.play(FadeIn(note))
        self.wait(2)
```
There is a branch of probability (quasi-stationary distributions) that studies exactly this "how does it behave before it escapes" situation, and it is the natural tool for making explanation 3 precise

_A bonus effect_: Studies of noisy gradient descent with small steps find that the noise gently pushes the weights toward the most singular solutions (the places where many valleys of perfect solutions meet). That matches SLT's prediction that the posterior concentrates on the most singular solutions, and is one suggested reason noisy training finds solutions that generalize. So steep walls make stepping fragile, while the flat singular bottoms shape where the chain likes to spend its time
##### Practical Tips for LLC Estimation
The three main knobs are the step size $\epsilon$, the temperature ($n\beta$) and the leash strength $\gamma$
- Plot the loss along each chain (the loss trace). A healthy chain rises from the starting loss and then levels off into a noisy flat line; the LLC uses that flat part
- Throw away the early part of each chain (burn-in), before it levels off
- Loss suddenly shooting up or turning into NaN (not a number, i.e. the computer overflowed) = the overshoot blow-up. Lower the step size and run the chain longer
- Negative LLC = the chain found lower loss than the starting weights, so $w^*$ wasn't really at the bottom of its valley. Smaller steps, shorter chains or a tighter leash help
- Leash too tight = samples stop reflecting the shape of the valley. Moderate values (around 1 to 10) are a common start
- Step size too small = the chain barely moves and the LLC comes out near 0
- Try several step sizes and temperatures and pick a range where the LLC doesn't change much; keep these settings fixed when comparing LLCs
- Run several chains (4 to 8) to spot a single bad one
- Adam-style samplers make the step size much easier to choose
##### SGLD as an Optimizer
Other papers prove that SGLD finds the best solution or escapes bad ones. They don't contradict any of this, because they assume away the problem: minima with no flat directions, or steepness that never grows faster than a bowl. Singular models break exactly those assumptions

**Q:** So what's the overall lesson ?  
**A:** Almost every guarantee for step-by-step Langevin assumes walls no steeper than a bowl. Singular models violate that, not at their minimum but far away from it. SGLD works in practice mostly because blow-ups are rare compared with the short chains used, helped by small steps, the leash and Adam-style step sizes. Whether batch noise also helps is still open
