All pages on this site · 35 published · newest first
A minimal page published to confirm the publishing connector works end to end. Safe to delete.
A buy/hold/sell analysis of Marvell's Q2 FY2027 earnings: every line beat and the stock still fell 8%. Why flat gross margin guidance mattered more than the beat, what the Google warrant actually is, and where the valuation sits.
A visual walkthrough of InterFormer (arXiv:2411.09852), Meta's CTR architecture: two parallel arches that exchange summaries every layer instead of crushing the behavior sequence early. Covers the Interaction/Sequence/Cross arches, PFFN weight generation, the no-collapse shape invariant, an interactive step-through of Algorithm 1, and the communication/compute overlap behind the +24% QPS.
An animated atlas of preconditioning, Fisher information, and per-coordinate scale invariance — why Adam does not follow the gradient.
Every gradient in a sparse MoE layer, spelled out term by term on one worked numeric example — with animated diagrams of the gate dot-product, the top-k discontinuity, the softmax Jacobian redistribution, and the load-balancing loss.
An animated, VJP-first guide to backpropagation for GPU kernel writers, with properly typeset math: linear/matmul, broadcast-reduce duality, the softmax Jacobian collapse, top-k routing, and FlashAttention save-vs-recompute.
A working reference on CUDA kernel fusion: epilogue fusion with cuBLASLt and CUTLASS, hand-written WMMA epilogues, online softmax and FlashAttention, and where fusion stops paying.
One scalar decay becomes a diagonal matrix, so every channel of the state gets its own memory horizon. Plus the hybrid: three KDA layers to one full-attention layer, and why the ratio is not arbitrary.
Twenty-three macrocycles, Multi-head Latent Attention, 898 experts in a compressed latent space, an activation function that made inference 3x slower, and AttnRes — attention run along the depth axis instead of the token axis.
Gated DeltaNet composes Mamba-2's uniform decay with the delta rule's targeted overwrite. Two verbs a fixed memory needs — release and replace — and why neither is expressible in the other.
The WY reparameterisation turns every in-chunk delta correction into one triangular solve, capping sequential depth at C instead of L — and reveals that attention and recurrence are one computation at two settings of the chunk-size dial.
Applying the feature map to q and k separately makes attention re-associate into a fixed d×d state: constant memory and constant decode time. The price is a capacity ceiling of d clean associations, with readback SNR decaying as √(d/N).
The decoder-only baseline, what softmax attention actually computes, and why the KV cache turns cubic decoding into quadratic decoding while making memory bandwidth the bottleneck.
DeltaNet's update is a generalized Householder transform that erases exactly one key's slot and leaves every orthogonal direction untouched — and is identical to one SGD step per token on a self-posed regression problem.
Every mechanism in the series in one table, and the claim they support: a fixed-capacity associative memory needs an eviction policy, and attention is the best selective read we have — along the token axis, the depth axis and the parameter axis alike.
Eight chapters tracing language model architecture from GPT-2 to Kimi K3, told as one question: what does the model keep, and what does it discard to make room? With full derivations, cost models and original diagrams.
A technical walkthrough: why transformers parallelize over sequence positions, how sparse and dense features enter attention in recommendation models, and how Meta's HSTU / Generative Recommenders architecture handles retrieval and ranking. With worked numeric examples.
Chapter 5: what a GPU actually does with 4-bit weights — W4A16, nibble unpacking, the memory hierarchy, SIMD lookup tables, and the KV cache knock-on effect.
Chapter 4: Unsloth Dynamic quantization — the DeepSeek-R1 720GB to 131GB case study, why KL divergence beats perplexity, real Gemma 3 charts, and a bit-allocation sandbox.
Chapter 3: the importance matrix derivation, calibration-data pitfalls, and why sub-3-bit quantization needs vector codebooks — with an animated 2D demo.
Chapter 2: NF4 quantile quantization vs uniform INT4, and a byte-by-byte anatomy of a GGUF Q4_K superblock, plus the full format table.
Chapter 1: float bit layouts, the affine quantization equation, the transfer-function staircase, and why block-wise scaling is the fix for outliers.
Interactive 7-part series on LLM quantization: the memory wall, affine grids, NF4, GGUF k-quants, importance matrices, codebooks, Unsloth Dynamic 2.0, and GPU kernels.
An interactive explainer of the Newton–Schulz iteration Muon uses to orthogonalize gradient matrices — polar decomposition, why the iteration only touches singular values, cobweb plots, and why Muon's five magic coefficients are what they are.
Chapter 7: a decision table for picking a quant, rules of thumb, the exact llama.cpp commands to build and measure your own, and a summary of the whole series.
Chapter 6: GPTQ Hessian error correction, AWQ salient channels, quantization-aware training, BitNet native ternary, and how QLoRA fine-tunes on a frozen 4-bit base.
An interactive walkthrough of the attention stack in GLM-5 — MLA latent compression and the absorption trick, DeepSeek Sparse Attention's cheap indexer, how that indexer is distilled from dense attention, prefill vs decode economics, IndexShare, and the ablations Zhipu published before choosing sparse over linear.
An interactive walkthrough of KDA — the hybrid linear attention behind Kimi K3. Fixed-size memory, the delta rule as online gradient descent, a live measurement of how lossy the state actually is, channel-wise forgetting, the DPLR constraint that makes the kernel fast, and Attention Residuals.
How a scalar reward becomes a gradient update in RL: the log-derivative trick, why the direction comes from the policy and only the magnitude comes from the world, and what baselines, reward-to-go and PPO clipping are actually fixing. With an interactive probability simplex.
Every MoE technique that matters, from 1991 gating to DeepSeek-V4's aux-loss-free routing — with interactive routers, load-balancing sims, real model data, and the math explained from scratch.
A visual, math-first walkthrough of the decoder-only transformer: how attention moves information between tokens, and how each token's embedding is rewritten layer by layer.
Interactive, animated explainer of GPU warps: lockstep SIMT execution with a clock-pulse sweep, warp divergence, a clickable shuffle playground (broadcast/shift/XOR), and a step-through butterfly reduction.
40 years of Nasdaq-100 mid-election-year seasonality overlaid on one chart, the closest historical analog (1998 by character, 2018 by regime), and a reasoned month-by-month projection for the rest of 2026. Interactive charts + full data table.
Live FIFA World Cup 2026 results, fixtures and standings, paired with an honest, math-based betting strategy guide and interactive odds, value and stake calculators.
A short field guide to the octopus — a second, independent origin of mind on Earth. Distributed cognition, color-changing skin it can't see, taste-by-touch, self-editing genes, and only two years to use any of it.