01 — the job
Two kinds of evidence about one person
Click-through-rate prediction asks a narrow question: given this user and this ad, what's the probability of a click? The interesting part is that the evidence arrives in two incompatible shapes.
The paper's example is a nice one: your profile says you're into electronics in general, but the last twenty things you browsed were all phones. Neither signal is enough alone. The profile is stable but vague; the history is specific but noisy.
02 — the diagnosis
Two failures the authors name
InterFormer is basically a reaction to two habits in earlier sequential CTR models. Understanding them is most of understanding the architecture, because every piece exists to undo one of them.
Failure one: the information only flows one way
The standard pattern is to use the static features to steer sequence modeling — the target ad becomes a query, attention pulls out the relevant slice of history. Useful. But the reverse never happens: what the model learned from the sequence never gets folded back into the feature-interaction side. The static branch keeps computing crosses between age and category as though the last hour of browsing didn't exist.
Failure two: the sequence gets crushed before anything can use it
Because attending over hundreds of features and a thousand events is expensive, models usually compress the sequence early — sum it, pool it, run it through an MLP — and hand a single vector to the interaction module. Everything downstream sees that one vector. Whatever the pooling threw away is gone before the model had the context to know what mattered.
03 — getting to the starting line
Everything becomes a d-dimensional token
Before any of the clever machinery runs, both modes get flattened into the same currency: matrices of d-dimensional column vectors. This is what makes it possible to concatenate a sequence summary onto a feature matrix later without any special casing.
d-tall matrix — so downstream code only ever handles a single sequence.04 — the block
Three arches, repeated L times
One InterFormer layer has three named parts. Two of them are the load-bearing columns, one is the span between them. They all run inside the same layer, and the layer stacks.
Interaction Arch
Feature crossing over the static side. Whatever backbone you like — dot product, DCNv2, DHEN. Takes the sequence summary as extra input columns.
out: behavior-aware X
Sequence Arch
Transformer-flavored sequence modeling. Multi-head attention with rotary position embeddings, preceded by a feed-forward whose weights come from the static side.
out: context-aware S
Cross Arch
The span. Compresses each side into a handful of tokens and hands them across. Kept separate precisely so the two columns never have to shrink.
out: X_sum, S_sum
Interaction Arch — feature crossing, told what just happened
The trick is almost anticlimactic: take the sequence summary and glue it on as extra columns of the feature matrix, then run your normal interaction module over the combined thing.
Because the sequence summary is now just more columns, the interaction module produces three families of crosses for free: static×static (the usual explicit interests), static×sequence (does this ad match what they're doing right now), and sequence×sequence (which of the summary tokens are actually active — low-scoring ones fade out). The trailing MLP restores the original column count so the layer can stack.
Backbone-agnostic by design. The experiments swap in dot product, DCNv2, and DHEN and the interleaving gains hold across all three. In the reported runs, DHEN is the one used.
Sequence Arch — attention, told who this person is
Two moves. First a Personalized FFN, and this is the part worth slowing down on, because it isn't the usual "concatenate the context and hope."
PFFN(X_sum, S) = f(X_sum) · S, where f is an MLP producing a d×d matrix. The static context doesn't get appended to the sequence — it becomes the weights that process the sequence.Then ordinary multi-head attention runs over the result, with rotary position embeddings so ordering survives. And at the very first layer only, the non-sequence summary is prepended to the sequence as CLS tokens — four of them, in the reported configuration. Those CLS slots start out as "who this user is," and attention lets them absorb the sequence around them. On the way out they're the natural place to read a context-aware sequence summary from.
Cross Arch — the only place anything shrinks
Both columns keep their full width, which is exactly why they can't talk to each other directly: too noisy, too big. The Cross Arch is the narrow channel between them, and it's deliberately built as a separate structure so that compression is a read, never a write.
05 — one layer, step by step
Watch a single block execute
This is the paper's Algorithm 1, one line at a time. The order matters: summaries are computed first, from last layer's outputs, and then both arches consume them in parallel.
Use ← and → once the walkthrough has focus.
06 — the invariant
Follow the shapes and the design explains itself
Everything above can be checked against one table. Both trunks preserve their dimensions end to end; only the side channels are small. That property is what lets you stack layers at all, and it's the mechanical form of "no aggressive aggregation."
| Tensor | Shape | Where it lives | Fate |
|---|---|---|---|
| X⁽ˡ⁾ | d × n | Interaction Arch trunk | preserved through all L layers |
| S⁽ˡ⁾ | d × (4+T) | Sequence Arch trunk | preserved through all L layers |
| X_sum⁽ˡ⁾ | d × n_sum, n_sum ≪ n | Cross Arch, recomputed each layer | consumed by PFFN |
| S_sum⁽ˡ⁾ | d × 8 (4 cls + 2 pma + 2 recent) | Cross Arch, recomputed each layer | concatenated onto X |
| ŷ | scalar | head, after layer L | the click probability |
| for contrast — a typical earlier model: | |||
| S | d × T → d × 1 | collapsed at layer 0 | the rest of the network never sees T again |
The paper tests this directly: replace selective aggregation with average pooling, then an MLP, then multi-head attention, each feeding a DHEN interaction module. Performance improves as the compression gets gentler, and InterFormer's arrangement — where the trunk isn't compressed at all — comes out ahead of all three.
07 — depth
Each layer looks at a different time scale
Since the sequence keeps its full length, every layer's attention gets a fresh crack at it, informed by a static summary that has itself been revised. The published attention maps show the layers specializing.
On the internal dataset this shows up as clean depth scaling. Going from one layer to two is the big jump — around 0.13% normalized entropy — with a third layer adding roughly 0.05% and a fourth about 0.04% on top.
08 — model–system co-design
The two arches have opposite bottlenecks, which turns out to be lucky
This is the part of the paper that's least about machine learning and most about why the architecture is shaped this way. Running the arches side by side isn't just a modeling choice — it's what makes the hardware behave.
Interaction Arch (DHEN)
Lots of parameters, comparatively little arithmetic. Under sharded data-parallel training, it spends its time waiting on network — gathering and scattering weights between GPUs.
communication-bound
Sequence Arch (Transformer)
Few parameters relative to the FLOPs it burns. It spends its time in the matrix units, with almost nothing to synchronize.
compute-bound
Run them one after the other and you pay for both serially. Run them concurrently — which the architecture permits, because neither arch consumes the other's output within a layer, only the previous layer's summaries — and the Interaction Arch's network stalls hide underneath the Sequence Arch's math.
09 — does it work
Numbers
CTR metrics move in small absolute increments and the field cares about the third decimal place, because at Meta's volume a hundredth of a percent is real money. Read these as basis points, not as percentages.
Public benchmarks — AUC, higher is better
| Model | Amazon | TaobaoAds | KuaiVideo |
|---|---|---|---|
| non-sequential | |||
| DCNv2 | 0.8807 | 0.6472 | 0.7426 |
| DHEN | 0.8790 | 0.6509 | 0.7424 |
| Wukong | 0.8765 | 0.6478 | 0.7423 |
| sequential | |||
| DIN | 0.8848 | 0.6507 | 0.7437 |
| DIEN | 0.8856 | 0.6519 | 0.7451 |
| TransAct | 0.8851 | 0.6498 | 0.7448 |
| InterFormer | 0.8865 | 0.6528 | 0.7453 |
Sequential methods beat non-sequential ones everywhere, which is the paper's first point: the two modes want different machinery. InterFormer then leads the sequential group, by up to 0.14% AUC and 0.9% grouped AUC over the strongest competitor.
Production — Meta Ads, 70 billion training samples
The scaling behavior is arguably the more interesting result. Adding two long sequences of 1,000 events to the existing six improved NE by 0.14%, and InterFormer's loss curve keeps descending with more training data where the internal cross-attention baseline flattens — a 0.06% NE gap by the end. There's also a pure efficiency lever: merging six sequences into one of length 600 bought 20% QPS and 17% MFU for a 0.02% NE cost.
10 — carry this away
The whole thing in five sentences
Static features and behavior history are kept in two separate trunks that never shrink. Once per layer, a dedicated Cross Arch compresses each trunk into a handful of tokens and hands them to the other side. The static summary becomes the weight matrix of the sequence model's feed-forward layer; the behavior summary becomes extra columns for the feature-interaction module. Stack that block three or four times, read a prediction off the final summaries, and you get better accuracy than one-way designs. Run the two trunks concurrently and you also get the throughput, because one is network-bound and the other is compute-bound.