How to write an x86 interpreter in raw WebAssembly Text

Wine-Assembly runs Windows 98 programs in the browser on an x86 interpreter written directly in WebAssembly Text (WAT), with no C, Rust or AssemblyScript in front of it. This is how that interpreter is put together: a decoder that turns x86 into threaded code, an indirect-call dispatcher, a block cache, and the measurements that decided which optimisations were worth keeping. It is the technical core of the project story.

Threaded code, not a giant switch

Most interpreters written in a systems language are a switch over opcodes inside a loop. WAT has no switch and no computed goto, but it does have call_indirect through a function table, and that is enough for Forth-style threaded code.

The decoder in src/07-decoder.wat reads x86 bytes once per basic block and emits a sequence of (handler index, operand) pairs into a thread cache. Every x86 instruction form has a small WAT function, a handler, that does its work and then calls $next. $next advances the thread pointer, loads the next handler index and jumps to it through the handler table (src/02-thread-table.wat, several hundred entries). A basic block is therefore executed as a chain of indirect calls, and the x86 bytes are never looked at again until the block is evicted.

flowchart TD
    X86["x86 bytes<br/>(guest memory)"] -->|"once per basic block"| DEC["Decoder<br/>src/07-decoder.wat"]
    DEC --> TC["Thread cache<br/>(handler, operand) pairs"]
    TC --> NEXT["$next<br/>load index, call_indirect"]
    NEXT --> H1["handler: add r32"]
    H1 -->|"return_call $next"| NEXT
    NEXT --> H2["handler: mov [mem]"]
    H2 -->|"return_call $next"| NEXT
    NEXT --> H3["handler: jcc"]
    H3 -->|"next block: cache lookup"| BC{"Block cache<br/>4096 slots"}
    BC -->|hit| TC
    BC -->|miss| DEC

Two consequences shape everything else:

The block cache

Each decoded block is found again by hashing its guest address: the slot index is (ga ^ ga>>12) & CACHE_MASK, over 4096 slots, and the threaded code itself lives in a 30 MB arena carved into eight per-thread partitions so guest threads never invalidate each other's code. tools/cache-slots.js replays a real working set through that hash to decide whether a miss storm is a size problem or an aliasing problem. On Caesar III the working set was 1,861 blocks in 4,096 slots and no alternative hash did better, which settled that its re-decodes were not a cache-size problem.

Execution is metered in blocks: the host's run(N) spends one budget unit per block, and a block gets a quantum of 1,000 threaded ops. A block is not a fixed amount of work. Measured on Diablo, its Smacker intro retires about 7 ops per block while its menu retires 282, a 41x spread inside one program, which is why the project quotes ops or wall time and never "batches per second".

What a dispatch costs

Because the whole machine is indirect calls, dispatch cost is the number that matters. The project measured it rather than guessing:

Folding loops into super-ops

Since dispatch is the cost, the obvious lever is fewer dispatches. src/07b-loop-match.wat looks at every self-loop block the decoder emits, classifies its ops into roles (induction variables, memory streams, side effects) and, when a known idiom holds, replaces the whole body with one super-op that runs the loop inside a single handler. REP MOVS/STOS were the first case, lowered to memory.copy and memory.fill. Table-lookup runs (LUT_RUN) are on by default; a run-length sprite blit fold for Caesar III bought about 7%, a rectangle-fill fold about 12%.

flowchart TB
    subgraph before["Before: one self-loop block, N iterations"]
        direction TB
        L1["load [esi]"] --> L2["xlat"] --> L3["store [edi]"] --> L4["inc esi, inc edi"] --> L5["dec ecx"] --> L6["jnz"] -->|"dispatch x6 per iteration"| L1
    end
    subgraph after["After: the matcher recognised LUT_RUN"]
        S1["LUT_RUN super-op<br/>whole loop in one handler"]
    end
    before -->|"src/07b-loop-match.wat"| after

The design doc, loop-idiom-superops-design.md, records the matcher's decline histogram across ten games: only about 2% of static self-loops match, and calls and multi-branch bodies are most of the declines. The lesson recorded there is to fold memory traffic, not control flow: a fold that removed a jump chain but kept every dispatch measured at roughly zero.

The nulls are written down too

Several ideas that sound obviously good were measured and found to be nothing, and the docs keep them so they are not retried:

A separate, smaller machine, the toy VM, exists partly so that dispatch designs can be compared on a machine small enough to rewrite in a day. Its shootout of dispatch strategies is in toyvm-dispatch-shootout.md, and its region JIT is the next step the main emulator is measuring (repl-tailcall-main-emu.md).

Further reading