Claude learns to paint

I asked for a sumi-e ink demo, then for Claude to fix the limitations it had listed.

An ink palette (pine soot, oil soot, indigo, ochre, gamboge, cinnabar), three xuan rice papers and a goat-hair fude (an East Asian ink brush) that can be loaded in three tones. Raw xuan didn't bleed any more than watercolor paper did, so Claude added sideways wicking and filtration. Then it added a separate layer of fine carbon that travels further than the rest, which makes the halo (nijimi). It also carved a seal and brushed a signature.

• Halo: a 6.8 mm core with a ~5.6 mm pale fringe at 10 to 14% of its darkness on raw xuan, against 0.9 mm on sized xuan (coated so ink spreads less).
• Still missing: a dark tide-line at the halo's edge. An edge-evaporation attempt was reverted.

I asked Claude to build itself a way to paint: generate a reference with gpt-image-2.5, save a checkpoint every few actions, judge each one, roll back what doesn't work, and count the rolled-back branch as a lookahead.

Claude drives the sim the way a painter works a sheet. It can test a stroke on scrap paper, dry a passage forward to see how it will settle, and calibrate how each pigment and load actually looks once dry. Every checkpoint can be returned to exactly, so trying something and taking it back costs nothing.

• The sim runs about 10× real time. 15 checkpoints in 27 s.
• The reference is the picture Claude is trying to paint. A blank sheet scores 12.25 against it.

I wanted an end-to-end video of it painting.

A scrap test showed raw xuan haloed every stroke, so the run used half-sized xuan. Calibration showed pine soot is steep: a load of 0.1 already gives L78 (lightness, 0 black to 100 white), so the mist needed loads around 0.03 to 0.06.

• Score 12.25 → 9.29, with 95 actions kept.
• Rolled back: the mist twice (too dark), blobby reed heads, a neck twice as thick as it should be, a carrot-shaped leg twice, and a cap blob from the fude splaying 3× at pressure 0.8.
• Darks painted into the wet body bled everywhere (the score got 1.87 worse). The fix was to dry the sheet first.

I asked whether the problems were limits of the simulator or of what Claude did with it.

"Mostly me." Most rollbacks came from misjudging how the brush turns a path into a mark. Claude found two real limits in the sim: pale ink has a floor, and edge softness only changes by switching paper. It also named its own process gaps: no model of its strokes up front, coordinates read by eye, batches of 6 actions, a score blind to texture, and only 3 lookaheads.

I gave it a list: a library of strokes to call on and adjust while painting, rollback to any checkpoint, a better score, and counting the pre-rollback branch as the lookahead.

Claude turned the first run's keep and reject calls into a test set. The old score was global, so a small batch's effect drowned in the whole sheet. The new judge only measures where a batch painted: error within a tolerance, over-darkening, precision and texture. It blames individual actions and returns BETTER, FLAWED or WORSE. Rollback now goes to any checkpoint, or to any single action inside one. The library turns "a 4 mm mark at L60, here" into brush moves using calibration sheets, and checks itself in a closed loop.

• The new judge flagged all 12 batches Claude had rejected in the first run (some rejections cost two checkpoints).
• Library accuracy, median miss: length 0.5 → 0.16 mm, width 7% → 6%, lightness 3.8 → 0.5.
• Hand-picked leaf coordinates were 5 to 10 mm off, so a tracing tool now reads strokes from the reference.
• Run 2 scored 8.98 against 9.29, but only by leaving out the leaves, cattail heads and crest.

I wanted the brush in the animation, and said we weren't getting enough texture.

Claude measured edge raggedness: 1.44 to 2.64 in the reference, 1.17 in Claude's painting. It found four causes. On the physics side, a dry brush gave paper speckle instead of hair streaks. The default water (0.38) always painted solid. Strokes laid into a wet wash merged. Resolution mattered least. The fixes were hair gaps anchored to the stroke, outer tufts that refill more slowly, and a texture setting (solid, natural, dry). A new "press" action gives an upright rosette or a leaning teardrop.

• The heron's back went to 1.53 raggedness against the reference's 1.44. The body underlayer is still 1.23 against 2.64.
• The repaint scored 9.37, slightly worse than run 2's 8.98. The score can't see texture, so it only counted the drier strokes as a small loss.

I asked how fast the sim could run, and whether Claude could branch from any earlier state, or recombine moves from different states and replay them.

Claude benchmarked everything first. Then it built a search that paints and scores dozens of variants of a move inside the page, keeping the best. Order matters in paint, so combined moves have to be replayed rather than merged.

• One stroke ~25 ms. A full replay of the heron from blank 3.4 s. Checkpoint save 3 ms, restore 1 ms.
• Scoring a candidate in the page: 131 ms dried, 18 ms wet.
• 48 painted variants in 7.8 s.
• On one leaf, the judge rated search's best stroke 63 against 20 for Claude's pick (on this per-stroke rating, higher is better). Claude had written the leaf backwards: "the score was right; I was wrong".

I suggested a codebook of strokes might work, since Claude could just experiment with lots of them, and asked it to fix the way the brush lands.

Brush contact first. The tip hairs had stopped 0.8 mm short, so contact snapped on at 4 to 7 mm wide. Now the point touches first and splay grows with contact time. Then the codebook: 12,000 gestures spread over 13 parameters. Each one is painted, isolated and measured, so a mark can be looked up by shape, and blended from its neighbors when there's no exact match.

• First touch 3.4 → 1.8 mm wide.
• 11,132 marks in 11 minutes on 696 sheets.
• Rotating a mark changed it by 0.9 to 3 mm, against 0.4 to 0.5 mm for just moving it, because the brush lean carried over from the last stroke. The fix was reverted, so the codebook is only approximate under rotation.
• Best automatic reed rated 60.5, against 63 for the best hand-found blade.

I asked why the results weren't amazing. That was the whole message.

Claude measured every stage and answered "mostly because of how I set it up, not the brush". Its traced strokes covered only 22% of the visible ink in the reed passage. Most marks the heron needed weren't in the codebook: the nearest entry was 1.5 to 3.6 times farther off than entries sit from each other. And the fit stopped at its first proposal. Painting accuracy was fine: marks landed within 0.3 mm of prediction. The fixes were stroke extraction from a skeleton of what's still missing, growing the codebook toward the target, refining every choice, and painting by value level (L40, 55, 68, 80). Pale smudges scored well because the score only counted darkness, so an outline constraint and a structure term went in.

• Reed passage: missing ink 9.95 → 4.06, then 3.76 with the outline constraint.
• Whole heron from blank: 129 of 157 marks kept out of 7,165 candidates, in 60 minutes. Missing ink by level 7.9 → 2.23. The codebook grew from ~11k to ~32k marks.
• Score 5.65, the best of the four ink herons (the hand-placed runs scored 8.98 to 9.37).
• Claude's verdict: it "isn't beautiful yet". The water is cloudy and the body blobby.