Motion-overlay — vclaw video motion-overlay

Turn an existing talking-head video into a polished reel with motion-graphics overlays synced to the speech, driven by Google Flow's Omni Flash V2V transport (which paints kinetic typography / icon / metaphor animations on top of footage while preserving the original voice).

The prompt IP — retention principles, a reel-anatomy load map, a concept→animation metaphor map, and two hard-won guardrails (no metadata leaking as on-screen text; no human portraits or the policy filter trips) — lives in deterministic, offline-testable code under src/video/motion-overlay/, not in markdown.

Plan/dry by default. No provider spend until you pass --execute --confirm-spend. The plan renders into the preview portal for human review first.

At a glance

vclaw video motion-overlay --input <video-path> (--project <slug> | --output-dir <path>)
  [--layout split|overlay|motion-only|avatar-host]   # default: split
  [--style apple-clean|editorial-dark|knowledge-tool] # default: apple-clean
  [--accent <hex>]                                    # configurable accent (replaces a hardcoded brand colour)
  [--lang <code>]                                     # STT + on-screen-text language (default: auto)
  [--max-take-seconds 10]                             # Omni hard per-clip limit
  [--transcript <path>]                               # optional: bring-your-own transcript JSON
  [--gb-character <Name:ID>]                          # avatar-host layout ONLY: the locked host character
  [--preview]                                         # also render review/review.html
  [--root <path>]
  [--execute --confirm-spend]                         # render (spend-gated)

What happens

Default (dry): ingest → STT → slice → compose → write work folder + manifest

• (optionally) preview HTML. No provider is called.

--execute (requires --confirm-spend, else exit-3 spend_confirmation_required): each take runs through omni-flash V2V, the original take audio is restored, and the takes are restitched into the final reel.

Work folder layout

<output>/                            # projects/<slug>/artifacts/motion-overlay/ OR --output-dir
├── motion-overlay-plan.json         # the manifest (schemaVersion 1)
├── README.md
├── source/  original.mp4 · audio.aac · transcript.json
├── takes/   take-01_0s-10s.mp4 ...  # frame-accurate cuts
├── prompts/ take-01_0s-10s.md ...   # one ready-to-send prompt per take
├── frames/  frame_01.jpg ...
├── host/    take-01_0s-10s.mp4 ...  # avatar-host ONLY: generated speaking host clips
├── v2v/     take-01_0s-10s.mp4 ...  # omni-flash overlay output (execute)
├── restored/ take-01_0s-10s.mp4 ... # audio-restored takes (execute)
├── review/  review.html             # preview-portal approval surface (--preview)
└── motion-overlay-reel.mp4          # the final stitched reel (execute)

Layouts

🔲split

Motion graphics in the top half, speaker untouched in the bottom half.

--layout split

🎞️overlay

Motion graphics composited over the speaker, safe areas around face and hands.

--layout overlay

📊motion-only

Speaker removed, full-frame motion graphics narrated by their voice (keynote style).

--layout motion-only

🧑‍🏫avatar-host

Like motion-only, but an identity-locked character host delivers the lines. Requires --gb-character Name:ID.

--layout avatar-host

split is the default. avatar-host (Layout D) is the heaviest path.

• split (default) — motion graphics in the top half, speaker untouched in the bottom half.
• overlay — motion graphics composited over the speaker, safe areas around face/hands.
• motion-only — speaker removed; full-frame motion graphics narrated by their voice (keynote style).
• avatar-host (Layout D) — like motion-only, but the frame is filled by an identity-locked character host delivering the lines, with synced overlays on top. Requires --gb-character <Name:ID>. This is the heaviest path; see below.

Avatar-host (Layout D)

The avatar-host layout replaces the speaker with an identity-locked character. There are two host engines (--host-engine), trading identity-lock against native voice — pick per project:

--host-engine	Identity	Voice	Best for
omni-r2v (default)	loose (mitigated, see below)	native + lip-synced	a presenter who actually speaks
veo-i2v	pixel-tight (same still every take)	silent (captioned / add VO)	brand-critical identity, talking-head B-roll

There is no scriptable path that gives both at once (Flow's @Character system locks identity+voice but is web-UI only), so this is a genuine fork.

omni-r2v (default) — the talking avatar

character reference image  +  the take's line  +  a voice preset
   → omni-flash R2V (ingredients:) generates the audio AND lip-synced video together
   → host/take-NN.mp4   (the avatar SPEAKING that line, genuinely lip-synced)
   → stitch the host clips in order        →  avatar-reel.mp4
   → re-STT the avatar's own speech + local lower-thirds  →  motion-overlay-avatar.mp4

Because Omni produces the speech and the mouth movement jointly, the avatar is truly lip-synced — no separate TTS, no muxing, no lip-sync model. And unlike the V2V "add-overlay" edit, R2V generation is not moderation-blocked.

veo-i2v — the tight-identity (silent) avatar

the SAME character still  (image: literal first frame)  →  Veo 3.1 I2V, per take
   → host/take-NN.mp4   (frame 0 of EVERY take is pixel-identical → identity locked)
   → stitch  →  avatar-reel.mp4
   → caption from the planned script, timed to each take's snapped duration
                                          →  motion-overlay-avatar.mp4 (silent)

Because every take starts from the identical still, identity/wardrobe/backdrop are locked frame-to-frame and take-to-take — far tighter than R2V — but the clip carries no speech (add a voiceover or soundtrack separately, or let the captions carry it). Veo I2V is landscape-only server-side. It shares the same retry + resume + scratch plumbing as omni-r2v (a slow provider queue can time a take out; just re-run to resume from it — completed takes are reused).

Flags: --gb-character Name:ID (label + go-bananas id, split on the final :), --gb-character-image <path> (the reference still — required at --execute; it is the R2V ingredient for omni-r2v and the literal first frame for veo-i2v), --host-engine omni-r2v|veo-i2v (default omni-r2v), --gb-voice <preset> (a Flow voice preset, default Puck; omni-r2v only), --host-retries <n> (per-take generation attempts, default 3), --host-look <text> (a stable appearance + setting descriptor — pinned on every take by both engines), --no-host-chain (disable cross-take chaining; omni-r2v only). Captions are timed to the avatar's actual speech (it is re-transcribed because Omni's pacing differs from the source); the caption pass is best-effort — the lip-synced reel stands on its own if it is skipped.

Character consistency (the hard part). omni-flash R2V treats the reference image as a loose influence, so each independently-generated take re-invents the face, wardrobe, and backdrop — the avatar drifts across a multi-take reel. There is no API-scriptable mechanism today that locks identity and generates native voice together (Google Flow's @Character system does, but it is web-UI only). Two levers reduce the drift on the native-voice path:

• --host-look <text> pins a stable appearance + fixed setting descriptor that is repeated on every take's prompt (e.g. the character's go-bananas base prompt + one locked studio). This is what stops the backdrop/wardrobe from jumping take-to-take. Without it the prompt uses a neutral default. (The old build hardcoded a Bunty-flavoured "friendly cartoon presenter in a sleek modern studio", which actively caused drift for any other character — that is now gone.)
• Chaining (on by default, --no-host-chain to disable) seeds each take's R2V with the previous take's last frame as an extra ingredients: reference, so the face/wardrobe carry forward. The last frame is extracted with ffmpeg to a hidden host/.chain-NN.png and chained into take N+1. Chaining survives resume (a resumed take re-seeds from its existing clip) and is best-effort (a failed frame extract just drops that one seed). For tight pixel-level identity instead, use the silent go-bananas-character → Veo-I2V path (no native voice).

Robust across multiple takes. The Flow safety filter rejects a benign R2V generation probabilistically, so on a multi-take reel one unlucky take would otherwise fail-fast the whole run. Each take is therefore retried up to --host-retries (a plain retry of the same line usually clears), and generation is resumable — a host/take-NN.mp4 already on disk from a prior run is reused, never regenerated, so re-running after a mid-reel failure never re-spends on the takes that already succeeded.

The generation runs only under --execute --confirm-spend; in plan/dry mode the work folder is laid out and the host clips are merely enumerated.

On-screen text guardrails (CRITICAL RULES)

Every composed prompt emits three standing rules:

1. AUDIO — pass the original voiceover through; the visual layer only.
2. NO METADATA ON SCREEN — never render px/ms/hex/font-name/easing/stroke as visible text; only quoted text renders.
3. TEXT ONLY, NO PORTRAITS — no drawn human silhouettes/portrait outlines/avatar icons (policy-filter guardrail). The avatar-host host base layer is exempt because it is generated separately and never described to Omni as a portrait.

Transcription

Gemini STT (reuses the Gemini key pool) by default, or bring your own with --transcript <path>:

{ "language": "pt", "segments": [ { "start": 0.0, "end": 3.2, "text": "..." } ] }

Module map (src/video/motion-overlay/)

Module	Purpose	Purity
types.ts	shared contract for the whole family	types
slice.ts	segments + maxTakeSeconds → sentence-boundary ≤10s splits	pure
analyze-reel.ts	per-take reel anatomy + anchor words + pauses	pure
motion-style.ts	style registry + concept→animation metaphor map; accent/brand resolution	pure
compose-prompt.ts	the IP — per-take prompt (CRITICAL RULES + layout block + style + shots)	pure
plan.ts	assemble the MotionOverlayPlan manifest	pure
transcribe.ts	Gemini STT (injectable transport)	side-effect, thin
ingest.ts	ffmpeg probe / extract audio / cut takes / extract frames	side-effect, thin
write.ts	emit folder + README + manifest + per-take prompts	side-effect
preview.ts	render the plan into the preview-portal review surface	side-effect
execute.ts	per-take V2V → audio-restore → stitch (injectable MotionOverlayStepRunner)	side-effect, tested via injection
avatar-host.ts	avatar-host: per-take speaking-host orchestration (injectable MotionOverlayHostGenerator)	side-effect, tested via injection
avatar-host-transport.ts	real omni-flash R2V+voice host generator (shells flow.ts); pure prompt/arg builders tested, spawn injectable	side-effect, pure builders tested
v2v-transport.ts	real Google Flow omni-flash V2V transport (upload → submit → poll → download); pure request helpers tested, network path injectable	side-effect, tested via injected fetch
render-local.ts	native local overlay renderer — segment → lower-third SVG → sharp PNG → ffmpeg overlay; SVG/filtergraph builders pure, sharp+ffmpeg injectable (--render local)	side-effect, pure builders tested

Every side effect — STT, ffmpeg, V2V, and the omni-flash R2V host generation — is behind an injectable interface, so the whole pipeline is unit- and e2e-tested offline with no network and no spend.

Schema

schemas/video/artifacts/motion-overlay-plan.schema.json is the canonical MotionOverlayPlan contract (allowlisted in the artifact-schema-coverage check).

Live execution & moderation (--execute)

--execute --confirm-spend runs each take through the Google Flow omni-flash V2V transport (v2v-transport.ts). Requires USEAPI_API_TOKEN and USEAPI_ACCOUNT_EMAIL in the environment (export them from .env); missing creds fail fast as provider_unreachable before any network call.

Proven request recipe (validated live 2026-06-05 against useapi.net):

1. Upload the take — POST {base}/google-flow/assets/{email} with Content-Type: video/mp4 and the raw bytes → mediaGenerationId.mediaGenerationId (this becomes referenceVideo_1).
2. Submit — POST {base}/google-flow/videos with { model: "omni-flash", prompt, referenceVideo_1, startFrameIndex_1: 0, endFrameIndex_1: <frameCount>, aspectRatio, email, async: true }. The response field is jobid (lowercase i), not jobId.
3. Poll — GET {base}/google-flow/jobs/{jobId} with the job id passed RAW / unencoded → status: created → completed | failed.
4. Result — response.media[].videoUrl; download to the take's output path.

Then execute.ts restores the take's original audio (ffmpeg -map 0:v -map 1:a) and clip-stitches the takes into the final reel.

Moderation gotcha. The Flow safety filter rejects some input videos for editing with PUBLIC_ERROR_UNSAFE_GENERATION / FINISH_REASON_INPUT_VIDEO_EDIT — this is input-specific and prompt-independent (a benign prompt on a flagged clip still fails; realistic-face footage is the usual trigger). The transport retries a failed verdict a few times (it is probabilistic), then errors with an actionable message. Use stylized / non-photoreal source clips. The sync bun run flow.ts --ref-video path returns the generic "All operations failed" and omits the frame-window params — prefer the async path above.

Delivery targets (--delivery)

--delivery is the high-level "where the animation comes from" knob (it sets the render mode + flow-pack defaults; --render/--emit-flow-pack stay as the low-level controls):

--delivery	What it does	Notes
local	render the finished reel locally	reliable, free, on your exact footage (default product path)
flow-web	render local and emit a Flow hand-off pack	the local reel is the instant fallback; the pack is for the real Omni animation, pasted into Flow's web editor by hand
flow-api	omni-flash V2V via the API	gated — needs --execute --confirm-spend; Flow currently safety-blocks the V2V edit for all inputs

The Flow hand-off loop (--emit-flow-pack / --restitch)

The source skill's real pipeline is write prompts → upload each take + paste its prompt into Flow → download → stitch. The Omni step was never automated and Flow's API blocks the V2V edit, so the hand-off stays manual — but it's now a first-class artifact instead of ad-hoc scripts (flow-pack.ts):

• --emit-flow-pack writes <work>/flow-pack/: each take's clip + a clean, ready-to-paste prompt .txt (the prompt body extracted from prompts/take-NN.md)
  • a README of the workflow.
• --restitch <flow-outputs-dir> takes the operator's downloaded animated takes, re-muxes each with its original audio (CRITICAL RULE 1), and concatenates the finished reel (<work>/motion-overlay-reel.mp4). Terminal — no render/spend.

# 1. produce a local reel now + the Flow pack to upgrade it
vclaw video motion-overlay --input clip.mp4 --output-dir out/ --delivery flow-web --style editorial-dark
# 2. (you upload each out/flow-pack/take-NN.mp4 + paste take-NN.txt in Flow, download to out/done/)
# 3. stitch the animated takes back, audio restored
vclaw video motion-overlay --input clip.mp4 --output-dir out/ --restitch out/done/

The pure pieces (prompt extraction, README, output pairing, concat/restore args) are unit-tested; fs + ffmpeg are injectable seams.

--render local (no spend, no moderation) — the reliable path

Because the V2V "add-overlay" edit is moderation-blocked for most inputs, the local renderer is the recommended way to actually produce a reel:

vclaw video motion-overlay --input clip.mp4 --output-dir out/ --render local [--kicker "BRAND"]

--render local (vs the default --render v2v) runs renderLocalOverlay (render-local.ts): each transcript segment becomes a broadcast lower-third — an SVG rasterised to PNG via sharp (resvg text shaping; NO system freetype / ffmpeg drawtext) — composited onto the source with one ffmpeg overlay filtergraph (each card fades + slides up on its window). Accent words reuse the prompt composer's anchor-word detector; --kicker sets the small brand label. It is free (no provider, no --confirm-spend gate) and the original audio is preserved. The pure SVG/filtergraph builders are unit-tested; sharp + ffmpeg are injectable seams.

Card styles (--style) and giant headlines (--headlines)

--render local honours --style, rendering three structurally distinct card looks faithful to the source skill's catalog (not just a colour swap — different panel geometry, label treatment and typography so they read as different over busy footage). The same --style/--accent/--kicker/--headlines also drive the avatar-host caption pass.

--style	Card look
apple-clean (default)	floating inset rounded frosted card · thin accent left bar · accent-dot + uppercase kicker · clean sans-serif sentence case
editorial-dark	full-bleed edge-to-edge near-black poster bar · thick accent top rule · solid accent tag block label · bold condensed UPPERCASE
knowledge-tool	left-anchored narrower cool-dark study card · thick lavender left rule · [[ MONO ]] bracket label · editorial serif body

Accent words render in the accent colour and get the skill's signature accent * beat-marker after them.

--headlines adds the skill's kinetic-typography beat: each detected anchor word also explodes to a frame-filling centred headline with the accent * beat-marker, synced to the moment the word is spoken, composited above the lower-third. Headline moments are capped (≤6, one per caption) so the reel doesn't strobe. Pure lowerThirdSvg/anchorHeadlineSvg builders are unit-tested.

--icons adds the skill's "visualize what's being said" layer (motifs.ts): the transcript is scanned and, at the moment a concept is spoken, a matching hairline icon (money → $, team → people, launch → rocket, AI → node graph, time → clock, …) or an infographic stat callout (a number, or a percentage with a ring gauge) pulses in the upper-right corner like a broadcast stat bug — in the style's accent, capped to ≤5 per reel. Icons are vector geometry (no portraits, no bitmaps), so they render reliably. Both --headlines and --icons work on --render local and the avatar-host caption pass. detectMotifs + the iconMotifSvg/statMotifSvg builders are pure and unit-tested.

--animate — real motion graphics (per-frame)

By default the local overlays fade and slide in, then hold. --animate swaps in a per-frame motion-graphics layer (animate.ts + animate-render.ts): every overlay is evaluated as a function of time t and one transparent frame is rasterised per output frame (30fps), then ffmpeg overlays the sequence on the footage. The motions:

• lower-third — fade + slide-up entrance, then a word-by-word reveal (words appear across the first ~60% of the take; the panel holds full size so it never reflows — later words render at fill-opacity 0);
• stat callout — the number counts up 0→N and the % gauge fills (eased);
• icons / headlines — pop in with a small overshoot (spring).

All the timing is pure math (easeOutCubic/easeOutBack, revealWordCount, countUpValue, gaugeProgress) and unit-tested; the sharp rasterise + ffmpeg run are injectable. It costs more (N frames rasterised — a 16s clip ≈ 480 frames, ~20s on a laptop) but stays fully local/free. Works with any --style and composes with --icons/--headlines.

vclaw video motion-overlay --input clip.mp4 --output-dir out/ \
  --render local --style apple-clean --icons --animate --kicker "BRAND"

vclaw video motion-overlay --input clip.mp4 --output-dir out/ \
  --render local --style editorial-dark --headlines --kicker "BRAND"