Claude beat GPT-5.6 on costlier AI music videos

A small open-source benchmark asked two frontier models to do one thing on their own: turn Bruno Mars and Mark Ronson’s “Uptown Funk” into a complete music video under a fixed budget. The harness gave each model a song, a short prompt, a time-stamped lyric transcript, and six tools. From there, the models had to research video generators, create clips, inspect their own output, edit with ffmpeg, and assemble a final cut.

The test ran Claude Fable 5 and GPT-5.6 Sol at $25 and $100 generation budgets, for four runs total. According to the source, all four runs completed without hitting step or time limits, and each produced a full-length video with the original song muxed in.

The tool set included:

• plan for internal thinking
• web_search for researching models and music-video references
• get_budget to check remaining spend
• generate_image and generate_video as the only paid tools
• run_command for local shell access with ffmpeg and ffprobe

Costs, runtimes, and model choices

At the $25 cap, both models nearly spent the full budget. At $100, they used much less than the ceiling: GPT-5.6 Sol spent $36.57 on generation, while Claude Fable 5 spent $48.60. The source says that extra budget did lead to more footage, with distinct clips per run ranging from 46 to 80.

The final totals changed once token costs were added. Using the source’s pricing assumptions for LLM usage, the full run costs were:

Recommended reading

Hassabis says STEM makes you 10x better at AI

• Claude Fable 5 · $25: $41.29 total
• GPT-5.6 Sol · $25: $27.45 total
• GPT-5.6 Sol · $100: $39.82 total
• Claude Fable 5 · $100: $73.65 total

Claude finished faster in both budget tiers, but it was also the more expensive option. Its token charges alone came to $16.99 and $25.05, roughly 30–40% of each run’s total cost. GPT-5.6 Sol’s token cost stayed around $3–4 despite similar token volume.

The models also made different production choices. Three of the four runs used pure text-to-video. The exception was GPT-5.6 Sol at $25, which generated still images first and then animated them into video. GPT-5.6 Sol at $100 also mixed three different video models in one run: Wan 2.5, Veo 3.1 Lite, and Hailuo 2.3 Standard. By contrast, Claude stuck to one video model per run.

What worked, and what didn’t

The source’s verdict is blunt: none of the videos were great. Character consistency broke across shots, storylines drifted, and both models took lyrics very literally.

“Make a dragon wanna retire, man” gets you an actual dragon on screen.

Beat matching was also limited. The cuts generally landed on the beat after ffmpeg-based beat detection, but motion inside the clips—dancing and camera movement—often failed to match the song’s tempo.

The most interesting exception was GPT-5.6 Sol at $25, which the source calls the most inventive editor. It added text overlays and animated stills with video effects, while most of the other runs mainly stitched clips together. But the source also says GPT-5.6 Sol’s $100 output included some genuinely low-quality clips, while Claude Fable 5 happened to select a model with more coherent results.

One notable pattern: neither model used Replicate, even though both FAL and Replicate keys were available. All four runs used FAL only.

The full harness is available at github.com/hershalb/music-video-arena, including logs of every message, tool call, charge, and error. The source’s takeaway is that autonomous video-making agents can already complete long, open-ended workflows, but they still do little real self-critique once the clips start rolling.