How we test new models
By Syntax Motion, Vienna. Updated: 16 October 2026
Generative video models appear at a monthly rhythm, and every one of them is announced as a breakthrough. Because we produce client films with them and not demos, we have a fixed test procedure. This post describes it, including the subjects on which most models fail.
Six subjects, always the same
A new model gets the same six tasks here as every one before it. Only that way is a comparison possible, and only that way does it show when a model has become better in one area and worse in another.
- A hand on the product. A hand picks up an object and turns it. The classic, and most models are still visibly working on it.
- Liquid in motion. Pouring, splashes, condensation. Here the models are strong by now.
- Packaging with type. A label that has to stay readable. Still fails regularly, see inpainting.
- A face in close-up, speaking. Three seconds work, ten rarely, see lipsync.
- A camera move through a room. Tests whether the perspective stays stable over time.
- The same character twice. Two shots, one person, see character consistency.
Each subject runs several times with an identical description and a noted seed. What counts is not the best attempt but how many of the runs are usable. A model that shines once and misses otherwise is more expensive in production than one that is solid throughout.
What a breakthrough usually is
The most visible progress of recent months was in movement and material, that is, where the models were strong anyway. What has hardly moved: type, hands in close-up and consistency across several shots. Those are not by chance the three points on which a production fails and a demo does not.
That is why we rarely switch right away. A new model only enters client projects when it passes the same test at least as well as the current one, and when it is clear what the terms of use say about commercial work and about training data.
Why we work with several
There is no best model, there are suitable ones. For moving images we use Seedance, Veo, Kling and Google Flow, for stills Seedream, Nano Banana, GPT Image and Seed 1.0. Which one a project gets is decided by the subject, not by habit. For confidential products the production runs over the enterprise interface, so the material does not flow into training.
What this means for clients
You notice nothing of this procedure, and that is the point. What you would notice is the opposite: a film produced with a freshly released model that happens to be weak on your subject. Our testing runs outside projects, in spec work.
Frequently asked questions
Which AI video model is the best?
There is no best one, there are suitable ones. Different models are strong for movement and material than for consistency across several shots. Which one a project gets is decided here by the subject, not by habit.
How long does it take until you use a new model?
Until it passes our six test subjects at least as well as the current one and the terms of use for commercial work are clear. As a rule that is some weeks after release, not days.
Do you test on client projects?
No. Testing runs outside projects, in spec work. A client film is the wrong place to find out whether a model is weak on a particular subject.
What about models that take training data from your input?
We do not use those for confidential products. There the production runs over an enterprise interface where input does not flow into training. We put that in writing for each project.
Read on: All posts • What we produce with • Generative AI
Which model suits your subject?
We answer that question in the first call, not after the invoice.
Book a first call