I’ve recently experimented with video-generative AI, trying various models and tools. I started my journey by trying out KlingAI and Pikalabs. Out of these two, I preferred KlingAI as it delivers more realistic videos. However, it is rather slow, often taking 15 minutes or more to produce a 5-second video. So after trying these two, I was not completely satisfied so I did some more research and came across Runway.
I first tried out their free Gen-2 model, while this model can produce decent videos using image-to-video or text-to-video techniques, their Gen-3 model is miles ahead. To illustrate this difference I used the same prompt in the Gen-2 and Gen-3 models.
As you can see the Gen-3 model produced a video which adheres better to the prompt and is far more realistic than the Gen-2 output. Whilst you could go and make yourself a cup of coffee and some lunch while KlingAI was generating your video, you don’t have to with Runway. A 5-second video will take about 60 seconds to generate and a 10-second video will take about 90 seconds. The only big downside of Runway Gen-3 is the cost, a 10-second video will set you back about a dollar in credits. If every prompt results in the desired outcome, which does not always happen, a minute of video will cost you at least 6 dollars. I made a 2-minute video which cost me probably five times that amount. This is only partly due to failed generations. The other side of the story is that it takes a bit of time to figure out the model and get to know the prompt structure that works best. To maximize the effectiveness of my prompts I used a GPT called “Runway Prompt Pro”. This GPT will convert your input into a prompt specifically tailored to Runway’s model. I’ll show you an example:
My input: drone shot flying over an active volcano
Runway Prompt Pro: [drone shot, smooth movement]: The drone flies over an active volcano, capturing the molten lava bubbling inside the crater. Ash clouds rise from the fiery pit, glowing red and orange in the twilight. Lava streams flow down the dark mountainside. The surrounding rugged landscape contrasts the intense heat and vibrant colours. [wide-angle, steady]
I used this prompt as part of the video I made, resulting in the following video:
Using this prompt GPT significantly boosted my first generation’s success rate. The video above was the result of the first generation. I experienced quite a good first-generation success rate when generating landscape videos. I however had some more difficulties when I tried to make videos of people and animals. While Runway is still very good at this, it might take a few more tries. Because generations are not cheap this can be quite frustrating. For this reason, Runway is often combined with Midjourney. Midjourney allows you to make hyper-realistic imagery which Runway can then transform into videos. If you are serious about video making with AI, I would recommend this combo.
Wow, your journey with video-generative AI sounds fascinating! It’s interesting to see how each tool has its strengths and weaknesses. I can relate to your experience with KlingAI and Runway—getting that balance between quality and speed is always a challenge. The detail you described between Gen-2 and Gen-3 really highlights how quickly the technology is advancing.
I’ve also found that using specific prompts can make a huge difference, and it’s great that you’re utilizing Runway Prompt Pro to refine your prompts. The success rate improvement you’ve seen with landscape videos is impressive, though I can see how generating videos of people and animals could be trickier. It’s also smart to combine tools like Runway and Midjourney to create hyper-realistic visuals; that collaboration seems like a game-changer for anyone serious about AI video production.
Thanks for sharing your insights! I’m definitely inspired to explore these tools further and experiment with my own projects. Have you thought about any other combinations or models that might enhance your workflow?
Firstly, I hope the money you spent came from an urge of personal interest and not to create a fascinating blog post.
I’ve also had some experience with runaway and found it jaw-dropping. A friend used it to create a hyper-realistic marketing campaign, and what he got from it (after the 5th attempt) saved him hundreds if not thousands of euros.
Firms spend tens of thousands on marketing campaigns, I mean really ask yourself how generic certain campaigns are! Our attention spans have gotten shorter anyway, which speaks for the usage of GenAI.
Also, this then obviously begs the question of the extent to which it may replace videography as a whole.
Actors like Bruce Willis have already old their facial recognition rights for digital use. Are movies next?
Thank you for your insights, Coen Timmerman