Generative AI is well known for its ability to create remarkable original content. However, its power extends even further when modifying existing material. By focusing on a specific portion of existing content, changes often seamlessly integrate into the overall presentation. This subtlety makes it less noticeable, and, in turn, more effective. This is particularly evident in the realm of lip-syncing, where generative AI is used to synchronize (AI-generated) audio with the mouth of an actor.
Flawless is a company that specializes in this kind of VFX. They create the professional tools that allow filmmakers to put this into practice. The first major value-creating application they advertise is dubbing, perfectly lip-syncing films in any language while maintaining the authenticity of the original. This allows movie or advertising studios to unlock new markets, or at least improve the quality over the current standard of simply replacing audio. (Flawless, 2023)
The second major value-creating application is the ability to edit new lines of dialogue and performance in a piece of film. Previously there were only physically filmed reshoots, now this can also be done in post-production saving both the time and money of going back to set. One prominently showcased example is censoring swear words to make films more suitable for younger audiences. (Flawless, 2023)
The video below shows a short demo of the above-mentioned applications.
I am most excited to see AI generative dubbing put into practice. As the Netherlands isn’t a big enough market to create unique versions of content for, I have seen plenty of poorly dubbed ads and movies, and have developed quite an aversion to them. The viewing experience and sincerity of the message significantly suffer in my opinion. This solution seems very promising, addressing the issue at its core. I hope to see more studios and filmmakers start utilizing it soon.
Sources:
Flawless. (2023, October 12). Flawless AI – AI filmmaking tools for VFX. Flawless AI. https://www.flawlessai.com/
Flawless AI. (2023, January 4). Flawless use of TrueSync on the movie “Fall” [Video]. YouTube. https://www.youtube.com/watch?v=iQ1OPpj8gPA
I think this is the first real practical example of GenAI that I am most excited about. Sure, gaining access to the previously unknown territory of Bollywood is something we should all be very excited about. However, I am mostly excited about this technology’s capability in eliminating impediments in traditional translation methods and effectively lowering the borders between nations. If we could make strides towards gaining a better understanding of each other through language, I believe the world would be a much better place.
Thanks for sharing Mark!
I love your post!
I am a big movie enthusiast myself, and I was not aware of the capabilities in the realm of filmmaking. I rather thought that AI could help with CGI or writing scripts, but I did not expect it to be utilized as a tool which can be used to edit mouth movements into movies after they had been shot. Furthermore, I believe this is a great example of how AI can influence a wide variety of businesses and I am curious whether this technology will be adapted throughout the industry, especially given the big strike of actors and writers against AI in Hollywood.
However, I do see the huge potential this could have in terms of movies being translated and not seeming completely of. This could actually revolutionize a big part of the film industry, especially for countries like the Netherlands which usually do not get Dutch dubbed movies but only subtitles. Therefore, I would have enjoyed reading more about the financial side of this application and the feasibility to use it wide spread but I am also aware that this is only a 300 word blog post.
Overall great post, very interesting and I am happy to have come across this Mark! I am curious how this will develop in the future.
Cheers