MacDirectory magazine is the premiere creative lifestyle magazine for Apple enthusiasts featuring interviews, in-depth tech reviews, Apple news, insights, latest Apple patents, apps, market analysis, entertainment and more.
Issue link: https://digital.macdirectory.com/i/1513481
Project Dub Dub Dub This may not be entirely impossible for humans, but it would be darn hard. Some consider art to be a universal language, but Adobe Sr. Research Scientist Zeyu Jin recognizes that in the case of video and film, there are some unavoidable limitations. If a piece was produced in another language, the viewer has the choice of hearing the voice and inflections of the original actor while reading distracting subtitles or they can watch the action, try to ignore the out-of-sync lip flapping and hope that the local voices dubbed in have some degree of acting talent. Mr. Jin saw that GenAI could provide a better option. Garden variety AI can handle the translation. That’s old hat and has been around for years (a few, anyway). GenAI, however can listen to the original actors’ voices and, in much the same way that the new Premiere Pro synthesizes good audio out of otherwise hopeless recordings, it regenerates the dialog in the new language with the old actors’ voices including the original star-quality delivery. As an introduction, he took a short recording he made backstage, plugged it in and with a mouse click, it played back in his voice in perfect Spanish. He continued the demo with a video clip from one of his colleagues spoken in Mandarin. This time he selected seven other languages and set his program to work, matching not only the lovely speaker’s voice and expression, but her face to the new words. The crowd, as the expression goes, went wild. True, he could have brought in a ringer and she was actually a linguist. But to put any doubts to rest, he brought in an actor we can all trust, Carey Grant. He chose a scene with co-star Rosalind Russell arguing in a newspaper office with all its Hollywood-perfect audio ambience. In just a few moments, we were listening to their quarrel in Spanish—and then German. Breaking through the language barrier with Project Dub Dub Dub