Video Professionals Going Deep
There are many applications in video (and movie) production where “deep learning” has been applied, now that the media industry has a better understanding of what deep learning actually means.
Basically, deep learning is a subset of machine learning where artificial neural networks – that is, algorithms inspired by the human brain – learn from large amounts of data. One of the more complex ways of doing this is by mimicking the neurons in a human brain. A neural network is a type of machine learning that, via a set of algorithms, allows a computer to learn and improve upon the task at hand. When we make these artificial brains, or neural networks, more complex, we call that deep learning.
Many are calling deep learning the new frontier for the video industry, as it allows video professionals to do things automatically that would have taken weeks of work in the past, as well as complete tasks that wouldn’t have been possible at all. For example, it allows a computer to take all the pixels in a frame of video and output something equally complex, such as all the pixels in a new, altered, frame of video. It may be shown frames with unwanted grain as input, and have its output compared to clean frames. By trial and error, it learns how to remove the grain from the input. As more and more images are passed through it, it can learn how to do the same thing for images that it was never shown.
Deep learning can also be used to match generated speech with human speech, so text-to-speech programs sound more natural. In a similar task, it is used by translation companies to teach computers how to translate from one language to another.
In Hollywood, Warner Bros. movie studio spend $25M on reshoots for its latest installment of “Justice League” and part of that money went to digitally removing a mustache on its star actor. Deep learning is being called a game changer for these are types of highly detailed tasks.
On the consumer side, easy-to-use solutions like Flo allow you to use deep learning to automatically create a video by describing what you want in it. The software will find the relevant videos from your library and edit them together automatically.
Video restoration is another application for deep learning. The UCLA Film & Television Archive said that nearly half of all films produced prior to 1950 have disappeared. And 90 percent of the classic film prints that do exist are in poor condition. The process of restoring these films is long, tedious, and expensive. Deep learning is now helping to bring those films back to their original glory.
The process of colorizing black and white footage has always been lengthy. There are thousands of frames of footage in a movie and coloring each one takes a long time. Even with advanced tools, the process can only be automated so much. Deep learning can now speed up the process significantly, with tools that only require an artist to color one frame of a scene. From there, the deep learning network automatically handles the rest.
Another problem that can be solved with deep learning algorithms is missing or damaged frames from a video. You can’t do reshoots on something that happened years ago. Restoring that type of movie before meant editing around the missing frames. Now, deep learning networks are being deployed to change that. Google has developed a technology that can realistically recreate part of a scene based on start and end frames.
By detecting the faces of everyone in a video, deep learning can allow you to search for any clip or movie that has a given performer. Alternatively, you could use the technology to count the exact screen time for every actor in a video. Sky News recently used facial recognition, powered by AWS, to identify famous faces during coverage of Harry and Meghan’s royal wedding.
However, the technology is not limited to detecting just faces, sports broadcasts now rely on camera operators to track the movements of the ball, or to identify other key elements of the game, such as the goal. Using object recognition, AI-powered tools can be also used to automate the production of a sports broadcast.
As we move into 4K streaming, it is using more data than ever before. Thanks to neural networks that can recreate high-definition frames from a standard-definition input, we could soon be streaming SD files over the Internet, while still enjoying HD quality.
Going forward, the use of deep learning in film and broadcast has only begun to touch the surface of what is possible. As with any new technology, the video industry and tech experts must come together to develop the standards of how tomorrow’s new normal might look. However, with the right approach, deep learning will take film and television to a whole new level.