Categories
Post Production

VideoGorillas’ Bigfoot super resolution converts films from native 480p to 4K

Bigfoot super resolution converts films from native 480p to 4K

A new production-assisted AI technique called Bigfoot super resolution, from VideoGorillas, converts films from native 480p to 4K by using neural networks to predict missing pixels.

While the industry is moving forward in terms of resolution used in films produced, with 4K and 8K, following the evolution of film and video standards, there continues to be an interest in revisiting older content, which does not meet today’s visual industry standards or the expectations of audiences. To offer those audiences a more immersive viewing experience, the industry has used the only tool available to revitalize older content to the resolutions now in use: remastering. The remastering process has become a common practice, allowing audiences to revisit older favorites and enjoy them in a modern viewing experience.

 

As resolution increases, though, remastering has a problem: artifacts. This means that studios have to apply additional resources and manage longer lead times to produce reasonable quality content. That costs both money and time. Lots of money! Now, there is a new solution promising a viable alternative: to meet the growing pace of innovation,  a company, VideoGorillas, is developing a new AI-enhanced solution to exceed visual expectations at lower costs.

A new solution: Bigfoot Frame Compare

Los Angeles-based VideoGorillas, a developer of state-of-the-art media technology incorporating machine learning, neural networks, visual analysis, object recognition, and live streaming, are no strangers to the industry, having worked with major Hollywood studios. The company announced, September 2018, the availability of its Bigfoot Frame Compare product, which is set to redefine the way film, television, and post-production companies manage assets, finishing, and remastering and preservation projects.

Bigfoot is a scalable, lightweight proprietary technology that leverages VideoGorillas’ patented computer vision/visual analysis, Frequency Domain Descriptor (FDD), and machine learning technology. It is designed to automate the manual-labor intensive conform process (which matches an original frame of film to the final edited work) and the compare process (which compares unique or common frames between different film cuts) by finding like “interest points” common across a series of images or frames of film.

On November 2018, VideoGorillas announced that Netflix’s November 2, 2018 theatrical and online release of “The Other Side of the Wind,” the final, previously unfinished film by iconic filmmaker Orson Welles, utilized VideoGorillas’ artificial intelligence (AI) powered Bigfoot Frame Compare technology in post.

Analyzing 12 million frames per second

“The Other Side of the Wind” presented an unusual challenge to the filmmakers and VideoGorillas, because it was in many ways a hybrid of a film restoration and new release project. This resulted in Bigfoot being used by the producers in an equally unusual way: as a business intelligence tool. With more than 100 hours of 40-year-old footage in a range of formats, it was essential that the producers and editors quickly gain an understanding of what footage they had before kicking off the editing process.

“During the completion of ‘The Other Side of the Wind’ we were faced with a very unique challenge,” said producer Filip Jan Rymsza. “We had a 3.5-hour reference cut assembled from various sources, which had to be conformed back to 100 hours of scans, which consisted of 16mm and 35mm negatives and 35mm prints. Traditionally, an assistant editor would have over-cut the reference picture, but some of it was blown up from 16mm to 35mm, or blown up and repositioned, or flopped, or printed in black-and-white from color negative, which made it very difficult to match by eye. Without VideoGorillas’ AI technology finding these precise matches this would’ve taken a team of assistant editors several months. VideoGorillas completed this massive task in two weeks. They were a crucial partner in this extraordinary effort.”

Scanning the amassed film footage from “The Other Side of the Wind” resulted in more than 9 million frames. Since Bigfoot was initially CPU-based software, VideoGorillas developed an experimental GPU-version to enable one local machine in its Los Angeles office to do the job that would normally take 200-300 servers. After two days of ingesting the footage it took only three days of GPU time to analyze the film, and at peak load Bigfoot was analyzing 12 million frames per second.

Bigfoot super resolution

Now VideoGorillas has another technology that incorporates AI techniques built on NVIDIA CUDA-X  and Studio Stack. By integrating GPU-accelerated machine learning, deep learning, and computer vision, their techniques allow studios to achieve higher visual fidelity and increased productivity when it comes to remastering. The innovation they’re developing is a new production-assisted AI technique called Bigfoot super resolution. This technique converts films from native 480p to 4K by using neural networks to predict missing pixels that are incredibly high quality, so the original content almost appears as it was filmed in 4K.

“Bigfoot Super Resolution is an entirely new approach to upscaling powered by NVIDIA RTX technology with a focus on delivering levels of video quality and operational efficiencies currently not achievable using traditional methods. We are very excited to bring this solution to market and look forward to helping our studio and broadcast partners unlock incremental value from their content libraries” says Jason Brahms, CEO VideoGorillas.

As VideoGorillas continues to refine this technique for release, they aim to provide broadcasters and major film studios a superior way to remaster their content libraries while preserving original artistic intent.

Bigfoot super resolution converts films from native 480p to 4K

Remastering while preserving artistic intent

“We are creating a new visual vocabulary for film and television material that’s based on AI techniques. We’re working to train neural networks to remove a variety of visual artifacts, as well as understand the era, genre, and medium of what we are remastering. Using these neural networks allows us to increase perceptual quality and preserve the original look and feel of the material”, says Alex Zhukov, CTO at VideoGorillas.

The research team at VideoGorillas trains a unique recurrent neural network (RNN) for each project, accelerated by NVIDIA GPUs. The network learns the characteristics of titles created during the same era, in the same genre, using the same method of production. New content that is then passed through this network maintains the look and feel of that era/genre thus preserving artistic intent.

A generative adversarial network (GAN) is used to remove unwanted noise and artifacts in low resolution areas while replacing them with new image synthesis and upscaling.  The outcome is a model that can identify when visual loss is occurring.

The networks are trained with Pytorch using CUDA and cuDNN with millions of images per film. However, loading thousands of images is creating a bottleneck in their pipeline. VideoGorillas is thus integrating DALI (NVIDIA Data Loading Library) to accelerate training times.

NVIDIA RTX powered AI technologies

A cornerstone of video is the aggregation of visual information across adjacent frames. VideoGorillas uses Optical Flow to compute the relative motion of pixels between images. It provides frame consistency and minimizes any contextual or style loss within the image.

This new level of visual fidelity augmented by AI is only possible with NVIDIA RTX, which delivers 200x performance gains vs CPUs for their mixed precision and distributed training workflows. Video Gorillas trains super resolution networks with RTX 2080, and NVIDIA Quadro for larger-scale projects.

The extra power offered by NVIDIA Quadro enables VideoGorillas to apply super-res to HDR, high bit depth videos to up to 8K resolution, as well as achieve faster optical flow performance. The Tensor cores from the RTX GPUs provide a major boost in computing performance over CPUs, making them ideal for the mixed-precision training involved in VideoGorillas’ models.

“With CPUs, super resolution of videos to 4k and 8k is really not feasible – it’s just too slow to perform. NVIDIA GPUs are really the only option to achieve super resolution with higher image qualities” says Alex Zhukov, CTO VideoGorillas.

And while on-prem solutions work perfectly for their training needs, they also expand their workloads to the cloud using NVIDIA Kubernetes, running both in local data centers as well as Amazon Web Services and Google Cloud Platform to orchestrate Super Res inference jobs accelerated by NVIDIA Tesla V100 GPUs.

Categories
Post Production PVC Experts

After Effects Classic Course: Color Stabilizer

While discussing color-oriented plug-in effects, one of the more common tasks we’ve covered is fixing color imbalances in an image. Well, what do you do if that imbalance changes over time? You can keyframe your changes to match the changes in the footage and hope you don’t get odd pulsing or flickering…or in some cases, you can get away with the automated Color Stabilizer effect:

This movie previously appeared in our Insight Into Effects course on Lynda.com/LinkedIn Learning. They’ve retired that course from their library, so we’re making the movies from it available publicly for free. Click here for the playlist of previous movies we’ve made available.

Categories
News Production

Nitecore Superior Prime: a completely new line of Cinema lenses

Nitecore releases a complete line of Cinema lenses

Featuring interchangeable PL, Canon EF and Sony E mounts for the needs of different camera users, the Nitecore Superior Prime family of full frame lenses offers five focal lengths from 25 to 100mm.

Nitecore is a company known for its illumination products. The company claims it has the most innovative designs and best user experience and performance in the industry, and they may well be right. Nitecore introduced the world’s first AA-based flashlight that featured Infinite Variable Brightness (the Nitecore NDI), the world’s first flashlight that featured Smart Piston Drive System (D10), the world’s first flashlight series that featured 3rd generation Smart Selector Ring technology. Nitecore also won the 2014 ISPO Award Gold Winner CR6 with white/right dual powerful output; the 2013 China Outdoor Industry Award Winner HC50, just to mention some of the awards received.

Nitecore flashlights are innovative, robust, super effective and intuitive illumination tools, used by many outdoors enthusiasts, among others. If you’re a photographer using flashlights to “paint” your photos, you probably know the Nitecore name. Now, it’s time to also include the name  in the section for lens manufacturers, as the company announces the release of what it calls “the innovatively designed Superior Prime Full Frame Cinema Lenses”.

Nitecore releases a complete line of Cinema lenses

Five Nitcore cinema lenses

Nitecore is another name joining the lists of those companies that offer cinema lenses. No one knows if there is a market for all those lenses, but that does not seem to stop manufacturers from presenting their solutions. Nitecore has five lenses to show, 25mm T/2.1, 35mm T/2.0, 50mm T/2.0, 75mm T/2.0 and 100mm T/2.0, all offering “an optimum optical quality and allow filmmakers to best attain their techniques of artistic expression.”

These full frame cinema lenses presented under the brand name Superior Prime (SP)  create, says the company, “more possibilities for the filmmakers and cover the full frame format which is fully compatible with ARRI Alexa LF, Red Monstro VV, Panavision DXL2, SONY Venice, Canon C700 FF and other film cameras to be released in the future.”

From PL mount to Canon EF or Sony E mounts

Also according to Nitecore, “the apochromatic optical design provides outstanding performance in terms of ratio and axial dispersion. It achieves a natural and pure color rendering capability with the purple/green fringing hardly seen either within focus or in bokeh.”

Designed with a unique optical coating which ideally controls dispersion while retaining plentiful details to create low contrast and a stylized flare, continues Nitecore, the SP 75mm lens is specially optimized on the skin details in a portrait.

The new lenses are offered in PL mount, but Nitecore SP Lenses also have a high flexibility to be equipped with interchangeable PL, Canon EF and Sony E mounts for the needs of different camera users, says the company.

Categories
Post Production Production

Review: Blackmagic Pocket 6K

The new Super35 6K Pocket Cinema Camera from Blackmagic was delivered into a few lucky people’s hands last week, and my friends at Stray Angel Films were one of them. They gave me a week with it, shooting some stuff for my artist friend Chase Lock surrounding his new gallery at Olympia Club in Santa Monica, and I’ve put together a sort of review/overview for your edutainment. I haven’t used the Pocket 4K before, so these are wholly “first impressions” with minimal comparison as the cameras are functionally identical anyway. There’s a shared user manual you can check out here, and you can see the footage in the video embedded at the end of this writeup.



To start, the camera is quite easy to use. The button layout and menu all are quite intuitive and I never found myself “searching” for features. At times the touch screen on the back could be “too simple”, where I’d out-think myself (“Where’d that go again? Oh there’s another page!”) but once that happens the first time you don’t forget. While recording (or not) you can change ISO, White Balance, Shutter Speed, F-Stop, trigger the one-shot AF and AE, or even take a 21mp DNG still image. There are redundancies, so if you’re more of a touch screen person than a button-pusher, you’ve largely got those options. There’s also a “Slate” feature you can access by swiping left or right on the touchscreen where you’ve got lens and filter data, Reel/Scene/Take, a “good take” toggle, shot type, Int/Ext and Day/Night toggles. There’s also an overall “Project” metadata section where you can put in Project Name, Director, Cam Op, and Camera Designation (A, B, etc). That was fun to find.

When I pulled the camera out of the box, it was already set at 6K BRAW, 8:1 compression, 24fps, 180 shutter, and 400ISO. Exactly where I was going to put it. I quickly threw a 256GB CFast card in there, my Sigma 18-35mm, and was ready to roll. A nice feature I noticed with the Pocket 6K was a sort of “dummy proofing” of the Card Format feature, where you have to hold a button down for a 3-second countdown. Said feature is easy to find, simply by tapping the UI where it shows your card/time remaining. You can format to exFAT or OSX Extended, depending on how you roll, and can use either a CFast card or SD card. If you go the SD route, you’ll want to make sure it’s as fast as you can afford or else the camera might not be able to write to it depending on the size/speed of your footage. The safe bet is to stick with CFast or get a USB-C SSD to record to (which honestly might be cheaper). The Samsung T5 is a popular choice but I believe most drives will work, such as my beloved GDrive Mobile SSD.



You can record to BRAW or ProRes, but some formats are restricted to one or the other. Essentially if you’re shooting in the 6K neighborhood you’re in BRAW, if you’re shooting 4K or under you’re in ProRes land. From my experimentation, the file sizes are basically the same so you might as well shoot BRAW. Surprisingly, my PC had absolutely zero issues editing the 6K raw files. Not a single glitch, hiccup, or freeze. However, you do have to edit said raw files in Resolve (which you get for free when you buy a Blackmagic camera). It is possible to edit in Premiere but that requires a third-party, pay-for plugin. Which sucks. Depending on your situation, ProRes might be the move. For some of my Filmtools reviews, I just shot 4K ProRes because I a) wanted to edit in Premiere and b) didn’t need to be doing any color corrections that raw would help out with.

In regards to coloring the BRAW files, I found them incredibly easy to edit and very flexible. There’s a surprising amount of data in those raw files, enough to where technically I just had to make sure the histogram wasn’t pinned in either direction and I was good to go. The sky never blew out and the shadows never went black. Even at night, I was shocked at how well the camera handled (which I’ll get to in a moment).

Working with the camera, I just had a simple half-cage with a top-handle, complete with 15mm rod support, and was using the Steadicam AIR for support (which will be reviewed in a later article). I didn’t find myself wanting for anything else for the most part, but an external monitor or eyepiece would have been nice as the screen on the back is highly reflective and you can’t reposition it, which can cause issues when you’re trying to check focus or frame up your shot if you’re not directly behind the camera. Pulling focus was easy enough with my hand but if I was in a more professional shooting situation I would have hooked up my MicroRemote follow focus.



In regards to the rails, I was using them to hold on to and get some gentle purchase on the focus ring while moving and for holding the battery plate hidden back there.

While I was shooting I didn’t find myself asking any “dumb questions” like “where is Feature X” or “how do you…” which I believe speaks to the intuitive nature of the touchscreen. However I did find myself asking “why does the AF suck so much!?” I would hit the AF button, and the hunt would begin. And then it would catch focus aaannnddd lose it again. And then give up. The camera doesn’t have too many downsides that I could find, but the Autofocus is truly abysmal. Oh well. Auto Exposure seemed to do its job but I didn’t use it.

While I opted to use a cinema battery -DTap to 12v- which lasted me all day, I did run a test to see if the reported poor battery life of the Pocket 4K had carried over. It has.

I turned on the camera and started recording to a freshly formatted 256GB CFast card and got the promised 43minutes of 8:1 raw stuffed in there before the camera died 7 minutes later. Assuming you’ll have the camera on but not running for longer than that, I highly recommend either investing in tons of LP-E6 batteries or simply getting a nice big 150w cinema battery with a DTap port. I had mine on a plate which was attached to the camera (giving it a bit of extra heft, which was a nice addition to the resulting footage) but you could easily chuck it in a backpack or side bag or something if you wanted. Go crazy.

Another issue that will require a proverbial tissue is the lack of an IR cut filter in the body. While testing the Atlas Orion EF Anamorphics, we noticed a horrendous amount of IR interference when using our ND 1.8. To show how this is unique to the 6K (and all Blackmagic cameras not named “Ursa Mini Pro”), we made a video running through a few strengths of ND/IRND/WSNDs on the Pocket 6K, Arri Amira, and C100mkII. The aforementioned tissue you’ll need to get is either an IRND of some kind (different ones perform better or worse, that’s a test for a different day) or a dedicated IR cut filter that you’ll combine with whatever ND you’re using.

 

Aside from the infrared pollution (or perhaps including it) I found the images and colors I got from the Pocket 6K to be very pleasing, although perhaps not entirely “accurate”. I say this with as little emphasis as possible as I didn’t find it to be “bad” at all and rather enjoy the “stock” look of the Pocket 6K, but you may need to do a bit of color correction if you hit a situation where color accuracy is paramount, in my case the paintings I was filming. Creatively (insofar as the video is concerned) it didn’t necessarily matter, but the artist noticed immediately and his point of “it should look like the paintings look” outweighed my counter-point of “but it’s pretty already!” Lesson learned. It’s also just a matter of making sure you take care to color your images, I was rushing. In regards to other imaging issues, I did notice a touch of rolling shutter when panning around quickly, but nothing to write home about. There can also be a bit of moire in certain situations that a OLPF might take care of, but I didn’t notice it often. The one time I’m thinking of was on a canvas that was side-lit.

The really impressive feature of the 6K was the low-light performance. Something everyone begs for but rarely is delivered on. At night we went into Downtown Los Angeles and filmed some extra stuff for the promo, and at 3200 ISO I was astounded at how clean and legible the picture was. There’s some noise, but honestly, it’s not fixed pattern (so it looks “filmic”) and if you’re not looking for it you really don’t see it. The camera apparently goes up to 25,000 ISO but I didn’t find the need to go above 3200. Perhaps if I was filming in ProRes, but in raw I had plenty of information.


The main thing here that I think people should key in to is that this camera isn’t wholly better than the 4K, but it is a direct upgrade. The Super35 sensor is essentially a film industry standard, where M43 is not, and the EF mount is just as ubiquitous. Using a Speedbooster may get you an extra stop of exposure, but I can’t say this camera needs it and that means you’ve invested in EF glass anyway. Plus you’re putting more glass between your lens and your sensor, which could potentially degrade your image depending on the quality of it. By going for the S35/EF standard, Blackmagic has brought the Pocket Cinema line into a weird place where it’s not really a “cinema” camera per se, but it sure smells like one. As an example, there are very few budget cameras out there that have an Anamorphic mode. The 6K does. With the aforementioned Atlas Orion lenses, you’ve got a very attractive package for very little relative cost if you’re looking to shoot your next project Anamorphic. At $8,000 per lens, the Orions are vastly cheaper than their brethren and can also be re-mounted to fit PL or even E/M43 (even though your camera probably wouldn’t have a 4:3 mode to use it with). That being said, if you already own a Pocket 4K I don’t know if you need to rush out and upgrade right now. The 4K is still a fantastic little camera. However, if you’re in the market for either/or, there’s no contest just get the 6K.

If you just wanted to go the usual route of spherical lenses, and want to know the general cost of ownership (which is essentially the same as the 4K), I calculated it out to be around $4500:

BMPCC6K BMPCC4K
Camera Body  2500  1330
Sigma 18-35mm  700  700
Speedbooster  650
Cage w/ Rails  464  464
77mm IRND  121  121
XLR Mini Cable  24  24
DTAP Cable  35  35
Samsung T5  164  164
Battery Plate  155  155
Steadicam AIR  400  400
TOTAL  4563  4043

So while the Pocket 6K isn’t necessarily a full-fledged cinema camera in regards to its chances on a backlot, and it has a few rough spots, it does create a really impressive image that rivals cameras 10 times its cost. I’m inclined to do some tests and see how it handles against the main contenders and where it falls apart.