4: Preprocessing with Roto Brush and Topaz Video AI

Why Start with These Tools?

I chose to begin with Roto Brush 2 and Topaz Video AI because they are accessible, well-documented, and integrate smoothly with my existing workflow. By starting with these, I can quickly assess their impact before considering more advanced or resource-intensive options like Boris FX Silhouette or delving deeper into Topaz's settings.

Experimenting with Adobe After Effects Roto Brush 2

Roto Brush 2 is designed to simplify the rotoscoping process by allowing users to easily separate foreground elements from the background. This separation could be crucial when applying generative AI models, as it can help the models focus on the primary subjects without background distractions.

Initial Observations

• Ease of Use: Roto Brush 2 proved to be quite user-friendly. With just a few clicks and some refinement, I was able to successfully isolate the main character in the first shot.

• Challenges with Complex Scenes: The second shot presented more difficulties. Highlights on the subject caused some issues, with certain areas being lost during the rotoscoping process.

• Hair Handling: Characters without hair or with simple hairstyles were easier to process. Complex hair, especially when interacting with light, posed a challenge.
• Light Artifacts and Bokeh: Scenes with light artifacts, like bokeh effects, confused the auto-rotoscoping. The tool sometimes misinterpreted these highlights as part of the subject, requiring additional manual corrections.

An important consideration is how the imperfections in the matte created by Roto Brush 2 might affect the downstream AI tool. The success of the generative model may depend on its ability to handle or compensate for these imperfections. This will be a key point of analysis in the next phase of testing.

Experimenting with Topaz Video AI

Topaz Video AI is a powerful tool for enhancing video quality, offering features like upscaling, motion blur reduction, and noise reduction. I want to see if preprocessing the footage with Topaz can provide better inputs for the generative AI model.

Initial Observations

• Quality Enhancement: The software was effective in recovering details from footage, especially in areas affected by motion blur.

• System Resource Demands: A significant drawback was the strain on my computer system. Processing high-resolution footage with Topaz was resource-intensive and time-consuming. On average, a 5-second clip would take over five minutes to process.

Deciding on When to Use Topaz

Given the resource demands, I plan to limit the use of Topaz Video AI to specific footage where quality enhancement is most needed, such as:

• Day and Night Footage: Scenes shot in varying lighting conditions that may benefit from noise reduction and detail enhancement.
• Footage with Motion Blur: Clips where motion blur significantly impacts the clarity of the subject.

Additionally, I'm considering whether it's more effective to apply Topaz processing before or after green screen compositing. This will be part of the next set of experiments.

Key Takeaways So Far

• Roto Brush 2 Effectiveness: While not perfect, Roto Brush 2 offers a relatively quick way to separate subjects from the background. However, complex elements like hair and light artifacts require additional manual work.
• Impact on Workflow: The time saved using Roto Brush 2 compared to manual rotoscoping is substantial, but the trade-off is in the quality of the matte, which may affect AI processing later.
• Topaz Video AI Benefits and Drawbacks: Significant improvement in image quality and detail recovery, which could enhance the performance of the generative AI model. However, the high demand on system resources and increased processing time potentially limit its use in the pipeline.