2: Exploring the Tools for my AI-Assisted Production Pipeline
Before getting started with experimentation, it is important to define the tools that I will be testing. For this research, I will be using both generative AI tools and machine learning (ML) tools, so I would like to provide a quick definition to be on the same page.
Generative AI tools are those that "generate" new content, either from a text prompt, extending an image or even video to video transformations in which the result is something new. On the other hand, ML tools often work behind the scenes; they typically have specific functions like scene detection, noise reduction, or object recognition. ML tools have been used long before generative AI tools; recent iterations have become ever more powerful, and I believe they could aid in reducing generative AI limitations and improving the overall results.
With the above, let's continue with the tools:
Generative AI Tools
RunwayML
RunwayML offers a suite of AI-powered creative tools. For this series, I will be focusing on using the video-to-video feature from the latest model Gen-3.
This new model allows for footage to be input before the generation process, enabling a high degree of control on the results. Here is an example from their site:
| Input | Prompt | Output |
| 3D halftone CMYK style. halftone print dot. comic book. vibrant colors in layers of cyan blue, yellow, magenta purple, and black circular dots. |
Why RunwayML?
- User-Friendly: Designed for artists without deep technical expertise.
- Powerful: Likely one of the most powerful tools available for video-to-video transformations.
- Real-Time Processing: Great for quick testing and iteration.
Cost and Accessibility:
Free tier available with limitations, including the model I wish to use. Paid plans start at around $15/month (10 seconds of video) up to $99/month (unlimited). There is a 20% discount for students on any plan.
Preprocessing and Post-Processing ML Tools
To address the limitations of the tools mentioned above and enhance the results, I will incorporate ML tools for pre- and post-processing the footage. My focus will be on comparing the result of rotoscoped material against that of raw footage. Additionally, I plan to experiment with various levels of video quality and resolution to evaluate how these factors influence the final results.
Adobe After Effects Roto Brush 2
Automates rotoscoping, the process of isolating elements within video frames. This will be a quick way to preprocess the material to feed into the AI tools.
Why Roto Brush 2?
- Efficient: Speeds up rotoscoping compared to manual methods.
- Integrated: Fits into my existing Adobe Creative Cloud workflow.
- Precise: Great at separating subjects from backgrounds, allowing auto-tracking and frame-by-frame adjustments.
Cost and Accessibility:
Available through Adobe's Creative Cloud subscription at around €19.99 a month for students (yearly commitment).
Boris FX Silhouette
A professional rotoscoping and paint tool with advanced features for VFX and post-production. I plan to test how it compares with Roto Brush 2 in both workflow and quality.
Why Silhouette/Mocha Pro?
- Advanced Rotoscoping: Powerful tools for complex tasks.
- Motion Tracking: Improves accuracy by tracking movement.
- Industry Standard: Trusted in professional VFX workflows.
Cost and Accessibility:
Subscription starts at $66 a month, with an educational license for full-time students offering a year of all BorisFX tools for free.
Topaz Video AI
Enhances video quality using machine learning for upscaling, deinterlacing, motion interpolation, and noise reduction. This tool helps test output quality differences and mitigates RunwayML’s 720p limitation.
(See more here)
Why Topaz Video AI?
- Quality Boost: Improves resolution and clarity, upscaling material up to 16K.
- Noise Reduction: Cleans up footage for better AI processing.
- Industry Standard: Used in professional VFX workflows.
Cost and Accessibility:
One-time purchase at $299.99; a free trial is available.
Wrapping Up
I'm excited to test these tools and see how they can support my project. The big question is how well they'll work together in a seamless workflow. Only by experimenting will I be able to assess compatibility, optimize the workflow, and evaluate the quality.
In the next article, I'll set up a testing framework to evaluate each tool and how they connect.
Continue to the next article:
3: Setting Up the Experimentation Framework