3: Setting Up the Experimentation Framework

3: Setting Up the Experimentation Framework

With the tools selected for integrating generative AI into my short film production, it's time to establish a solid experimentation framework. This framework will guide my testing process, allowing me to evaluate each tool and determine the most effective workflow.

Types of Footage for Testing

I will be using three different types of footage to test the AI tools. The main purpose of picking distinct footage is to experiment with scene lighting, camera movement, subject movement, and footage quality. I will also be testing color corrected and raw versions of the same footage.

Daylight Shot, Natural Light: This includes three shots of the same character, captured in natural daylight. The character moves and the camera pans slightly.

Daylight Shot 1 Daylight Shot 2

Nighttime Shot, Basic Lighting: Three shots captured at night with basic lighting. The character moves towards the camera, and the camera follows the movement.

Night Shot 1 Night Shot 2

Handheld Sequence: A fast-paced handheld sequence with some shots losing focus.

Handheld Sequence 1 Handheld Sequence 2

Initial Testing Approach

The first phase involves inputting each type of raw footage into RunwayML without preprocessing to establish a baseline understanding of:

  • Tool Capabilities: How well the AI tool performs.
  • Natural Limitations: Identify issues linked to lighting, movement, and color management.

Data Collection

  • Document Settings: Record settings and parameters used.
  • Capture Outputs: Save output samples for comparison.
  • Note Observations: Track processing times, artifacts, and errors.

Next Step: Preprocessing

To optimize the footage for generative AI, the next step is to dive directly into preprocessing techniques, which will enhance the footage quality and provide the AI tool with cleaner inputs. This phase will focus on isolating the main subjects from their backgrounds and applying quality enhancements to ensure the footage is ready for AI processing.

Rotoscoping and Quality Enhancement

  • Rotoscoping Tool: Adobe After Effects Roto Brush 2 will be the primary tool for isolating characters from their backgrounds, allowing for more targeted AI generation by reducing distractions from background elements.
  • Quality Enhancement Tool: Topaz Video AI will be used to upscale footage, reduce noise, and improve clarity, helping to maximize the quality of AI outputs.

These preprocessing steps aim to streamline the AI workflow and improve the final aesthetic quality. In the next article, I’ll discuss the preprocessing results, detailing how the Roto Brush and Topaz Video AI influenced the AI tool's performance.

Continue to the next article:

4: Preprocessing with Roto Brush 2 and Topaz Video AI
Back to blog

References