6: Analysing AI Generated Results

Daylight Footage

Raw Footage

Processing the raw daylight footage provided underwhelming results:

• Color Representation: Colors appeared slight and pastel, lacking vibrancy.
• Detail Loss: Significant loss of detail, especially around the face.

The outputs were flat and lacked the depth needed for the desired aesthetic. This confirmed that color correcting the footage was a necessary step.

Color Corrected Footage

Applying color correction before processing made a noticeable difference:

• Improved Details: Enhanced facial features and overall details.
• Color Matching: The character's colors were much closer to the reference, making the output more faithful to the original footage.
• Background Issues: Backgrounds remained distinct from each other, posing challenges for matching them later.

The results were pleasing, and the enhanced colors contributed to a more engaging visual style.

Topaz Enhanced Color Corrected Footage

Applying Topaz Video AI to the color corrected footage:

• Minimal Differences: Outputs were very similar to the previous step.
• Slightly Vivid Colors: A minor increase in color vibrancy, but not significantly noticeable.

The enhancements didn't offer enough difference to justify the additional processing time.

Rotoscoped Footage

Processing the rotoscoped footage led to significant improvements:

• Detail Level: Substantial increase in character detail.
• Color Consistency: Colors were more consistent and matched well across different shots.

Isolating the character allowed the AI to focus on the main subject without background distractions, which seems to be the right approach. This raises the question of how to handle or generate the backgrounds moving forward, presenting an interesting creative opportunity.

Rotoscoped Footage with Topaz Enhancement

Adding Topaz Video AI to the rotoscoped footage:

• Little Difference: No significant improvements over the previous step.
• Runway's Capabilities: RunwayML handles small defects in the matte effectively without additional enhancement.

Again, the minimal benefits may not justify the extra processing time.

Nighttime Footage

For the nighttime footage, I focused on processing the color corrected and rotoscoped versions, as testing raw footage was deemed unnecessary based on previous observations.

Color Corrected Footage

Processing the color corrected nighttime footage:

• Varied Results: Outputs were more varied compared to the daytime footage.
• Feature Alterations: Facial features changed noticeably; shadows sometimes turned into unintended elements like beards.
• Consistency Issues: The AI had difficulty maintaining consistent representations of the character.

The results were less reliable, indicating challenges with nighttime footage, possibly due to basic lighting with hard shadows and blown-out highlights.

Topaz Enhanced Color Corrected Footage

Applying Topaz Video AI to the color corrected nighttime footage:

• Similar Results: Outputs were very similar to those without Topaz enhancement.
• Footage Quality: Since the original footage wasn't low quality or out of focus, Topaz didn't contribute significant improvements.

Topaz may be more beneficial for rescuing lower quality footage rather than enhancing already decent footage.

Rotoscoped Footage

Processing the rotoscoped nighttime footage:

• Improved Detail: Better detail in the character, similar to the daytime footage.
• Shadow Handling: The AI did not mistakenly turn shadows into unintended features.
• Greenscreen Challenges: Harder to get RunwayML to handle the greenscreen, leading to potential issues requiring additional rotoscoping or keying.

While there were improvements, nighttime footage presented more challenges than daytime footage.

Rotoscoped Footage with Topaz Enhancement

Applying Topaz to the rotoscoped nighttime footage:

• Slight Increase in Detail: A minor improvement.
• Color Management Issues: Maintaining consistent color was difficult; fully separating the greenscreen proved challenging.

The benefits did not significantly outweigh the drawbacks.

Conclusions

Importance of Color Correction

• Consistency: Setting up contrast, saturation, and matching shots is crucial for achieving consistent, high-quality outputs.
• Future Tests: Pushing colors and contrast further might yield even better results.

Need for Background Separation

• Keying Out Backgrounds: Keying out backgrounds may be necessary, especially with nighttime footage.
• Greenscreen Use: Shooting on a greenscreen could simplify background removal and improve AI focus on the subject.
• Creative Exploration: Deciding how to handle backgrounds offers opportunities for narrative and creative development.

Daytime vs. Nighttime Footage

• Daytime Advantages: Daytime footage provided better and more consistent results, suggesting shooting in well-lit conditions is preferable.

Shot Composition Matters

• Close-Up Shots: Medium and close-up shots provided more intricate detail for the character.
• Processing Variations: Processing the same footage twice—once focusing on the face and once on the full shot—could allow for combining them to enhance facial expressions.

Next Steps

• Further Testing: Experiment with close-up shots to see if they improve facial detail and expression capture. Tweaking color and contrast could also be useful.
• Background Generation: Explore methods for generating or incorporating backgrounds to complement the rotoscoped characters.
• Pipeline Refinement: Adjust the pipeline to optimize quality and efficiency, possibly omitting unnecessary preprocessing steps like Topaz enhancement unless dealing with low-quality footage.