Adobe is making a significant leap into generative AI video technology.
Explore the #CreativeCloud and learn from Adobe's vibrant community of creatives at #AdobeLive! What's on today ⬇️ 🧵
— Adobe Creative Cloud (@creativecloud) October 22, 2024
The company’s Firefly Video Model, initially announced earlier this year, is now available across several new tools, including some integrated directly into Premiere Pro. These new features will enable creatives to extend footage and generate videos from still images and text prompts.
The first tool, Generative Extend, is launching in beta for Premiere Pro.
That's a wrap! 🎬 This year at #AdobeMAX we revealed incredible new tools & basked in the amazing creativity all around us. Whether you joined us in person or online, we're honored to have you as part of our creative community–see you next year! https://t.co/nZrvwBfjWs pic.twitter.com/qHAfTI3Cy3
— Adobe Creative Cloud (@creativecloud) October 21, 2024
This tool can extend the beginning or end of footage that’s slightly too short or make adjustments mid-shot, such as correcting shifting eye-lines or unexpected movements. Though the tool is limited to extending clips by only two seconds, it can be a valuable asset for minor tweaks, thereby reducing the need for retakes.
The extended clips can be generated in either 720p or 1080p at 24 FPS.
Our hearts are filled to the #AdobeMAX over your overwhelming response to Generative Extend in #PremierePro (beta) and #AdobeFirefly's Generate Video (beta). (Also, haiiiii, @JoelMcHale) pic.twitter.com/OFTKWuyVdm
— Adobe Video & Motion (@AdobeVideo) October 22, 2024
Generative Extend also works on audio to smooth out edits, extending sound effects and ambient “room tone” by up to ten seconds, though it does not support spoken dialogue or music. Additional tools, Text-to-Video and Image-to-Video, are launching on the web as part of a limited public beta in the Firefly web app.
Text-to-Video allows users to generate video by plugging in a text description, supporting various styles such as “real” film, 3D animation, and stop motion. Users can refine generated clips using a selection of camera controls that mimic camera angles, motion, and shooting distance.
Firefly expands creative video tools
Image-to-Video goes further by allowing users to add a reference image alongside a text prompt for more control over the results. Adobe suggests this could be used to create b-roll from images and photographs or to visualize reshoots by uploading a still from an existing video. However, the technology is not yet capable of replacing reshoots entirely, as several visual inconsistencies, like wobbling cables and shifting backgrounds, can still occur.
Currently, the maximum length for clips generated by Text-to-Video and Image-to-Video is five seconds, with quality limited to 720p and 24 frames per second. In comparison, OpenAI’s Sora can generate videos up to a minute long while maintaining visual quality and following user prompts. Adobe’s tools take about 90 seconds to generate content, but the company is working on a “turbo mode” to speed up this process.
Adobe asserts that its AI video model is “commercially safe” because it is trained on content that the company has the right to use. This could be a crucial advantage, given that models from other providers like Runway have faced scrutiny over their training data sources. Another benefit is that videos created or edited using Adobe’s Firefly video model can carry metadata to disclose AI usage and ownership rights when published online.
The AI video tools were announced at Adobe’s MAX conference, where the company also introduced updates across its creative apps. Although the tools are still in beta, they are publicly available, offering a glimpse into the future of AI-assisted video creation.
