Going Deep with Adobe Firefly

Where were you when you used Adobe Firefly for the first time? For many designers, it’s a “flashbulb” memory that will never be forgotten.

Mine occurred when I first used Generative Fill on one of my vacation photos. I opened an image of islands near the Norwegian coast, casually whipped out the Rectangular Marquee tool, and prompted Photoshop to add a whale breaching the water surface. When I saw the result a few seconds later, I was astonished. I knew right away that this technology was going to have a massive effect on how people would perceive and perform the job of editing images in Photoshop.

From Sensei to Firefly

So, how did we get here? If you’ve been using Photoshop for a while, you probably remember hearing about Adobe Sensei, which referred to a family of machine-learning algorithms built to help you do your work faster. One example is the Content-Aware Fill feature in Photoshop, which allows you to select and remove subjects with a click. It does so by finding patterns in your image that look similar to the area you have selected, and then replicating that pattern until the selected area is filled.

The quality of results you get with Content-Aware Fill depends on the presence of enough source material to work with. If your selection is small or your image offers a lot of available source material, the result can be seamless. But if the image is too small, however, or if the selected area is larger than the available source material in the image, the results can vary from poor to awful.

The principle of generative AI is similar but takes away the limitation of being able to look at only the source material present in the image itself. Instead, the algorithm can generate more image content, based on a massive database of information that is available in the background (hence the term generative).

Because of this, an internet connection is required when using the Adobe Firefly features within Photoshop such as Generative Fill or Generative Expand. But the results can be mind-blowing (Figure 1).

Figure 1. Suppose you need to remove the ceiling lights in the original image. You can clearly see how the limited available source material in the image is hampering Content-Aware Fill’s capabilities. Generative Fill in Photoshop (using Adobe Firefly in the background) did a far better job by generating additional content from scratch.

Still, the addition of generative AI doesn’t negate existing Sensei-driven features within Photoshop (like Content-Aware Fill). There are many ways to achieve the results you want in Photoshop and knowing which tools you have available within your arsenal will always remain a key element to success.

Behind the Scenes: The Legal and Ethical Foundation of Firefly

Any AI algorithm is only as strong as the information database that feeds it. Adobe Firefly is no exception. This means that in order to better understand the results and behaviors of Adobe Firefly, you need to understand how it was trained. Firefly uses two main sources of information:

• Adobe Stock: The Adobe Stock database, rich with hundreds of millions of images (and thousands more added every day), is key to the development of Adobe Firefly. Note that within the Adobe Stock database all “editorial” images featuring real people and brands, as well as images generated using third-party AI solutions are excluded from the training set.
• Public images: These are images available in the public domain, where the copyright has either expired or never existed.

It is Adobe’s policy to exclude copyrighted images and any other kind of intellectual property from Firefly’s training set. In fact, asking Firefly to generate images of Mickey Mouse, a Porsche, Coca-Cola bottles, or other types of intellectual property will result in an error message as brands and characters get flagged (Figure 2).

Figure 2. Adobe Firefly is flagging inappropriate keywords in my prompt, in this case Superman because it is protected content.

Some might argue that this could limit creativity. But Adobe builds tools for professional users, so they understand that Firefly’s output is going to be used for commercial purposes. Because of this, results generated using Firefly are considered “commercially safe” from a legal standpoint. And Adobe offers indemnification for enterprise customers in case coincidental similarities in generated results do occur. On top of that, none of your prompts or uploaded images are indexed by Adobe nor are they used to train Firefly’s algorithm (as opposed to other players out there whose algorithms are trained by scraping the internet without regard for consent, credit, or compensation to the original creators). The only exceptions are when you want to flag or report generated results to Adobe, or when you use certain exclusive features for enterprise customers (more about those later).

In essence, this protection exists to protect users from what they don’t know. We all know that trying to generate images of Homer Simpson working out on a Peloton bike would break many IP rules (plus, it doesn’t sound like anything he would do). But this protection is especially important during situations when you honestly don’t realize you’re doing something wrong. One of my customers (a manufacturer of sweets and chocolates) recently mentioned how they almost got into trouble by using third-party solutions. As part of a campaign image, they created a scene that featured their sweets positioned next to an innocent cup of coffee (i.e., a normal looking cup and saucer). But little did they know that the design of the porcelain coffee cup was trademarked by a famous coffee brand. Luckily, they caught it in time, and the crisis was averted. The overall advice would be to default to a safer system. Campaign images can be taken down after launch, but brand damage would take years to repair.

Business Impact of Generative AI

Speaking to customers across many different industries it’s easy to see patterns emerge in the wishes and dreams of these organizations. And it’s clear that while 2023 was the year of emerging technology and experimentation, 2024 and 2025 are the years when organizations want to get serious by investing in AI technology, reorganize their business, and change how they work, create, and collaborate. This also means adopting solutions like Adobe Firefly.

Organizations see a lot of value in Firefly and similar solutions, partly because they need to generate more content faster than ever—banners, ads, social media graphics, marketing copy, video and animation, games,  and anything printable. Three drivers fuel this need:

• Hyper personalization: Different content and ads need to be generated according to audience segmentation. Maybe you want to show one version of an ad to a 60-year-old who enjoys the company of dogs, going to the opera, and drinking craft beers, but offer different content to a 23-year-old who likes hiking, chess, and Netflix documentaries. You can imagine how distinct the tone of voice and images need to be.
• Image relevance: A company can no longer afford to re-use campaign images from 2024 in a campaign set to launch in 2025; reinventing campaign images and having a fresh look are important. Market research made clear that consumers are expecting an average of eight product images before they are willing to buy online.
• Variety of channels: Millennials might prefer Facebook, while Gen Z prefers to watch clips on TikTok. A presence on both sites is needed—as are different design specifications

It’s also about basic math. If you are selling 6 unique products or services and need 12 unique assets per product in 15 different channel sizes and in 6 different languages, you quickly end up requiring thousands of unique assets.

Needless to say, producing this volume of content using only traditional methods (photographer, stock images, designing from scratch) is challenging if not impossible for many companies. This creates an opportunity where organizations are using Adobe Firefly to streamline their campaigns and produce imagery faster (Figure 3).

Figure 3. Mattel uses Adobe Firefly to transition from traditional sketches to generated concepts to speed up ideation and approval processes. Source

The main applications for Adobe Firefly—and thus the biggest wins—can be found in:

• Ideation: It is incredibly easy to sketch and ideate by prompting Firefly. You can generate multiple designs for a campaign image at once or create mood boards, which in turn results in quicker approvals and reduced production time.
• Image manipulation: Using Adobe Firefly within Photoshop allows you to quickly manipulate existing images as part of a design proposal. There’s no need to find the appropriate stock images and edit them by changing their perspective, masking, adding shadows and reflections, or recoloring them. Building compositions has never been easier and more accessible to users with an intermediate level of Photoshop skills.
• Change implementation: Need to add or remove objects in Photoshop, or simply expand an image for use in different aspect ratios? With Firefly you can quickly and easily make alterations to existing artwork based on customer feedback.

Getting Started

Adobe Firefly’s technology is available across many different platforms within Creative Cloud—Adobe Express, Photoshop, Illustrator, InDesign, Premiere Pro—and from your web browser. Depending on your industry, the two best places to start are the Adobe Firefly webpage and Photoshop.

Creating a prompt

Creating and adjusting an image from your browser is easy.

Open a browser, and navigate to the Adobe Firefly webpage.

Log in to your Creative Cloud account.

Click the Text to Image card to view the gallery and start creating your first prompt.

In the Prompt bar, enter a description, such as a coffee travel cup standing in a huge pile of coffee beans, dark brown gradient background, and then wait for the results to load.

To see a larger version of one of the results, click it. You can also use the arrow keys to cycle through the different generations.

To make changes, select the options you want to adjust (e.g., choose Landscape (4:3) from the Aspect Ratio menu), and then click the Generate button to refresh the results.

To access the Effects, Color and Tone, Lighting, and Camera Angle controls, scroll down. Try changing a few of the visual options while you’re here.

When done, save your image to your hard drive or add it to a Creative Cloud Library.

Prompting best practices

At first, it can be tricky to point Adobe Firefly in the right direction, and you may get unexpected results (sometimes called hallucinations) if your prompt is too general. To make Firefly better interpret a request, you can use a combination of written instructions (a text prompt), settings (like effects), and visual input (like style and composition references). That way, it’s much more likely to generate the kind of image you’re looking for.

Prompt writing is an art in itself. Some words might be confusing, too vague, too specific, or even contradictory (When you ask for a “close-up” in your prompt but select Wide Angle for your Camera Angle setting, which one wins?). Luckily, there are a few tips you can take into consideration when writing your prompts.

Always remember that any detail you exclude from your prompt is a detail that you leave up to Firefly to interpret in its own way. So, if you are generating an image of a red car, you might get a red car driving on the beach because you didn’t specify it should have been driving on the road. Here are five qualities of a good prompt:

• Descriptive: Don’t give Firefly commands. Don’t say “create an image that …”, just describe what you want.
• Detailed: Describe all objects in the scene as you envision them, including the background of the scene.
• Specific: Leave as little room for interpretation as possible.
• Stylistic: Describe the visual style and color of both the subject and the background.
• Creative: Let your imagination run wild! You can also use subjective adjectives (fun, cozy, scary, exciting) and allow Firefly to interpret the prompt in its own way.

Figure 4 shows the results from the general and bland prompt: a car driving through a landscape.

Figure 4. Notice that the environment and details of the scene are either missing or interpreted in an unintended way because the original prompt didn’t include any specifics.

Figure 5 shows the results of a better version of the same prompt: a blue car driving through a snowy landscape during sunset, there are trees, mountains, and a lake visible in the background, the scene has beautiful orange tones as the sun colors the snow landscape during sunset, wide angle.

Figure 5. Notice how different this version of the image is.

Let’s break down why this prompt was effective:

• A blue car describes the subject.
• Driving through a snowy landscape during sunset, trees, mountains, and a lake visible in background describes the environment and details.
• Beautiful orange tones describes the visual style.
• Wide angle describes the camera angle.

Production Techniques

The possibilities of Adobe Firefly’s text-to-image, especially when integrated in apps like Photoshop or Express have allowed users to create things that were impossible to do before. And the more you experiment with existing features, the more “oh wow!” moments you’ll encounter. Here are a few examples to try out.

Use Generative Fill to clean up hair selections

Complex selections like hair, fur, foliage, and other textures are challenging to master, even for seasoned Photoshop users. Sometimes the source materials simply lack the required resolution, and other selections require more time or skill than you might have. Either way, you may end up with results like Figure 6.

Figure 6. Notice the purple color cast around the edge of the hair, a remnant of the original background.

When you see the dreaded lingering background outline, don’t despair. You can clean the fluffy subject’s outline easily now: Create a selection over the outer hair edge and run the Generative Fill command. The results will suddenly look very different—and better (Figure 7).

Figure 7. After using Generative Fill, the purple hue is gone and the hair has a much sharper edge.

Create unique text effects

By employing a combination of prompts and a quick sketch (expand Composition, expand Reference, then upload your image), you can use the Firefly website to create almost any image you want. Your reference image can be as simple or complex as you need.

For Figure 8, I used Illustrator to create a simple composition of characters that spell out the word dry, saved it as an external reference, then imported it to the Firefly website. I adjusted a few other settings, entered my prompt, and generated a result that would have been virtually impossible to make in the past, let alone in the handful of minutes it took me to complete the process.

Figure 8. Combining a sketch with a detailed prompt can yield amazing results.

Scaling Production

Adobe has reserved a few powerful production boosters for its enterprise customers.

Using proprietary content: Enterprise users can train Firefly on their own content, which helps them to ideate faster. In essence, they can generate more fresh content based on their existing content. They can create images featuring their products and brand’s distinct visual style, and then reuse the model across the organization (Figure 9).

Figure 9. You can input data (top) and prompt Firefly to mimic the same visual style. Using traditional photography this would take several hours to prepare and complete. After a one-time training, however, the algorithm generated 15 images (bottom) in around three hours.

Automation: Enterprise users can employ various Firefly APIs for automating prompts or expanding images in conjunction with other Photoshop APIs to form one seamless workflow (Figure 10).

Figure 10. An example of the kind of workflows enterprise customers can build with Firefly

Imagine starting with a database of images featuring a product, and then:

1. Removing the background
2. Generating a custom background
3. Expanding the background
4. Fitting the image into existing Photoshop templates
5. Exporting and renaming all files at once

These methods allow organizations to scale campaigns across markets, different target groups, and languages. In fact, these are exactly the same tools that Adobe uses internally to scale their campaigns. To give you an idea, over 52,000 unique assets were created for Adobe’s 2023 Black Friday campaign, over a period of five days (Figure 11).

Figure 11. Adobe uses the same tools they sell to large customers, for the same reasons.

An Intelligent Approach

It’s important to understand that generative AI is here to stay and that users will need to learn how to work with (or alongside) this new technology. So, how do you get started? Here are a few tips:

• Explore the Adobe Firefly website and practice prompting. Use the tips in this article to try out different approaches.
• Familiarize yourself with the Style and Composition Reference options as soon as you can. These allow you to direct your scene easily and will make for more effective prompting.
• Explore where and how Firefly is introduced in other applications like Illustrator, InDesign, or Express. There are unique generative AI features in many of these solutions that will quickly expand your existing toolset. For example, you can generate seamless vector patterns in Illustrator or use generative AI for copywriting your social media posts.
• Follow the latest updates by Adobe; they arrive often and are quickly released. At the time of this writing, Adobe is set to release their new text-to-video solution and have announced more AI capabilities for their Substance 3D products.
• Learn from others and help others learn by sharing ideas, prompts, and results.