Bart’s defringe/refringe object extraction method.
2006-Jan-21
This is a masking method I’ve been refining over the last few weeks and I haven’t seen it elsewhere. The ultimate goal is to extract a foreground object from an arbitrary background and convincingly blend it with any other arbitrary background. I’m hoping the folks here can help me troubleshoot it for any shortcomings—perhaps we can turn it into a real tutorial. It’s developed in Paintshop Pro X, but the methods are all pretty fundamental and can be applied with little or no translation to Photoshop and even Photoshop Elements with a few alterations.
The original image:
Step 1: Prerequisite—advanced object masking
We start with what I’ll call the fringy object. I’m calling it the fringy object because around the edges, the background color blended with the foreground colors due to blurriness in the photo (which makes the hair act like a semi-transparent material) and natural blending of backlight and forelight that occurs due to reflections.
The well-documented procedure (eg., Russel Brown tutorial) to arrive at the fringy object will be called step 1 and we’ll skip it since everybody already knows how to do it (prerequisite for this tutorial.) So we’ll start with step 2:
Step 2: Create the second mask—Mask 2
What we'll need to do to fix this is selectively remove the old background color (which is a flat blue in this case) from the fringy object. The fringy object is shown in figure 1. You can see the out-of-place bluish fringe around the edges.
Figure 1—fringy object (left) and fringy object on black background (right)
I will first create a mask that is light in color only in the places where the hair has the blue mixed into it (aka. the fringy area). Figure 2 shows how to do this with a channel mixer.
Figure 2: The making of Mask 2
In the layer palette (left side of figure 2) is "Raster 2" which is the fringy object on a black background, "Channel Mixer 1" which is channel mixer layer, and "Manual fix" which I use to manually paint parts of the mask that aren't anywhere near the delicate fringe areas and I want to make sure they are black.
Whenever we want to selectively mask something (in our case, the bluish fringy area), we must start by understanding what is distinguishing it from adjacent areas in the photo (non-adjacent areas can be hand-painted in the “Manual Fix” layer.) Relative to adjacent areas in the Raster 2 layer, the fringy area is relatively strong in blue and weak in red (easily seen by hovering the eyedropper around.) So to accentuate that distinguishing characteristic, we need to add more blue and subtract more red in the channel mixer which is what I've done to sort of an extreme in figure 2 (-111% red and +168% blue). How did I come up with these particular numbers? I knew I wanted red turned down and blue turned up so I just started moving the sliders with that in mind until the mask looked about right. You'll get a feel for it after you've done it a few times. Note that the “Monochrome” box is checked—this greyscales our result which is what we want for a mask. In other words, the mask is constructed from the formula -1.11R - .23G + 1.68R. (As a point of comparison, if you wanted to create a simple luminance-based B&W image, you’d use 0.30R + 0.59G + 0.11B.) A closeup of our mask-to-be from the figure 2 adjustments is shown in figure 3. From now, it’ll be known as “Mask 2”.
Figure 3: Mask 2
Note that Mask 2 is lighter in the areas that are more blue (ie., the fringy parts). You could do further tweaking on this using a curve tool—just like everything else, you’ll get the hang of it with practice.
Step 3: Defringe
Time to remove the fringe color using Mask 2. Figure 4 shows how the layer palette will look.
Figure 4
At the very bottom is the “New Background” layer—for now it’s just plain black. This gets combined with the “Composite” layer group which will be the defringed extracted object. Inside the Composite group, you see the fringy object from step 1—it’s created by applying Mask 1 to the original image. The next layer group is “Remove Old Fringe” which has a blend mode of multiply and opacity at about 80%. Inside of that group we have Mask 2 (from step 2) being applied to the inverse of the background color. Okay, what’s going on here?
A decent approximation to removing a certain hue of light from an RGB image is to multiply the image by the inverse of that color. This is what happens with subtractive color spaces such as CMY. You start with white and then remove the colors you don’t want. If you want to remove red, you invert it (to get cyan), mask it, and multiply that with the image. So that’s what I’m doing here—I created a blank raster layer, sampled and filled the old background color into it, then inverted it. That is then masked by Mask 2 (we only want to remove color in the fringy region) and the result is multiplied into the original fringy extracted object. Our now defringed object is shown in Figure 5
Figure 5: defringed object on a black background
So the fringing is gone and hair saturation and hue are reasonably well preserved. The nice thing here is you see in addition to removing the old background, we effectively added back in some of the eroded detail from the first mask due to the fact that the blue was overwhelming the faint hair strands. This restored detail also has the color fringing removed of course.
I always make Mask 2 a bit on the generous side (ie., err on the side of making it too bright.) This will actually cause too much fringing to be removed and then I can tune it down using the transparency slider on the “remove old fringe” group layer. In this case, it put it at 81%--I didn’t set it that precisely, it just landed there when I dragged it.
If we were only going to a black background, we’d be done. However, the more common application (or at least the more challenging application) is to put this object on a completely arbitrary background. Let’s do that next.
Step 4: Refringe
Let’s start by just putting our defringed object on a new background. Figure 6 shows how it looks if I replace the black background shown in figure 5 with a bright partially cloudy background.
Figure 6: Defringed object on new background
It looks sort of okay, but the edges of the hair look unnaturally dark and shadowy. Why is this? It’s because we removed the fringing color in step 3. If you think about it, that process makes the foreground object appear as though it were shot in front of a black background—we subtracted the old-background-fringing, but now we must add in new-background fringing so it looks like it was actually shot in front of the new background.
We need a fringing mask to do this—luckily we already made one in step 2—just reuse Mask 2. This time, instead of multiplying the inverted background, we will screen the non-inverted background. This is mathematically somewhat like removing ink from paper in the color of the new background.
Figure 7 shows the updated layer palette with this new layer group (“Add new fringe”). It is positioned right above “Remove old fringe” group layer.
Figure 7: Add New Fringe
Inside of this group we have the “new background” being masked by Mask 2 (an identical copy of the new background layer at the bottom of the stack.) I slid the opacity slider to 75% which seemed to be about right. The final result is shown in Figure 8.
Figure 8
That’s more like it! We’ve maintained hair detail and color while also getting the new background to appear convincingly even in the frilly nooks and crannies of the hair.
Okay, okay, you’ll notice it’s still not quite perfectly perfect. Part of the problem is jpeg artifacts—I started with a not-too-high quality image and these artifacts tend to show up in the mask creation process. A clean original won’t have this problem. On top of that, I haven’t had much practice doing this procedure yet myself.
In principle, the old background could be non-uniform as well—the fringe-removing layer can handle this case just as easily as the uniform old background. The extra challenge in that case would be generating Mask 2 in the presence of the non-uniform background (and thus, non-uniform fringing). I think this can be done, but I haven’t had a chance to work it out yet. You would also create a pseudo old background by cloning inward from the edges of the foreground object. That’s a topic for the future.
At this point, the layer stackup we’ve created will work with just about any background with no further adjustments—the necessary math is built into the layer stack. To plop in new backgrounds, just paste them into the two layers called “New Background”. Here are three quickie examples—all I did was paste them into the two instances of “New Background”—no further adjustments were made.
