Image Shadow Removal in Mathematica

Question

The task is:

Image shadow removal is an important topic in image processing. If you feel curious about it, you may take a look here, or just google for it image shadow removal

There are also a few questions on SE around it. For example:

• https://dsp.stackexchange.com/questions/2247/how-can-i-remove-shadows-from-an-image

• https://stackoverflow.com/questions/9081930/how-to-remove-the-shadow-in-image-by-using-opencv

However, I did not find any Mma implementations out there, and I don't want to reinvent the wheel if it is already rolling.

Does anybody know of some Mma codebase for using it as a starter?

Note: I have to do this for a few images, so I may go the manual way better than spend many hours in developing a full blown algorithm.

Answer 1

EDIT: I found a version of the source image with fewer jpeg artifacts.

My idea would be to extract pixels along the border of the shadow, inside and outside of the shadow. Then I have a list of pairs of RGB values, and I can find a suitable transformation from "shadowed" pixels to "non-shadowed" pixels.

But first, I must find the shadow area. I would apply a mean shift filter to remove the texture:

im = Import["http://www.cs.sfu.ca/~mark/ftp/Eccv04/path.jpg"]
ms = ms = MeanShiftFilter[GaussianFilter[im, 2], 10, 0.1, MaxIterations -> 5]

I can simply binarize this with default thresholds:

bin = DeleteSmallComponents[ColorNegate[DeleteSmallComponents[Binarize[ms]]]]

Next step: Extract edges and normal directions:

edges = EdgeDetect[bin];
dx = GaussianFilter[ImageData[bin], 5, {0, 1}];
dy = GaussianFilter[ImageData[bin], 5, {1, 0}];

edgePixels = Position[ImageData[edges], 1];
normalDirection = Transpose[{Extract[dy, edgePixels], Extract[dx, edgePixels]}];

Now edgePixels contains a list of edge pixel indices and normalDirection contains a list of normal vectors for each of these edge pixels.

Next step: for each border pixel, go 40 pixels in the normal direction, and get the RGB value at that point. Go 40 pixels the other way, and pick up that RGB value, too:

pixels = ImageData[GaussianFilter[im, 5]];
colorsInsideShadow = Extract[pixels, Round[edgePixels + normalDirection*40]];
colorsOutsideShadow = Extract[pixels, Round[edgePixels - normalDirection*40]];

Just to illustrate, the colors picked up look like this:

ImageAssemble[{{Image[ConstantArray[colorsInsideShadow, 10]]}, {Image[ConstantArray[colorsOutsideShadow, 10]]}}]

I would simply try a linear model from colors "inside the shadow" to colors "outside the shadow":

rgbToModel[{r_, g_, b_}] := {1, r, g, b}
colorTransform = LeastSquares[rgbToModel/@colorsInsideShadow, colorsOutsideShadow]

Now I can apply this linear model to the shadow area:

applyColorTransform = ImageApply[rgbToModel[#].colorTransform &, im, Masking -> DeleteSmallComponents@ColorNegate[Binarize[ms]]]

As a last step, I'll use inpainting to remove the artifacts at the border of the shadow area:

result = Inpaint[applyColorTransform, Dilation[EdgeDetect[Binarize[ms]], DiskMatrix[1]]]

The whole algorithm is still quite ad-hoc (no camera calibration, I didn't even try to model the illumination invariants described in the linked papers), but considering it's simplicity (less than 15 lines of code), I'd say the result isn't that bad.

Answer 2

This is a difficult problem, as the referenced literature in the linked SE questions may attest.

I can provide some steps towards your goal, but I don't have the full answer.

im = ImageTake[Import["https://i.sstatic.net/rSOow.png"], {10, -10}, {10, 220}]

First, trying to find the shadow mask, ChanVeseBinarize seems to work best here. Note that I used a marker value to indicate the area I'm interested in. Blurring gets rid of very small components first.

ChanVeseBinarize[Blur[im], {{99, 39}}]

Getting rid of another set of additional spurious elements using DeleteSmallComponents:

mask = ColorConvert[
           DeleteSmallComponents[ ChanVeseBinarize[Blur[im], {{99, 39}}]], 
           "GrayScale"
       ]

From here, the plan goes awry. The idea was to separate the image in hue, saturation, and brightness components, assuming that the shadows should principally influence brightness and leave hue and saturation alone.

imHSB = ColorConvert[im, "HSB"];

Interactively increasing the brightness in the shadow area until the shadow disappears:

Manipulate[
   ImageMultiply[imB // Image, ImageAdjust[mask // Blur, 0, {0, 1}, {b, 1}]], 
   {b, 0, 1}
]

Bummer, it seems the impact of the shadow on the lower-left part of the image is different from the upper-right part. I don't seem to be able to remove it everywhere.

Another problem becomes apparent if we look at the hsb components now:

GraphicsRow[Image /@ {imH, imS, imB}]

The effects of the shadow doesn't seem to be restricted to the brightness alone. The non-linearities in the camera have caused the color components to be affected too.

For now, I'm out of options, but at least you have a mask.