Why don't cameras record light data throughout the entire shutter?

Question

Sorry if the question title is confusing, I don't know the best way to word this,so feel free to change it if you can think of something better. I learned electronic shutters can capture the image all at once instead of using a mechanical shutter which uses the curtain. This gave me an idea. Let's say a given shot would be properly exposed at 1/200s, but the dynamic range of the image is far too wide for the camera to capture.

Why cant a camera with an electronic shutter continuously capture and record light data from an image throughout the entire duration of the shutter instead of just collecting light data and finally storing it as one picture? It would be like seeing a room start from darkness and gradually increasing in brightness. The camera would then be able to capture the entire dynamic range of an image and compile the data into an image with the entire dynamic range in just one photo instead of needing multiple exposures for an HDR. This would also allow for exposure adjustment in post processing without any loss of information since the camera has stored the light data from an entire range of exposures. Why isn't this idea currently being implemented?

Answer 1

It's been done in X-rays.

The TimePix is a 256x256 detector. It has three operating modes:

• the usual "total energy in this pixel since we started integrating";
• Time-over-Threshold (TOT): the detected pulse height is recorded in the pixel counter in the TOT mode; and
• Time-of-Arrival (TOA): the TOA mode measures time between trigger and arrival of the radiation into each pixel.

This technology has been adapted to optical imaging. The TOT mode is best described as acting like a Wilkinson ADC -- the readout corresponds to total time where accumulated charge is at or above threshold. Subtracting this from the shutter time tells you how long it took for this pixel to saturate. Thus, for each pixel, you can draw the line from 0 to saturation over time since the shutter opened. So you can pick whatever virtual shutter time you want (as long as all the pixels saturated) and use each pixel's line to estimate its accumulated light up to that virtual shutter time.

---

A more direct implementation of your idea has been done in CMOS. Each pixel records and reports its time to reach a threshold charge. (Rather than ADC the pixels that do not saturate in time, the threshold is swept, so every pixel eventually exceeds a sufficiently lowered threshold.)

---

I recall that the Pixim Digital Pixel System (example) also did this using an ADC per pixel and nondestructively reading out the accumulated charge repeatedly (to get the accumulation slope). But I can't find current corroborating evidence.

Answer 2

You are missing some obvious problems with this idea.

You want to "continously" capture the light data, but that's already being done.

Apparently you mean to have a series of images available after the exposure, each exposed from the start to times advancing withing the whole exposure. The later images would have more detail in shadow areas, but might have bright areas clipped. The camera firmware could then assemble a single image with larger dynamic range than any of the individual images.

The two glaring problems with this are:

• how to read out all the millions of pixels so fast, and
• where to put the results.

The technology isn't available today to do that.

Answer 3

You suggest "Or every time a photon hits a pixel on the sensor give it a timestamp" — this would be a huge amount of data. A quick search suggests that each pixel — or sensel — in a digital camera saturates at somewhere between 20,000 and 100,000 photons. Let's say that we're happy with a 12 megapixel camera and are okay with the lower side of sensitivity here. That's still a quarter of trillion data points. If we're talking a 50 megapixel camera with a lot of dynamic range, maybe five trillion. Even if we make our timestamps just be two bytes each (one byte only gives 256 values, so unlikely to be enough to make this all worthwhile), that's... a lot of data for one image. I mean, literally terabytes.

That's clearly not currently feasible in terms of the data pipeline with today's technology, let alone putting it somewhere.

Answer 4

What you're asking for, continuous light sampling, might be theoretically possible but practically too expensive. It might be possible to approximate it with a very high sampling rate. This could be done with a high-speed (slo-mo) video camera with very high frame rates. Then the output could be post-processed to create an image.

A quick search shows suff like this Phantom

These things work by having fast sensors and by being able to move and store huge amounts of data. Attempting continuous sampling, or a fast enough sampling rate that it looks continuous, magnifies this problem and the cost.

Answer 5

Electronic shutter is a step up already. We are now able to take all the pixels at the same time, then tell them to stop collecting (ie, sample each pixel), and measure the information for each color of each pixel serially, capturing data about an image that was taken simultaneously.

This did not used to be the case.

We still have to do some hacks for the HDR scenery, though, however it's not a bad as it used to be, again due to advancements in sensor technology. We now have greater sensor sensitivity and dynamic range, so a photo that used to require a two bracket shot and post processing can now be captured in camera because the sensor can measure both the highs and lows of certain images. In fact, the sensors have become so good that you'll rarely come across a situation that requires more than three bracketed shots to get the entire dynamic range. Older sensors may have required 5 or more bracketed shots.

Your idea, as I understand it, requires continuous measurement on a per pixel basis.

While this is a great idea, implementation remains a problem. Cameras are designed to stream data from the sensor serially. There isn't a line for each pixel to the processor, instead the image sensor has logic that allows the processor to read the value for a pixel, or many pixels, at once but not all at once. It has to iterate through all the pixels, and this takes time.

We can't overcome this because we're not going to be able to have 50 million wires going between the sensor and the processor. We could integrate more of the processing into the sensor, but the sensor is specialized to do one thing, and do it well. Adding digital circuitry would result in more noise, and probably smaller pixels even if 3D ICs were used. Further, the processes used to create a good light sensitive silicon are different than those used to create a good, low power, fast processing digital silicon.

All of these things are impediments, however for some specialized applications they are already being used. Usually in the scientific and industrial area.

But that doesn't mean we're being left out in the cold. As sensors improve, particularly in dynamic range, you'll find that eventually you'll get "HDR" in camera with no brackets - the sensors will simply be sensitive enough to get the full range, and the lenses and camera body will be good enough to prevent bleed, reflection, and other issues that prevent the sensor from achieving its full capability.

So while the idea isn't bad, it's complex, expensive, and we still have room to grow in other improvable areas so that your method may not even become necessary.

Answer 6

The real answer is price. If you're willing to pay 10-100x more for your camera, you can get some really fancy sensors.

The desired result you describe is a higher dynamic range for each pixel. There are a few ways to do this. The obvious way is to get a better ADC and CMOS sensor, but that costs money and isn't in the vein of what you are thinking. The next approach would be to suck charge off in a continuous process, in analog. This would permit you get a continuous function describing how many photons hit the pixel. However, this sort of analog hardware is tremendously difficult. In your camera, all of the pixel data is schlepped through a rather small number of ADCs. Part of the beauty of our sensors is how they can do that, yielding cheaper hardware by factors of hundreds. To do that continuously would require each and every pixel to have an extraordinary amount of fine tuned analog hardware.

Which brings us to the digital sampling approach. You mentioned the idea of capturing data every 1/1000th of a second, which suggests to me that you weren't actually thinking of a continuous process, as much as a sampling process where you get many datapoints for thin time-slices and stitch them together. As mentioned in other answers, HDR+ on some phones does exactly this. It takes several photos in rapid succession, and blends them to get the HDR effect. To do so, they obviously have much higher ADC bandwidth than you'd need for a single image, but they wouldn't need as much as it'd take to treat every pixel continuously.

From the sound of it, you'd like each pixel to do this time sampling on its own. To do this, we'd first need to take a foray into 3d integrated circuit design. You don't want any of the hardware on each pixel taking up space on the sensor's surface, or you'll have trouble with having too few pixels or losing a bunch of light when it falls on non-sensor parts of the IC. The only way to accomplish that is to build a 3d chip. These are really a future technology. We're starting to explore how to do this, but it's not easy. If you have hundreds of thousands of dollars to spare for your camera, we can make this kind of thing happen.

In the end, it sounds like you'd like the output of each pixel to be a "floating point number" instead of an "integer." That is to say each pixel would have a value for how many photons hit, and an exponent which basically says how much to multiply that value by to get the actual number of photons. As the pixel is exposed, it would sample at a very high rate (perhaps 5000Hz), and if the photon count gets too big, it picks a bigger exponent.

Now the real question is how much benefit do you get from this? Remember, the HDR+ approach is current technology, for cellphones in the hundreds of dollars. You're talking about using ultra-cutting edge technology with far more demanding tolerances than any camera out there. That will come at a cost. What did it buy you? What did those single-pixel during-the-shutter devices actually buy you that the cheap CMOS tech Google is pushing didn't? The answer is not much. There may be a few teeny tiny corner cases where this is a preferred approach, but with a price tag substantially higher than existing technology, it's a commercial nonstarter.

Answer 7

Something very similar is being implemented. It still operates by distinct frames, because there are significant advantages of a digital rather than an analog approach. But approaches exists with a time resolution in the pico seconds.

https://www.ted.com/talks/ramesh_raskar_a_camera_that_takes_one_trillion_frames_per_second

Answer 8

Why cant a camera with an electronic shutter continuously capture and record light data from an image throughout the entire duration of the shutter instead of just collecting light data and finally storing it as one picture?

What I think you're really proposing here is to describe an image not in terms of "how much light was collected during the entire exposure?" but rather "how bright was the scene at each point?" That's a great thought, and I can think of several ways to do that, but the thing they all have in common is that they add complexity to the sensor.

Camera manufacturers have long been working to provide more pixels, and I'd guess that keeping the structure of each individual pixel simple helps in that effort. Now that DSLRs generally have sensors with between 20 and 50 million pixels, perhaps we'll see them work instead on building better pixels. We're already seeing that in some ways -- dual pixel autofocus is one example. And there are certainly companies working on building sensors that provide more dynamic range, less noise, etc.

In short, I think it's likely that we'll see something along the lines of what you've proposed in the future, even if it doesn't work quite that way, and the reason we're not there already is probably just that other goals like increased pixel densities were higher priorities in the past.

Answer 9

It can be done in a somewhat different way. Instead of one picture you take a number of frames with different exposure times. Then you stack the pictures to get some sort of average depending on the algorithm you are using for stacking.

For instance with the recent total solar eclipse the amount corona visible to the naked eye was much larger than that which any one exposure time for the camera would show. That is because the eye has a logarithmic dynamic range whereas the eye has a linear dynamic range. So with stacking various exposure times you can much better approximate in a picture what the observers saw by eye.

Answer 10

Olympus Live Bulb and Live Time modes go in the direction you're describing.

From the OM-D E-M5 manual:

To view the progress of the exposure during shooting, choose a display interval for [Live BULB] (P. 89) or [Live TIME] (P. 89).

Here's a video. Note that you only get one exposure at the end, even though you see multiple ones during the process. The sensor pixels only care about the total amount of photons they received during the exposure, they don't know when or in which order those photons landed on the sensor.

Answer 11

You have the right idea. Sony is essentially doing something to this effect in the RX100M5 and other cameras which sport a feature they call D-Range Optimizer--analyzing the scene and adjusting and compensating for trouble areas.

The D-Range Optimizer function instantaneously analyzes the captured image data and automatically corrects for optimal exposure and tone reproduction. Often when taking backlit scenes, the subject's face or other areas in shadow appear darker in the photograph than they appear to the human eye. The D-Range Optimizer function discriminates between different conditions for the photographed scenes automatically corrects the gamma curve, exposure level, and other parameters to remove portions that are darker than they would appear to the human eye.

The D-range optimizer function also contains Standard mode, which adjusts the entire image uniformly (effective for correcting aspects like exposure), and Advanced mode, which automatically corrects areas within the composition. By using the Advanced mode, the photographer can produce a clear picture where both the subject and background are pictured with appropriate brightness, even when there is a large difference in the brightness of the two.

source: https://sony-paa-pa-en-web--paa.custhelp.com/app/answers/detail/a_id/26259/~/what-is-the-function-of-d-range-optimizer%3F