How the Levels Calculator Works

When you pass sampled data through a nonlinear function and quantize, you may or may not lose levels, depending upon the type of function, the number of bits of input and the number of bits of output. The Levels calculator will show you how the number of levels is affected.

To use the calculator, simply configure the input and output according to your needs. The Gamma Exponent value is used only if you select Gamma as either the input or output type. The Calculate button is needed only after changing a Gamma Exponent value, otherwise the number of output levels is continually updated. Note that the Gamma Exponent must be in the range of 0.1 to 10.0.

As an example of what you might do with this calculator, suppose you have a monitor whose native gamma is 2.2. Further, suppose that you wish to coerce its apparent behavior to that of gamma 1.8 by loading a nonlinear function into the video look up table. Since almost all video cards have 8-bit data paths, we set the number of bits per channel at 8 for both input and output. Select Gamma 2.2 for the input and Gamma 1.8 for the output. Here is what we get:

Example 1

You can see from this that the 256 possible input levels are reduced to 237 output levels by the function that maps gamma 2.2 to gamma 1.8.

Here is another example. Suppose you have an 8-bit scale of L* tones, where L* = 0.0 is represented by 0 and L* = 100.0 is represented by 255. If you convert this to the luminance domain (Y), you will lose many levels in the darker colors unless you increase the number of bits used to represent the output:

Example 2

You can see that with 8-bit output, the number of levels has dropped from 256 to 174. How many bits do you need in order to preserve all 256 unique input levels? To find out, simply begin increasing the number of output bits until you first reach 256. This will occur at 12-bits:

Example 3

For the mathematically inclined, this same value of 12-bits may be arrived at independently by examining the first derivative of the CIE L* equation, which has a maximum value of kappa near zero (further details on this equation may be found here). Scaling the L* range of [0, 100] to 256 levels, we get:

Equation 1

Back to the Calculator