Spectral Upsampling & Subtractive Mixing

[briend]

General Procedure:

1. Convert RGBA to straight color, RGB data
2. Upsample to spectral reflectance curves
3. Determine Weights/ratios
4. Blend 2 or more curves with Weighted Geometric Mean
5. Convert back to RGBA

Convert RGBA to straight color

Since an RGBA value like 1,1,1,0 represents a pure emission, I don't think we can sensibly create a reflectance curve for such a color. That's beyond the scope of this article, anyway. I am assuming you have color data with alpha >0.

So, let's say you have two RGBA colors, yellow and blue, that you want to mix like paint and get green. Those colors might be partially transparent, like these:

Blue: (0.0, 0.0, 0.4, 0.5)
Yellow: (0.70, 0.70, 0, 0.75

First divide both colors by their alpha to get "straight" color:

Blue: (0.0, 0.0, 0.8)
Yellow: (0.93, 0.93, 0)

Upsample to spectral reflectance curves

Next, we need to "recover" a potential spectral reflectance curve that could account for these colors on a real-world object illuminated by a light source, such as the D65 that your computer is probably using to represent "white".

Scott Allen Burns has an excellent article on this here:
http://scottburns.us/reflectance-curves-from-srgb/
There are several other methods to generate these curves, such as Smits and Meng:
https://colour.readthedocs.io/en/develop/colour.recovery.html

To do this in real time we need to cheat somehow to avoid calculating the curves for every single pixel. Meng describes a way to use interpolation between a small number of pre-computed colors. A silly method I tried is to generate curves for all 256^3 8 bit sRGB colors, and use this table to fetch reflectance curves quickly. This requires a huge data file and about 2.4 GB of RAM. I have found a simpler way and that you can have pretty good results by just upsampling three colors, your RGB primaries. This probably works best with Rec 709 and won't work at all for imaginary primaries or Rec 2020 laser primaries.

The easy way:
Download Scott Burns' excel spreadsheet from the previous article:
http://scottburns.us/wp-content/uploads/2015/04/reflectance-from-sRGB.xlsx
This spreadsheet is already setup with a D65 illuminant weighted with the CMFs, and sRGB Rec 709 conversions.
Generate three curves for red, green, and blue. Save these as 36 float arrays "spectral_r, g, and b"
You could also use the matlab program he provides:
http://scottburns.us/wp-content/uploads/2015/04/ILLSS.txt

Now, to upsample from R,G,B to spectral, we can simply multiply each RGB channel by its respective spectral primary, and then sum the results. Is this a hack that I just made up? Yes. Will this violate energy conservation? Yes, likely (R+G+B certainly does, as some reflectances end up > 1.0). Will it work reasonably well and look ok? Yes, as far as I've seen.

So, for Blue we sum (R * spectral_r + G * spectral_g + B * spectral_b)
and end up with a 36 float array Spectral_Blue[36]. Do the same to get Spectral_Yellow[36].

Important note. We should avoid absolute 0.0 for our RGB channels, since that represents a black hole basically. Take the max(RGB, 0.0001) or something like that. You won't notice the lifted black.

Determine Weights

Again we can return to Scott Burns site where he discusses the blend method:
http://scottburns.us/subtractive-color-mixture/
The weighted geometric mean seems to be a well-known approximation for mixing paints. We just need to know the weights for our two colors. This gets a bit tricky and is not really documented anywhere. We have to consider two things: what ratio do we want to mix our two paints, and also how should their original alpha values affect this ratio? I have decided that both need to be accounted for.

First, we can normalize our alpha values to sum to 1.0:

Blue Ratio = 0.5 / (0.5 + 0.75) = 0.4
Yellow Ratio = 0.75 / (0.5 + 0.75) = 0.6

If we wanted to mix equal parts of each, we're done. 0.4 to 0.6 would be "equal" weights after considering the original alpha values (using my own fuzzy logic). Now what if we wanted two parts yellow, one part blue? We can multiply and normalize our ratios again.
Blue Ratio = 0.4 * 1/3 = 0.1333 / (0.1333 + 0.4) = ~0.25
Yellow Ratio = 0.6 * 2/3 = 0.4 / ( 0.1333 + 0.4) = ~0.75

Blend

Now we have our weights and can do the mixing, finally:

Spectral_Green[36] = Spectral_Blue ^ 0.25 * Spectral_Yellow ^ (1 - 0.25)
Here's a depiction of what this looks like, and the resulting green reflectance curve:

You can actually blend N colors as long as the weights sum to one: color_a^0.25 * color_b ^0.25 * color_c^0.5

Back to RGB

Hurray. Now we need to get back to RGB. Scott Burns has made this simpler by combining several matrices together into one, so we can simply multiply Spectral_Green[36] by the T_Matrix here:
http://scottburns.us/wp-content/uploads/2015/03/matrix_T_tab_delimited.txt
Then, sum each row to get r, g, and b:

python:
r, g, b = np.sum(Spectral_Green*T_MATRIX, axis=1)

Next you can apply the original alpha with the normal alpha calculation
a = 0.5 * 1/3 + 0.75 * (1 - 1/3) = 0.6666~
RGBA = [r * a, g * a, b * a, a]

Creating Tones, Tints, and Shades

Now that we can mix virtual paints, we might want to make our typical color controls (such as HSV) follow the Painter's mixing model, that is, tints, tones, and shades. Because of the Abney effect, we can't just add white light to our color without skewing the hue. This is fairly simple, but in order to mix these we need four things: the starting color, a black color, a white color, and a grey color. Once you have these you can create your tints tones and shades by just mixing these together using the spectral upsampling method already discussed.

How to get black and white

You could just use RGB 1,1,1 and 0,0,0 for white and black, but to accommodate for alternate white points you might want to calculate your white point from a particular XYZ. For instance, you might want to use a warm white or a cool white for your palette. Once you have the spectral reflectance curve for this color, you can simply multiply it by a small number like 0.01 to get a very dark paint. Then you just mix those with your paint to get tints and shades.

How to get grey

Grey is trickier because the proper grey to mix tones of Yellow is going to be very different than the proper grey to mix tones of Blue, for example. So, we can employ a color appearance model to find the grey color to match a given paint, and then mix that in to get our tones. Set the Chroma channel to zero and you'll have your grey color.

How to rotate your hue

This is also tricky and I don't think we really need the spectral model for this. Rather, just use a color appearance model to change the hue while retaining the value and chroma of your color. Another possibility would be to have a defined palette of colors, and match the nearest color in the direction you are rotating. Then each step would mix in some of that color until you arrive at the pure palette color. Then the next hue rotation would match to the next-nearest palette color, etc. Or, the way I've implemented it, we choose a target color +/- 60 degrees on the color wheel and mix in some amount of that color. This shifts the color while simultaneously desaturating it, and eventually you end up with mud similar to what would happen with real paints if you just kept adding colors from around the wheel.