Add reference images

[virtualritz]

It would be great to have a set of reference images for each layer and each common combination of layers for which renderers who add an implementation of this shader could test their implementation against for look consistency.

Certainly, some implementation may choose to use another specular BRDF or expose some additional closure parameters that the reference implementation lacks.
But when those are at default values the produced look should match the reference implementation as closely as possible.

Only then will this be useful for stuff like sharing shading networks that have this as the final node between renderers/DCC packages or cross facility asset transfer.

[iliyang]

Thanks for the suggestion @virtualritz! Indeed, we are planning to include reference images and/or wedges with the effects of various parameters as seen in other material specifications.

You are right that the exact look can vary between renderers and platforms, e.g. ray tracing vs rasterization. We're also looking into expanding more on the real-time implementation of the shader.

[virtualritz]

Any update on this?

[iliyang]

Not yet, I'm afraid. It's actually hard to provide a reference look that is independent from the renderer and its implementation of the closures, especially with transmission and subsurface scattering.

That said, we want to produce some shader-ball wedges, and release the shader ball model too. I hope to be able to share more news here soon.

[f5-bit]

Hello there! I'm not sure if it concerns exactly this topic, but I recently compared the material models of the renderers Cycles and Eevee from Blender 2.83.1, Arnold for Maya 2020 and Unreal Engine 4.25 in the context of my bachelor thesis. I developed a Python based batch rendering framework that varies the desired material parameter over a specified range with a certain stepsize.

So far, I only examined Roughness, Specular, Metallic and Clearcoat ranging from 0 to 1 with a stepsize of 0.1 in different lighting situations.

If this is interesting or useful for you, I would be happy to share my findings with you.

[virtualritz]

@iliyang I don't think blocking this because we will have look differences related to closure choice/implementation makes sense.
The reference images are supposed to be guidelines. Not stuff you run your owns results against via tiffdiff or the like. Having known, expected results for the parameters we can cover w/o worrying about the closure stuff will go a long way.
On that note: this can be used for CI too. Aka reject commits to master that will produce different reference images with whatever implementation one chooses to create them in first place. Also known as image-based testing.

@f5-bit That sounds like a great start to me. 3Delight also has a Standard Surface in beta now that is already included with the renderer as an OSL shader. Maybe worth adding this too. And then put the whole shebang into a repo and link this here, for starters.

[pleprince]

Hi @iliyang
I also have StandardSurface prototype and a bunch of reference renders would be useful.
Cheers

[f5-bit]

@virtualritz Sorry that it took so long, but I have finally uploaded my source code for the rendering framework I was talking about. You can find it here: PBRMaterialModels
So far, the documentation is not really detailed and also I did not upload the code for creating material charts yet.
Let me know if you are still interested in this topic! Hopefully I can update the repo soon.

[virtualritz]

@f5-bit And sorry I didn't reply. This is useful but needs more work to help in the context.

I think best would be some CLI way of launching a series of test renders over [a subset of] an n-dimensional parameter space. Mostly you will want the corners of the parameter space only to keep the number of images bounded. I.e. pow(2, n) images at most for n parameters. I think we can limit ourselves to those 17 below.
Have IOR parameters and color parameters locked at some default and leave out emission for now.

- diffuse_roughness  
- specular_roughness  
- specular_anisotropy  
- metalness  
- transmission  
- transmission_scatter_anisotropy  
- transmission_dispersion  
- subsurface  
- subsurface_scale  
- subsurface_anisotropy  
- sheen  
- sheen_roughness  
- coat  
- coat_roughness  
- coat_affect_color  
- coat_affect_roughness
- thin_film_thickness 

This will result in 131,072 images to compare against. This should be fully automatic, via tiffdiff. The images can be very low res, I think. Any images that don't pass the threshold against the reference will be marked. That way anyone implementing this can quickly and automatically get an idea how well they match and where they need to improve.

@iliyang I think what is needed badly from Autodesk is simply a commitment to a reference implementation. I know that 3Delight, who recently added a Standard shader to their distribution, are using aiStandard as a reference because that is out in the wild and does not match the OSL reference in this repo.

Autodesk have to decide what is the reference here: the OSL skeleton or the Arnold implementation.
Does it make sense to have a reference implementation that differs from one's own implementation in the wild, as it the case here?

For something like a surface shader specifying the what and some of the how is not enough. You also have to detail the how exhaustively. The white paper is not enough here as the aforementioned disparity shows.

Reference images are a simple way to do that. And it may be a good way forward until the reference implementation is fixed or Autodesk takes a clear stance that this is in fact not the reference and Arnold's aiStandard is instead.

[f5-bit]

@virtualritz no worries! At least you replied faster than me.. 😅

Indeed, my rendering framework should be extended to vary several parameters at once to explore the n-dimensional parameter space you have mentioned. So far, only one parameter at a time can be changed (as you may have seen). I am afraid I cannot implement this functionality by myself in the near future as I have limited time at the moment. But if I can support you in some way, just let me know how.

This should be fully automatic, via tiffdiff

I did not know this tool yet, thank you for the suggestion! I assume you are talking about the program in Renderman and not the tiffdiff from the color management engine lcms?
This would simplify and speed up the process enormously and provide more reliable results than subjective evaluation, as I have done so far.
I have seen there is also pdiff, are there any differences between the two tools?

[virtualritz]

I did not know this tool yet, thank you for the suggestion! I assume you are talking about the program in Renderman and not the tiffdiff from the color management engine lcms?

No, I was indeed talking about the one in lcms. tiffdiff gives you a nice number on stdout, easy to parse and threshold against. I used this for image based regression testing for 20 years with great success (quoting Borat). ;)

I never used or even heard of pdiff. Looks interesting. I suggest to use whatever works best for this case.

[pleprince]

If you have OpenImageIO installed, diff and pdiff are available in oiiotool.

[f5-bit]

@virtualritz @pleprince Thank you very much for these recommendations!

Unfortunately I did not manage to use tiffdiff yet (I tried liblcms2-utils on Debian, but somehow tiffdiff was missing in the package?). However, I was able to use idiff from OpenImageIO. This tools comes in so handy!

Hopefully I will find the time to integrate it into my project soon as it looks very promising. I will inform you if I should make any progress.