Advice on combining CLIP with the neural renderer

[thuangb]

Hi, this is an very interesting and impressive work in using AI for art. Now, as CLIP has created more and more astounding painting from text, I just want to ask some advice, if possible, on combining CLIP and your work together which can be very interesting. Thanks

[jiupinjia]

Hi @thuangb, thanks for your suggestion. Yes, we can definitely add a CLIP loss to our framework. I believe it should work well although I haven't tried that. This would be interesting and worth trying. I will check this out in the near future and probably add this new feature to our method.

[thuangb]

Hi, I just want to ask that what part I have to modify ? I think I must add an new CLIP loss in loss.py and modify the optimization loop in the demo files but I do not familiar with the neural renderer part in your code so I need some advice. Thanks. FYI, I think you can refer to this repo if you plan to add CLIP feature in the future.

[jiupinjia]

Hi @thuangb, since almost everything in the rendering pipeline is differentiable, all you have to do is to add a CLIP similarity loss to the loss function part. I have been working on this recently. Hopefully, this feature will be available very shortly. I will let you know when it is ready.