Autocast

[haytham2597]

Soon i will try make AMP (Automatic Mixed Precision) with GradScaler.

[haytham2597]

@dotnet-policy-service agree

[NiklasGustafsson]

Is this a work-in-progress PR, or something you're submitting for approval and merging? If the latter, then please create an issue to track "to do" items and add some unit tests.

[haytham2597]

U can merging this if you want, this dont break anything (far as I know). But may useful for someone who want use that autocast function manually. My idea and plan is to make AMP, GradScaler, these modules use the functions I added.
Thank, I will try take into account about issue "to do" and unit tests. Sorry.

[NiklasGustafsson]

@haytham2597 -- thank you for your first PR! Much appreciated. Please see the comment I made in the review.

[haytham2597]

Do not merge, i keep have some issue.

[NiklasGustafsson]

Lots of errors in the build on everything except the .NET FX builds (which don't have System.Range):

https://dev.azure.com/dotnet/TorchSharp/_build/results?buildId=103093&view=logs&j=80b813b5-9a08-5859-11a8-dc0e5b556e52&t=d3977768-5d05-5555-eccf-169680cb7093

[lintao185]

I am very happy to see this proposal.

[NiklasGustafsson]

@haytham2597 -- just a gentle ping! I think this PR would be very valuable, but it's still a draft, and thus I will not merge it. I also had some comments in my review.

[haytham2597]

@haytham2597 -- just a gentle ping! I think this PR would be very valuable, but it's still a draft, and thus I will not merge it. I also had some comments in my review.

Yeah, but sorry i am very busy with studied and work. I need managed very well about my time for making some progress on this pull requests, i mean this is very useful for me too.
But i can provide some idea about this if you want continue.

• While the autocast is inside on scope automatically convert the tensor to dtype of autocast.
  For example

torch.Tensor a;
using(var ac = torch.NewAutocast()){
      torch.Tensor b = a;
      torch.Tensor c = torch.arange(...)
}

The b and c should automatically converted to float16 (if that is dtype of mixed precision from f32) including all weight/bias of modules that found inside i mean the module, example: ResNet should passed to mixed precision.

The idea Is very similar that you do with

using (var d = torch.NewDisposeScope())

And in outer scope need back to original dtype. Because the neural should backward with original dtype (on my understood)
With my external THS_Autocast u can determine the dtype that should passed/work and if is enabled/disabled too
I don't know if I explained myself correctly, but feel free to ask.

[NiklasGustafsson]

Yeah, no pressure!

We all have other things to do, so I understand completely. Just wanted to let you know we haven't forgotten about your work, and that it will be appreciated, if and when you find time.

[GilesBathgate]

I would also like to see this completed. It should help with #1136 as well.

[ingted]

Really need this!! Thank you!!

[haytham2597]

About AMP or Autocast, @NiklasGustafsson do you have any idea what the "only" (or more abstraction) method is to obtain the tensor? Because in autocast for example, inner-scope on Autocast should all tensors pass to Float16, So the problem is Tensor have so much operation (ie: sum, prod, some linalg, div, etc.) And i should in every method cast the tensor to specific ScalarType. But I want to see where is one method for that, I thinking about using the IntPtr of Tensor and each call of this (because some method uses that, like prod, sum, etc use that IntPtr) and casting to that ScalarType. Is best idea work with IntPtr tensor right?

P.D: I don't know why i can Compile but cannot run Test so rare.

[haytham2597]

Hi, the last commit AMP Problem outscope have problem outscope, in this eample code:

var cast = AMPManager.GetInstance();
var b = torch.rand(new long[] { 3, 3 }, torch.ScalarType.Float32, device: new torch.Device(DeviceType.CUDA));
Debug.Assert(b.dtype == torch.ScalarType.Float32, "b.dtype == torch.ScalarType.Float32"); //OK
using (cast.Enter())
{
    b = b.mul(1);
    Debug.Assert(b.dtype == torch.ScalarType.Float16, "b.dtype == torch.ScalarType.Float16"); //OK
    var a = torch.rand(new long[] { 3, 3 }, torch.ScalarType.Float32, device:new torch.Device(DeviceType.CUDA));
    Console.WriteLine($"A: {a.dtype}"); //OK Print Float16
}
//Debug.Assert(b.dtype == torch.ScalarType.Float32, "b.dtype == torch.ScalarType.Float32");
Console.WriteLine($"B: {b.dtype}"); //BAD: This print Float16 instaed of Float32
b = b.mul(2);
Console.WriteLine($"B: {b.dtype}"); //BAD: This print Float16 instaed of Float32
var c = torch.rand(new long[] { 3, 3 }, torch.ScalarType.Float32, device: new torch.Device(DeviceType.CUDA));
Console.WriteLine($"C: {c.dtype}"); //OK Print Float32

After using cast.Enter() the tensor B is not converting to Float32 (Original ScalarType of B is Float32, but cant revert this, i don know why) So right now i have one problem, cant "uncasting" that tensor. All Work well except when i used a tensor inside of scope and when go to outside cause problem with variable B, somebody can discover what is the problem?

[GeorgeS2019]

For users to understand this PR

Automatic Mixed Precision package - torch.amp

https://pytorch.org/docs/stable/amp.html#autocasting

[NiklasGustafsson]

It looks like you're expecting the element type of 'b' to change after you exit the dynamic scope, is that right?

That would mean that you have to do the type conversion in place, at least from the perspective of the managed instance that 'b' refers to -- i.e. replace the handle to the native tensor rather than create a new managed instance. Is that what your code is doing?

[haytham2597]

@NiklasGustafsson

Yes I trying change the dtype of B. But i think is not bad my code, because of Cuda OPS and the example of [§4]

A few hours ago I noticed that t in my code i keep certain IntPtr values ​​that can change. In all instances I keep IntPtr but both outside and inside the scope they always differ.

[§1]

//From https://github.com/dotnet/TorchSharp/blob/b032342a78435ba6eb197e4e7db53469ac176aa8/src/TorchSharp/Tensor/Tensor.Math.cs#L1289
public Tensor mul(Scalar target)
{
    //For example Handle = 0x168
    var res = THSTensor_mul_scalar(Handle, target.Handle); //Now res is 0x196
    if (res == IntPtr.Zero) { CheckForErrors(); }
    return new Tensor(res);
}

[§2]

//From my src/TorchSharp/Amp/AMPManager.cs
private void Revert()
{
    for (int i = 0; i < TensorsCasts.Count; i++) {
        var tc = TensorsCasts[i];
        tc.Handle= To(tc.Handle, tc.Dtype); 
    }
}

Now like my last comment b=b.mul(1); the B is now completely new IntPtr, That is, I am saving the tensor "references" wrong. Because always change, so i never can revert in that way. I think.

In my code for holding IntPtr i do this:
[§3]

//From src/TorchSharp/Tensor/Tensor.cs in internal Tensor(IntPtr handle)
if (AMPManager.GetInstance().IsEnabled) {
    this.handle = AMPManager.GetInstance().Work(handle, this.handle); //Can ignore second argument because i was testing other things
} else {
    this.handle = handle;
}

I'm getting dizzy but I think that in these code examples; §1, for example 0x168 is no longer available except 0x196.

Update:
[§4]

a_float32 = torch.rand((8, 8), device="cuda")
b_float32 = torch.rand((8, 8), device="cuda")
a_float32_mul = torch.rand((8, 8), device="cuda")
print(f"Dtype of a_float32 Before autocast: {a_float32.dtype}")
print(f"Dtype of a_float32_mul Before autocast: {a_float32_mul.dtype}")
with torch.autocast(device_type="cuda"):
	e_float16= torch.mm(a_float32, b_float32)
	a_float32= torch.mm(a_float32, b_float32)
	a_float32_mul= a_float32_mul.mul(2) 
	print(f"Dtype of e_float16: {e_float16.dtype}")
	print(f"Dtype of a_float32: {a_float32.dtype}")
	print(f"Dtype of a_float32_mul: {a_float32_mul.dtype}")
	
print(f"Dtype of a_float32 OUTSCOPE: {a_float32.dtype}")
print(f"Dtype of a_float32_mul OUTSCOPE: {a_float32_mul.dtype}")

Dtype of a_float32 Before autocast: torch.float32
Dtype of a_float32 Before autocast: torch.float32
Dtype of e_float16: torch.float16
Dtype of a_float32: torch.float16
Dtype of a_float32_mul: torch.float32
Dtype of a_float32 OUTSCOPE: torch.float16
Dtype of a_float32_mul OUTSCOPE: torch.float32

Only certain operator (like torch.mm) keep same dtype. Mmm that mean my code is nothing wrong. I Should change dtype only for certain operator example torch.mm or another.

Glad to be closer to AMP and GradScaler.

Conclussion: I need read very well the documentation and testing well in python.

[NiklasGustafsson]

b = b.mul(1);

What this statement does is overwrite the variable b with a completely new instance, both native and managed.

On the other hand:

b = b.mul_(1);

would do the multiplication in place, i.e. modify the existing instance:

public Tensor mul_(Tensor target)
{
        THSTensor_mul_(Handle, target.Handle);
        CheckForErrors();
        return this;
}