We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
举例,在Key Scaled = 0的时候,Average key norm始终为0。这个值应该是逐渐升高的。 一旦Key开始Scale之后,average key norm显示为tensor(0.2138, device='cuda:0')。这也是不对的。
epoch 38/150 steps: 25%|█████▎ | 76/300 [33:45<1:39:28, 26.64s/it, Average key norm=0, Keys Scaled=0, avr_loss=0.116] saving checkpoint: E:/LoRA models\caoXLlokrV1224M42-000038.safetensors
epoch 39/150 steps: 26%|▎| 78/300 [34:38<1:38:36, 26.65s/it, Average key norm=tensor(0.2138, device='cuda:0'), Keys Scaled=1, avr_l saving checkpoint: E:/LoRA models\caoXLlokrV1224M42-000039.safetensors
tensorboard中同样也会有显示错误,并且不包含Key Scaled =0 阶段的average key norm.
The text was updated successfully, but these errors were encountered:
No branches or pull requests
举例,在Key Scaled = 0的时候,Average key norm始终为0。这个值应该是逐渐升高的。
一旦Key开始Scale之后,average key norm显示为tensor(0.2138, device='cuda:0')。这也是不对的。
epoch 38/150
steps: 25%|█████▎ | 76/300 [33:45<1:39:28, 26.64s/it, Average key norm=0, Keys Scaled=0, avr_loss=0.116]
saving checkpoint: E:/LoRA models\caoXLlokrV1224M42-000038.safetensors
epoch 39/150
steps: 26%|▎| 78/300 [34:38<1:38:36, 26.65s/it, Average key norm=tensor(0.2138, device='cuda:0'), Keys Scaled=1, avr_l
saving checkpoint: E:/LoRA models\caoXLlokrV1224M42-000039.safetensors
tensorboard中同样也会有显示错误,并且不包含Key Scaled =0 阶段的average key norm.
![image](https://private-user-images.githubusercontent.com/17035106/311753794-d9730237-f674-4fcd-bd24-a36810730c18.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjAyNDI2MTUsIm5iZiI6MTcyMDI0MjMxNSwicGF0aCI6Ii8xNzAzNTEwNi8zMTE3NTM3OTQtZDk3MzAyMzctZjY3NC00ZmNkLWJkMjQtYTM2ODEwNzMwYzE4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MDYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzA2VDA1MDUxNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTZiMGMwZjdiYTViODZiNzNjMjA0NGI3MDI5NWZjZDg5N2NhMzQ5ZjhiMjA1MTlkNmExMmQwOGY3YTAxZmJhZDMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.GHhvwMIeDIEkqR_0IBWB02yuVrLXlO0xhXUfz4e1GCo)
The text was updated successfully, but these errors were encountered: