Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[QNN EP] Initial INT4 support #21171

Draft
wants to merge 57 commits into
base: main
Choose a base branch
from
Draft

[QNN EP] Initial INT4 support #21171

wants to merge 57 commits into from

Conversation

adrianlizarraga
Copy link
Contributor

@adrianlizarraga adrianlizarraga commented Jun 25, 2024 •
edited
Loading

Description

  • Adds support for int4 quantized weights (per-tensor and per-channel) on QNN EP
  • Adds test script that creates an INT4 qdq model with a Conv
  • Adds a unit tests demonstrating accuracy issues.

Motivation and Context

This is the next step in being able to run models that use 4-bit quantized weights on QNN EP.

@adrianlizarraga
Update include/framework/ with int4
40da679
@adrianlizarraga
Update onnxruntime_c_api.h with int4 type
e3e8a6b
@adrianlizarraga
Update cpu_contrib_kernels.cc with int4 Q/DQ
5e01e0f
@adrianlizarraga
Update framework/data_types.cc with int4 types
44e0e02
@adrianlizarraga
Update onnxruntime map type info with int4 types
ce03eb2
@adrianlizarraga
Update Tensor methods to calc int4 tensor data sizze
e159007
@adrianlizarraga
Update function to map tensor_proto int4 to onnxruntime enum
46c3d0d
@adrianlizarraga
Update tensorprotoutils to handle int4 protobufs
583dae1
@adrianlizarraga
Add functions to map Int4x2 to an onnxruntime tensor type enum
0009a47
@adrianlizarraga
Update com.microsoft.DequantizeLinear schema to support int4 types fo…
d11a3d4 
…r input
@adrianlizarraga
Add option to disable int4 type in Conv and MatMul qdq node group sel…
208c403 
…ectors
@adrianlizarraga
Add DequantizeLinear with int4 support (missing block quant)
f91ae69
@adrianlizarraga
update transpose helper to support int4
7323793
@adrianlizarraga
Update provider bridge with int4 apis
8c79905
@adrianlizarraga
Update quantizer tool with int4
fc695ca
@adrianlizarraga
Remove duplicate enum
eeacb78
@adrianlizarraga
Remove MatMulSelector constructor arg
6f9da04
@adrianlizarraga
Remove unnecessary explicit template instantiation
7e8c458
@adrianlizarraga
Add static_cast
3b7ed5f
@adrianlizarraga
Add temporary CPU EP Int4 test (qdq conv)
c7086a5
@adrianlizarraga
Update operator docs
34dfa17
@adrianlizarraga
Update testing version of tensorprotoutils with int4 helpers
e7bec9c
@adrianlizarraga
Run lintrunner
cd8912e
@adrianlizarraga
Fix api to create int4 ort value
4cf3a75
@adrianlizarraga
Wrap long lines in tensorprotoutils
ca785c2
@adrianlizarraga
Add operator unit tests for Dequant int4/uint4
f87e785
@adrianlizarraga
Remove comments
d028f2f
@adrianlizarraga
Add QuantizeLinear int4 impl
24cc617
@adrianlizarraga
Update operator docs
f35b09e
@adrianlizarraga
Disable potentially bugged onnx tests
e33f198
@adrianlizarraga
Add TODO username
10f28aa
@adrianlizarraga
Fix warning as error and clean up
de1ded4
@adrianlizarraga
Merge branch 'main' into adrianl/dq-transpose-int4
378718e
@adrianlizarraga
Mlas kernels to quantize int4 (not blocked). Missing powerpc
746312b
@adrianlizarraga
branchless update of 4-bit element
2935f79
@adrianlizarraga
more branchless update of int4 lane
807537c
@adrianlizarraga
Fix cast warning as error
a36a128
@adrianlizarraga
Remove decrement of N
bc44557
@adrianlizarraga
Clean up Int4x2 class
f40992d
@adrianlizarraga
Github linter fixes
d0e17e2
@adrianlizarraga
Remove temporary unittest
6568d48
@adrianlizarraga
Case statement missing :
b8d5869
@adrianlizarraga
Merge branch 'adrianl/dq-transpose-int4' into adrianl/qnn-int4
80dadc5
@adrianlizarraga
Ported over INT4 work for qnn ep
b01f52c
@adrianlizarraga
Add debug API to qnn's get_qnn_qdq_config() for int4
35d3fa5
@adrianlizarraga
lintrunner
a6a3397
@adrianlizarraga
Fix use of int4 qdq options
1a528f1
@adrianlizarraga
Merge main and fix conflicts
58044f9
@adrianlizarraga
Remove unnecessary changes
168f4e0
@adrianlizarraga
Workaround that dels calibrator.model
7ede88b
@adrianlizarraga
Merge branch 'main' into adrianl/qnn-int4
ca6e5f9
@adrianlizarraga
Add Conv int4 per-channel test
4b2af75
@adrianlizarraga
Merge main
578d47d
@adrianlizarraga
Clean up
9770a97
@adrianlizarraga
Merge branch 'main' into adrianl/qnn-int4
eb9687a
@adrianlizarraga
Run lintrunner
92efb6d
@adrianlizarraga
Add more comments
05fee60
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@adrianlizarraga