[GPU/OpenCL] Initial version of SwiGLU Layer with OpenCL ops #2624

niket-agarwal · 2024-06-06T11:29:42Z

Added initial version of SwiGLU Layer for GPU. This is a basic implementation using naive kernel.

Changes added with this PR:

swiglu_cl.cpp added containing the new SwigluLayerCL class for OpenCL implementation.
Added unittest_layers_swiglu_cl.cpp to test Swiglu Layer on GPU.

Signed-off-by: Niket Agarwal [email protected]

taos-ci · 2024-06-06T11:29:45Z

📝 TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #2624. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/.

taos-ci · 2024-06-06T11:29:50Z

cibot: @niket-agarwal, nntrainer/layers/cl_layers/swiglu_cl.h does not include Doxygen tags such as @file @brief @author @bug. You must include the Doxygen tags in the source code. Please refer to a Doxygen manual at http://github.com/nnstreamer/TAOS-CI/blob/main/ci/doc/doxygen-documentation.md

taos-ci

@niket-agarwal, 💯 All CI checkers are successfully verified. Thanks.

taos-ci · 2024-06-07T13:12:44Z

cibot: @niket-agarwal, nntrainer/layers/cl_layers/swiglu_cl.h does not include Doxygen tags such as @file @brief @author @bug. You must include the Doxygen tags in the source code. Please refer to a Doxygen manual at http://github.com/nnstreamer/TAOS-CI/blob/main/ci/doc/doxygen-documentation.md

taos-ci · 2024-06-07T13:12:46Z

cibot: @niket-agarwal, The last line of a text file must have a newline character. Please append a new line at the end of the line in nntrainer/layers/cl_layers/swiglu_cl.h.

myungjoo · 2024-06-08T03:13:24Z

Your file format is not consistent. Refer to the static check: https://github.com/nnstreamer/nntrainer/actions/runs/9417608692/job/25967822229?pr=2624

Maybe you are using "MS-DOS" type of text file. Please use Linux/Unix type. The utility "dos2unix" will easily fix this. You may need to configure your code editor properly to prevent this issue, too.

myungjoo · 2024-06-08T03:15:33Z

Please add proper doxygen tags, too.
Without them, we are required to add doxygen tags for every class and function later right before a milestone release, which is going to be a huge task if we don't add tags for each commit. Please add them at each commit when you write a new function or class:

[ERROR] File name: api/ccapi/include/layer.h, 300 line, SwigluCl(const function needs @brief tag 

[ERROR] File name: nntrainer/layers/cl_layers/swiglu_cl.h, 98 line, void function needs @brief tag 

[ERROR] File name: nntrainer/layers/cl_layers/swiglu_cl.h, 101 line, void function needs @brief tag 

[ERROR] File name: nntrainer/layers/cl_layers/swiglu_cl.h, 104 line, void function needs @brief tag

nntrainer/layers/cl_layers/swiglu_cl.cpp

test/jni/Android.mk

myungjoo · 2024-06-08T03:20:27Z

test/unittest/layers/unittest_layers_swiglu_cl.cpp

+                           "nchw", "fp16", "fp16");
+
+GTEST_PARAMETER_TEST(SwigluGPU16, LayerGoldenTest,
+                     ::testing::Values(swiglu_basic_plain_w16a16));


Please add negative test cases with _n prefixes.
To pass the release criteria, the number of negative cases should be larger than or equal to the number of positive cases.

If you are going to add negative test cases after this PR, it's ok.

pallaviNNT · 2024-06-10T08:49:23Z

nntrainer/layers/cl_layers/swiglu_cl.cpp

+      break;
+    }
+
+    size_t dim1_size = sizeof(float) * dim1;


Shouldnt this be cl_half instead of float?

Yes right. Fixed.

taos-ci

@niket-agarwal, 💯 All CI checkers are successfully verified. Thanks.

s-debadri

LGTM!

EunjuYang · 2024-06-12T03:01:35Z

api/ccapi/include/layer.h

+ * @brief Helper function to create SwigluCl layer
+ */
+inline std::unique_ptr<Layer>
+SwigluCl(const std::vector<std::string> &properties = {},


SwigluCl doesn't seem to be consistent with the previous implementation (e.g., fc_layer_cl). Naming the layer with Cl seems only to support cl kernel. What do you think about this?
I think separately exposing layer for cl version for API doesn't seem to be proper.
As I understood, compute_engine is used to hide this.

The CPU version of SwiGLU is not yet present for the NNTrainer. We can update this API once both CPU and GPU are available. If required I can change the naming of the layer and remove cl from it for now, or this could be done when adding the CPU version, what do you suggest?

About the API policy, we may need to listen other reviewers' opinion as well. In my opinion, what about leaving @todo in the comment in order to replace the API later ?

Added a generalized naming for the swiglu layer in the latest commit. Please have a look and let me know if you think todo comment is still required.

taos-ci

@niket-agarwal, 💯 All CI checkers are successfully verified. Thanks.

djeong20

please check the size of the buffer in SwiGLULayerCl::swiglu_cl() and swiglu_cl_fp16() and fix supportBackwarding().

djeong20 · 2024-06-21T01:10:00Z

nntrainer/layers/cl_layers/swiglu_cl.cpp

+    size_t dim1_size = sizeof(float) * dim1;
+    size_t dim2_size = sizeof(float) * dim2;
+    int dim = int(dim1 * dim2);
+    opencl::Buffer inputA(context.context_inst_, dim1_size * dim2_size, true,


should the size be as follows?

Suggested change

opencl::Buffer inputA(context.context_inst_, dim1_size * dim2_size, true,

opencl::Buffer inputA(context.context_inst_, sizeof(float) * dim1 * dim2, true,

Yes right! Corrected in latest commit.

nntrainer/layers/cl_layers/swiglu_cl.h

djeong20 · 2024-06-21T01:12:50Z

test/jni/Android.mk

 	 ../unittest/layers/unittest_layers_input.cpp \
 	 ../unittest/layers/unittest_layers_loss.cpp \
-	 ../unittest/layers/unittest_layers_fully_connected_cl.cpp \


any reason to remove this?

It's not removed, just shifted to line 445.

taos-ci · 2024-06-21T05:29:07Z

To contributor, We have used 'Signed-off-by:' notation by default to handle the license issues, that result from contributors. Note that 'Is there a Signed-off-by line?' is important because lawyers tell us we must have to it to cleanly maintain the open-source license issues even though it has nothing to do with the code itself.

taos-ci

@niket-agarwal, 💯 All CI checkers are successfully verified. Thanks.

taos-ci

@niket-agarwal, 💯 All CI checkers are successfully verified. Thanks.

myungjoo · 2024-06-25T04:16:42Z

Please rebase to the recent main.
Any big commit or PR will suffer longer time to merge. In the next time, please try to submit smaller PRs.

jijoongmoon · 2024-06-25T11:57:08Z

nntrainer/layers/cl_layers/swiglu_cl.cpp

+
+Layer *create_swiglu_layer_cl() {
+  auto layer = new SwiGLULayerCl();
+  std::cout << "swiglu created\n";


you do not need to std::out hear.

jijoongmoon · 2024-06-25T11:58:09Z

nntrainer/layers/cl_layers/swiglu_cl.cpp

+
+Layer *create_swiglu_layer_cl() {
+  auto layer = new SwiGLULayerCl();
+  std::cout << "swiglu created\n";


plz remove this std::cout.

jijoongmoon · 2024-06-25T11:58:43Z

nntrainer/layers/cl_layers/swiglu_cl.cpp

+}
+
+void destroy_swiglu_layer_cl(Layer *layer) {
+  std::cout << "swiglu deleted\n";


taos-ci

@niket-agarwal, 💯 All CI checkers are successfully verified. Thanks.

jijoongmoon · 2024-06-25T12:07:48Z

nntrainer/layers/layer_context.cpp

@@ -694,6 +695,10 @@ std::string RunLayerContext::getKernelName(LayerKernel layerKernel) {
    return "addition_cl";
  case LayerKernel::ADD_FP16:
    return "addition_cl_fp16";
+  case LayerKernel::SWIGLU:


Not this PR, but we need to rethink that this Kernel Name and actual kernel object placed in RunContext. Originally RunContext is just for the input / output / weight and Tensors to compute.. and I think this ClKernel should be handled at cl layer itself as in CPU Kernel.

Added naive version of OpenCL implementation for SwiGLU Layer. Incorporated kernel for ops used. Added unit test for SwiGLU_layer_cl. Signed-off-by: Niket Agarwal <[email protected]>

taos-ci

@niket-agarwal, 💯 All CI checkers are successfully verified. Thanks.

jijoongmoon

LGTM

taos-ci approved these changes Jun 6, 2024

View reviewed changes

github-actions bot added the Need Review label Jun 6, 2024

niket-agarwal force-pushed the swiglu_gpu branch from f3f0f0f to 1cd044c Compare June 7, 2024 13:12

niket-agarwal changed the title ~~[WIP][GPU/OpenCL] Initial version of SwiGLU Layer with OpenCL ops~~ [GPU/OpenCL] Initial version of SwiGLU Layer with OpenCL ops Jun 7, 2024

niket-agarwal marked this pull request as ready for review June 7, 2024 13:14

niket-agarwal requested review from myungjoo, jijoongmoon, again4you, jaeyun-jung, leemgs, wooksong, helloahn, kparichay, gichan-jang, anyj0527, zhoonit, lhs8928, songgot, jihochu, DonghakPark, SeoHyungjun, baek2sm, skykongkong8, djeong20, EunjuYang and a team as code owners June 7, 2024 13:14

myungjoo reviewed Jun 8, 2024

View reviewed changes

nntrainer/layers/cl_layers/swiglu_cl.cpp Outdated Show resolved Hide resolved

myungjoo reviewed Jun 8, 2024

View reviewed changes

test/jni/Android.mk Show resolved Hide resolved

myungjoo reviewed Jun 8, 2024

View reviewed changes

pallaviNNT reviewed Jun 10, 2024

View reviewed changes

niket-agarwal force-pushed the swiglu_gpu branch from 1cd044c to cd0696f Compare June 11, 2024 06:47

taos-ci approved these changes Jun 11, 2024

View reviewed changes

s-debadri approved these changes Jun 11, 2024

View reviewed changes

EunjuYang reviewed Jun 12, 2024

View reviewed changes

niket-agarwal force-pushed the swiglu_gpu branch from cd0696f to ab0d87c Compare June 20, 2024 10:13

taos-ci approved these changes Jun 20, 2024

View reviewed changes

djeong20 reviewed Jun 21, 2024

View reviewed changes

taos-ci approved these changes Jun 21, 2024

View reviewed changes

niket-agarwal force-pushed the swiglu_gpu branch from 3536e3a to 095eb3f Compare June 24, 2024 12:28

taos-ci approved these changes Jun 24, 2024

View reviewed changes

niket-agarwal force-pushed the swiglu_gpu branch from 095eb3f to f1a289a Compare June 25, 2024 11:27

jijoongmoon reviewed Jun 25, 2024

View reviewed changes

taos-ci approved these changes Jun 25, 2024

View reviewed changes

jijoongmoon reviewed Jun 25, 2024

View reviewed changes

[GPU/OpenCL] Initial version of SwiGLU Layer with OpenCL ops

3457fb0

Added naive version of OpenCL implementation for SwiGLU Layer. Incorporated kernel for ops used. Added unit test for SwiGLU_layer_cl. Signed-off-by: Niket Agarwal <[email protected]>

niket-agarwal force-pushed the swiglu_gpu branch from f1a289a to 3457fb0 Compare June 25, 2024 12:09

taos-ci approved these changes Jun 25, 2024

View reviewed changes

jijoongmoon approved these changes Jun 25, 2024

View reviewed changes

github-actions bot added PR/READY2MERGE and removed Need Review labels Jun 25, 2024

jijoongmoon merged commit ed2d27f into nnstreamer:main Jun 25, 2024
24 checks passed

	opencl::Buffer inputA(context.context_inst_, dim1_size * dim2_size, true,
	opencl::Buffer inputA(context.context_inst_, sizeof(float) * dim1 * dim2, true,

[GPU/OpenCL] Initial version of SwiGLU Layer with OpenCL ops #2624

[GPU/OpenCL] Initial version of SwiGLU Layer with OpenCL ops #2624

Conversation

niket-agarwal commented Jun 6, 2024 • edited Loading

taos-ci commented Jun 6, 2024

taos-ci commented Jun 6, 2024

taos-ci left a comment

Choose a reason for hiding this comment

taos-ci commented Jun 7, 2024

taos-ci commented Jun 7, 2024

myungjoo commented Jun 8, 2024

myungjoo commented Jun 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pallaviNNT Jun 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

taos-ci left a comment

Choose a reason for hiding this comment

s-debadri left a comment

Choose a reason for hiding this comment

EunjuYang Jun 12, 2024 • edited Loading

Choose a reason for hiding this comment

niket-agarwal Jun 12, 2024 • edited Loading

Choose a reason for hiding this comment

EunjuYang Jun 19, 2024 • edited Loading

Choose a reason for hiding this comment

niket-agarwal Jun 20, 2024 • edited Loading

Choose a reason for hiding this comment

taos-ci left a comment

Choose a reason for hiding this comment

djeong20 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

taos-ci commented Jun 21, 2024

taos-ci left a comment

Choose a reason for hiding this comment

taos-ci left a comment

Choose a reason for hiding this comment

myungjoo commented Jun 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

taos-ci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

taos-ci left a comment

Choose a reason for hiding this comment

jijoongmoon left a comment

Choose a reason for hiding this comment

niket-agarwal commented Jun 6, 2024 •

edited

Loading

pallaviNNT Jun 10, 2024 •

edited

Loading

EunjuYang Jun 12, 2024 •

edited

Loading

niket-agarwal Jun 12, 2024 •

edited

Loading

EunjuYang Jun 19, 2024 •

edited

Loading

niket-agarwal Jun 20, 2024 •

edited

Loading