[Tensor] Int4QTensor with quantized 4-bit integer data type #2895
+1,050
−134
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
djeong20
requested review from
myungjoo,
jijoongmoon,
again4you,
jaeyun-jung,
leemgs,
wooksong,
gichan-jang,
anyj0527,
lhs8928,
songgot,
jihochu,
DonghakPark,
SeoHyungjun,
baek2sm,
skykongkong8,
EunjuYang,
dkjung and
haehun
as code owners
skykongkong8
approved these changes
Jan 23, 2025
nntrainer/tensor/int4_tensor.cpp
Outdated
Comment on lines
233
to
248
| /// @todo this func should be template function | ||
| void Int4QTensor::addValue(unsigned int b, unsigned int c, unsigned int h, | ||
| unsigned int w, float value, float beta) { | ||
| auto const &idx = getIndex(b, c, h, w); | ||
| float output = getValue(idx); | ||
| output *= beta; | ||
| output += value; | ||
|
|
||
| // if result value is out of range, clamp to max/min value | ||
| int8_t val = std::trunc(std::clamp((int)output, -8, 7)); | ||
|
|
||
| // encode result value to int8 data | ||
| ((int8_t *)getData())[idx / 2] = | ||
| (idx % 2 == 0) ? (val << 4) | (((int8_t *)getData())[idx / 2] & 0x0f) | ||
| : (((int8_t *)getData())[idx / 2] << 4) | (val & 0x0f); | ||
| } |
There was a problem hiding this comment.
Quick question:
Do we just expect the user to consider scale factor in input float value and float beta?
I am curious about how basic math in int4Q tensor goes...
There was a problem hiding this comment.
Thanks for asking! Currently, no. This is to modify the quantized value directly.
EunjuYang
reviewed
Jan 24, 2025
nntrainer/tensor/int4_tensor.cpp
Outdated
| // encode result value to int8 data | ||
| ((int8_t *)getData())[idx / 2] = | ||
| (idx % 2 == 0) ? (val << 4) | (((int8_t *)getData())[idx / 2] & 0x0f) | ||
| : (((int8_t *)getData())[idx / 2] << 4) | (val & 0x0f); |
There was a problem hiding this comment.
As I understood, the computation should be :
Suggested change
| : (((int8_t *)getData())[idx / 2] << 4) | (val & 0x0f); | |
| : (((int8_t *)getData())[idx / 2] & 0xf0) | (val & 0x0f); |
I'm quite confused with it. Please let me know If I'm wrong :)
There was a problem hiding this comment.
you're right! thanks for pointing it out :)
EunjuYang
reviewed
Jan 24, 2025
nntrainer/tensor/int4_tensor.cpp
Outdated
Comment on lines
260
to
261
| (idx % 2 == 0) ? (val << 4) | ((int8_t *)getData())[idx / 2] | ||
| : ((int8_t *)getData())[idx / 2] | (val & 0x0f); |
There was a problem hiding this comment.
I find we need to clear out the space we want to append the value.
Suggested change
| (idx % 2 == 0) ? (val << 4) | ((int8_t *)getData())[idx / 2] | |
| : ((int8_t *)getData())[idx / 2] | (val & 0x0f); | |
| (idx % 2 == 0) ? (val << 4) | (((int8_t *)getData())[idx / 2] & 0x0f) | |
| : (((int8_t *)getData())[idx / 2] & 0xf0) | (val & 0x0f); |
This pull request presents the class, a powerful solution for efficiently storing quantized 4-bit integer data. By packing each 4-bit integer into an 8-bit memory space, we utilize memory resources effectively—where the first four bits hold the first 4-bit value and the last four bits hold the second. 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: Donghyeon Jeong <dhyeon.jeong@samsung.com>
djeong20
force-pushed
the
add/tensor/qint4x2
branch
from
d60db7a to
f526f3a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request presents the class, a powerful solution for efficiently storing quantized 4-bit integer data. By packing each 4-bit integer into an 8-bit memory space, we utilize memory resources effectively—the first four bits hold the first 4-bit value, and the last four bits hold the second.