Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(java): Chunk by chunk predictive map serialization protocol #1722

Merged
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
8ce5f13
map general write
Jul 8, 2024
6e45eb6
map serializer chunk by chunk, key is not allowed null
Jul 10, 2024
7fd2dc6
preserveByteForHeaderAndChunkSize and extract method
Jul 10, 2024
9242f76
checkChunkSize and modify max chunk size to 127
Jul 10, 2024
7fbe54d
use Preconditions check chunk size
Jul 11, 2024
07d7911
save code
Jul 26, 2024
1e03081
update
Jul 29, 2024
b382f1d
update
Jul 29, 2024
ca2d221
debug
Jul 30, 2024
82cb1fb
remove useless commit
Jul 30, 2024
c492e8e
delete useless code
Jul 30, 2024
77c6364
添加配置isChunkSerializeMapEnabled,是否开启mapchunk序列化
Jul 31, 2024
fe2bb76
mvn spotless apply
Jul 31, 2024
55f8440
reset key and value serializer is null
Aug 2, 2024
3752339
modify code
Aug 6, 2024
48b8a75
add apache license
Aug 7, 2024
c06fdc7
use local variable and bug fix
Sep 9, 2024
aa364f5
use local variable and big method
Sep 11, 2024
c6c8235
Merge remote-tracking branch 'ant/main' into support_map_chunk_serial…
chaokunyang Jan 20, 2025
dd01e09
Merge branch 'main' into support_map_chunk_serialization_protocol
Jan 21, 2025
1aad170
merge main branch and delete useless file
Jan 21, 2025
1bd870e
remove isChunkSerializeMapEnabled config
Jan 21, 2025
311c614
remove useless code
Jan 21, 2025
b461314
split big method to short method
Jan 23, 2025
444959f
bug fix
Jan 23, 2025
d9a44b0
set useChunkSerialize =false
Jan 24, 2025
d785d46
fix javadoc style
Jan 24, 2025
90a267e
testDifferentKeyAndValueType
Jan 24, 2025
98fe77f
update
Jan 24, 2025
a8e2181
remove useless code
Jan 24, 2025
e5b63d7
add unit test
Jan 24, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions java/fury-core/src/main/java/org/apache/fury/Fury.java
Original file line number Diff line number Diff line change
Expand Up @@ -884,6 +884,15 @@ public Object readNullable(MemoryBuffer buffer) {
}
}

public Object readNullable(MemoryBuffer buffer, ClassInfoHolder classInfoHolder) {
byte headFlag = buffer.readByte();
if (headFlag == Fury.NULL_FLAG) {
return null;
} else {
return readNonRef(buffer, classInfoHolder);
}
}

/** Class should be read already. */
public Object readData(MemoryBuffer buffer, ClassInfo classInfo) {
depth++;
Expand Down Expand Up @@ -1380,6 +1389,10 @@ public Class<? extends Serializer> getDefaultJDKStreamSerializerType() {
return config.getDefaultJDKStreamSerializerType();
}

public boolean isChunkSerializeMapEnabled() {
chaokunyang marked this conversation as resolved.
Show resolved Hide resolved
return config.isChunkSerializeMapEnabled();
}

public boolean compressString() {
return config.compressString();
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,11 @@ public class Config implements Serializable {
private final boolean asyncCompilationEnabled;
private final boolean deserializeNonexistentClass;
private final boolean scalaOptimizationEnabled;
private final boolean chunkSerializeMapEnabled;
private transient int configHash;
private final boolean deserializeNonexistentEnumValueAsNull;


public Config(FuryBuilder builder) {
language = builder.language;
trackingRef = builder.trackingRef;
Expand Down Expand Up @@ -89,6 +91,7 @@ public Config(FuryBuilder builder) {
asyncCompilationEnabled = builder.asyncCompilationEnabled;
scalaOptimizationEnabled = builder.scalaOptimizationEnabled;
deserializeNonexistentEnumValueAsNull = builder.deserializeNonexistentEnumValueAsNull;
chunkSerializeMapEnabled = builder.chunkSerializeMapEnabled;
}

public Language getLanguage() {
Expand Down Expand Up @@ -229,6 +232,10 @@ public boolean isAsyncCompilationEnabled() {
return asyncCompilationEnabled;
}

public boolean isChunkSerializeMapEnabled() {
return chunkSerializeMapEnabled;
}

/** Whether enable scala-specific serialization optimization. */
public boolean isScalaOptimizationEnabled() {
return scalaOptimizationEnabled;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ public final class FuryBuilder {
boolean scalaOptimizationEnabled = false;
boolean suppressClassRegistrationWarnings = true;
boolean deserializeNonexistentEnumValueAsNull = false;
boolean chunkSerializeMapEnabled = false;
MetaCompressor metaCompressor = new DeflaterMetaCompressor();

public FuryBuilder() {}
Expand Down Expand Up @@ -304,6 +305,16 @@ public FuryBuilder withScalaOptimizationEnabled(boolean enableScalaOptimization)
return this;
}

/**
* use chunk by chunk method to serialize map, TODO: generate code for chunk by chunk method
* @param chunkSerializeMapEnabled
* @return
*/
public FuryBuilder withChunkSerializeMapEnable(boolean chunkSerializeMapEnabled) {
this.chunkSerializeMapEnabled = chunkSerializeMapEnabled;
return this;
}

private void finish() {
if (classLoader == null) {
classLoader = Thread.currentThread().getContextClassLoader();
Expand Down
Loading
Loading