Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Speed up function _serialize_dataframe by 123% in PR #6044 (refactor-serialization) #6078

Merged
merged 24 commits into from
Feb 3, 2025
Merged
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
7eda663
feat: Implement serialization functions for various data types and ad…
ogabrielluiz Jan 31, 2025
1045c7a
feat: Enhance serialization by adding support for primitive types, en…
ogabrielluiz Jan 31, 2025
42bd591
fix: Update Pinecone integration to use VectorStore and handle import…
ogabrielluiz Jan 31, 2025
6cb6983
test: Add hypothesis-based tests for serialization functions across v…
ogabrielluiz Jan 31, 2025
9767766
refactor: Replace custom serialization logic with unified serialize f…
ogabrielluiz Jan 31, 2025
b20e0f9
refactor: Replace recursive serialization function with unified seria…
ogabrielluiz Jan 31, 2025
9acdbfc
refactor: Replace custom serialization logic with unified serialize f…
ogabrielluiz Jan 31, 2025
bbb5286
refactor: Enhance serialization logic by adding instance handling and…
ogabrielluiz Jan 31, 2025
4b563dc
refactor: Remove custom dictionary serialization from ResultDataRespo…
ogabrielluiz Jan 31, 2025
d8182dc
refactor: Enhance serialization in ResultDataResponse by adding max_i…
ogabrielluiz Jan 31, 2025
a898ab1
refactor: Move MAX_ITEMS_LENGTH and MAX_TEXT_LENGTH constants to seri…
ogabrielluiz Jan 31, 2025
1a0116d
refactor: Simplify message serialization in Log model by utilizing un…
ogabrielluiz Jan 31, 2025
ba01a10
refactor: Remove unnecessary pytest marker from TestSerializationHypo…
ogabrielluiz Jan 31, 2025
01676d5
optimize _serialize_bytes
ogabrielluiz Jan 31, 2025
7ebb304
feat: Add support for numpy integer type serialization
ogabrielluiz Jan 31, 2025
f16110b
feat: Enhance serialization with support for pandas and numpy types
ogabrielluiz Jan 31, 2025
550bec4
test: Add comprehensive serialization tests for numpy and pandas types
ogabrielluiz Jan 31, 2025
573130c
fix: Update _serialize_dispatcher to return string representation for…
ogabrielluiz Jan 31, 2025
bcfe6f9
fix: Update _serialize_dispatcher to return the object directly inste…
ogabrielluiz Jan 31, 2025
9b51778
optmize conditional
ogabrielluiz Feb 3, 2025
824ae5f
optimize length check
ogabrielluiz Feb 3, 2025
59ad780
fix: Update string and list truncation to include ellipsis for clarity
ogabrielluiz Feb 3, 2025
e9010ce
⚡️ Speed up function `_serialize_dataframe` by 123% in PR #6044 (`ref…
codeflash-ai[bot] Feb 3, 2025
1c1c997
Merge branch 'main' into codeflash/optimize-pr6044-2025-02-03T12.09.54
ogabrielluiz Feb 3, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions src/backend/base/langflow/serialization/serialization.py
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,10 @@ def _serialize_dataframe(obj: pd.DataFrame, max_length: int | None, max_items: i
"""Serialize pandas DataFrame to a dictionary format."""
if max_items is not None and len(obj) > max_items:
obj = obj.head(max_items)
obj = obj.apply(lambda x: x.apply(lambda y: _truncate_value(y, max_length, max_items)))
return obj.to_dict(orient="records")

data = obj.to_dict(orient="records")

return serialize(data, max_length, max_items)


def _serialize_series(obj: pd.Series, max_length: int | None, max_items: int | None) -> dict:
Expand Down
Loading