Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft #1284 streamreader return cell #1291

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

thomascharbonnel
Copy link
Contributor

PR Details

In order to make the stream reader API behave more like StreamWriter, we have to change the type of data that is returned by Rows and Columns.
This PR reuses the Cell structure defined in the StreamWriter API.

Description

WIP

Related Issue

Related to #1284

Motivation and Context

How Has This Been Tested

Types of changes

  • Docs change / refactoring / dependency upgrade
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@xuri xuri added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 27, 2022
@thomascharbonnel thomascharbonnel force-pushed the 1284-streamreader-return-cell branch from 764eb44 to 964c563 Compare July 27, 2022 08:01
@@ -475,6 +484,61 @@ func (c *xlsxC) getValueFrom(f *File, d *xlsxSST, raw bool) (string, error) {
}
}

func (c *xlsxC) getTypedValueFrom(f *File, d *xlsxSST) (interface{}, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to take care of the performance impact of this feature, I agree with you for reused the Cell structure, and I think we needn't add the String() function on Cell, keep applying number format and for the Cell.Value by default, and not convert the string cell value by cell's data type, all the cell value will be return in string data type, add a CellType field in the Cell structure which represents the type of the cell, let user convert the data type of the cell value from string by CellType if in need.

If a cell value was numeric, which applies a text or data time number format, then get the cell's value without specifying the RawCellValue option, it will convert the cell value to string data type after applying the number format style, if no CellType field, at this time, it will be loose the original data type information, so we need to add this field instead of convert cell value to Go data type to represents the cell data type.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main reason i did the conversion is because (if i'm not mistaken) the StreamWriter API will read the Golang type in order to decide the type in the output XML. (So this way i can read a cell and convert it to Cell then use StreamWriter to write the same value.
So in order to keep the ability to do in := streamread(…); streamwriter.Write(in) (pseudocode), we'll need to change StreamWriter to support CellType?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest that add the CellType field in the type Cell which is only used for the stream reader, the stream reader always returns cell value in string data type. This design has no ability for in := streamread(…); streamwriter.Write(in), which needs the user to prepare the needed cell's value from the string data type, and use them in the stream writer. Because there are some differences with set the cell's value on the stream writer, the cell value depends on the style of the cell when reading the cell value, such as the number format, but the library doesn't support applying all number formats for the cell currently, so if we convert the cell to Go data type when reading cells, there is currently no guarantee that the values of all cells can be correctly converted to the data type of the Go language, so I don't suggest adding this ability at this time, what do you think about it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it makes more sense if i explain my use case: basically we want to write a lot of data in the worksheets (hence stream writer), but we also want to reuse the style, formula and data of an excel "template", so an Excel file we use as a reference and we read while writing to stream writer. This works fine right now for values but not for formulas and styles. I think it's possible for me to do some magic outside of Excelize (eg. use getTypedValueFrom in my own code), but you pointed out that performance can be an issue (as well as data quality) with this approach so maybe i should reconsider 🤔
So i was thinking of doing something a bit different: 1. copy the template file to use as base, 2. create new worksheets 3. delete the original worksheets. This way, the styles would already be present in the output file, and i can just copy the StyleID over from the input to the output. With that design, i think i will only need to write the StyleID and the Value without caring about the conversion, correct?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The style of the cell was storage in the workbook globally, so the same style ID represents the same style between the template and generates the workbook, but the style with number format only can be used for the cell in numeric value, so you need to convert the cell value to correct data type when streaming writing. I'm not sure where the input data come from in your case, if it comes as you say in := streamread(…); streamwriter.Write(in), because all cells value of the in input data was in string data type, if any cell's style contains a number format, which style will be invalid after streamwriter.Write(in).

Thomas Charbonnel added 4 commits August 3, 2022 16:21
@thomascharbonnel thomascharbonnel force-pushed the 1284-streamreader-return-cell branch from 964c563 to 4db31ef Compare August 4, 2022 03:21
@xuri xuri linked an issue Aug 26, 2022 that may be closed by this pull request
@xuri xuri force-pushed the master branch 2 times, most recently from 79958aa to 0c3dfb1 Compare May 25, 2024 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

StreamWriter-like API when reading
2 participants