BOM will be removed even if I set #define HAS_CODECVT

[hkottmann]

I'm programming with Qt and included rapidcsv as it is the most efficient library to edit *.csv files. According to the documentation, I've defined HAS_CODECVT before I include rapidcsv.h in my program

`#ifndef HAS_CODECVT
#define HAS_CODECVT

#include "smartdcpd.h"
#include "ui_smartdcpd.h"
#include "rapidcsv.h"
`

But the BOM which was set by creating the document will be still removed by rapidcsv

[d99kris]

Hi @hkottmann - I'd like to clarify more on the steps taken. Would the following be correct?

1. The original csv file is created using another software.
2. The original csv file has UTF-16 byte-order-mark.
3. Then rapidcsv is used to read the original csv file, modify its content and write to a new file.
4. The resulting new file written by rapidcsv does not have UTF-16 byte-order-mark.

If not, please help specify.

Thanks!

[hkottmann]

Dear Chris

Sorry for my late answer. The *.csv file is created by one of my own written Software using Qt with C++ and MinGW 11.2.0-64Bit that we are using in our research environment. As per default and fits our needs, it's 8-Bit encoded and has therefor an 8-bit BOM on it's header (HEX EFBBEF) is an 8-Bit UTF-BOM. BTW, as rapidcsv is that much powerful and reliable, I've fixed in that way:
``qint64 SmartDCPD::processDataFile(const QString &fileName)
{
QElapsedTimer t;
t.start();
QString tempfile = fileName + ".tmp";
rapidcsv::Document doc(fileName.toUtf8().constData(), rapidcsv::LabelParams(-1, -1), rapidcsv::SeparatorParams(';'));
QString stamp = QDateTime::currentDateTime().toString("dd.MM.yyyy hh:mm:ss.zzz");
doc.SetCellstd::string(1, 2, stamp.toUtf8().constData());
doc.Save();
QFile::rename(fileName, tempfile);
QFile file(tempfile);
file.open(QIODevice::ReadOnly);
QTextStream str(&file);
QFile f(fileName);
f.open(QIODevice::WriteOnly);
QTextStream ts(&f);
ts.setGenerateByteOrderMark(true);
ts << str.readAll();
f.close();
file.close();
QFile::remove(tempfile);
return t.nsecsElapsed();
}`

Have many thanks for your great library...

Kind regards,

Han

[d99kris]

Thanks for sharing the details 👍 I will take a look and see how/if we can add a fix in rapidcsv.

[d99kris]

This issue should be fixed now in bb2db43 - rapidcsv should retain the original UTF-8 BOM when performing Save().

FYI: For UTF-8 encoded files you do not need to define HAS_CODECVT (although it wont cause any issues). The HAS_CODECVT define is only required when dealing with UTF-16 encoded files.

[hkottmann]

Dear Kris

Thank you for your great work, I've just tested it and it works fine. I wish you a nice weekend.

Kind regards,

Hans

[d99kris]

Thanks 👍 Have a nice weekend you too!