Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OFX cannot be imported by Denaro #327

Open
Holzhaus opened this issue Dec 10, 2024 · 9 comments · May be fixed by #332
Open

OFX cannot be imported by Denaro #327

Holzhaus opened this issue Dec 10, 2024 · 9 comments · May be fixed by #332

Comments

@Holzhaus
Copy link

The OFX files generated by this tool cannot be imported by the personal financing app Denaro (GitHub, FlatHub).

Import fail with "Nothing to Import". I guess some fields expected by Denaro are missing.

@Holzhaus
Copy link
Author

Related issue on the Denaro issue tracker: NickvisionApps/Denaro#813

@kedder
Copy link
Owner

kedder commented Dec 10, 2024

Do you have an idea what is missing in the generated OFX file?

@Holzhaus
Copy link
Author

Nope. Here's a redacted version of the generated file (after passing it through xmllint --format for readability):

<?xml version="1.0"?>
<!--
OFXHEADER:100
DATA:OFXSGML
VERSION:102
SECURITY:NONE
ENCODING:UTF-8
CHARSET:NONE
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:NONE
-->
<OFX>
  <SIGNONMSGSRSV1>
    <SONRS>
      <STATUS>
        <CODE>0</CODE>
        <SEVERITY>INFO</SEVERITY>
      </STATUS>
      <DTSERVER>20241210194354</DTSERVER>
      <LANGUAGE>ENG</LANGUAGE>
    </SONRS>
  </SIGNONMSGSRSV1>
  <BANKMSGSRSV1>
    <STMTTRNRS>
      <TRNUID>0</TRNUID>
      <STATUS>
        <CODE>0</CODE>
        <SEVERITY>INFO</SEVERITY>
      </STATUS>
      <STMTRS>
        <CURDEF>EUR</CURDEF>
        <BANKACCTFROM>
          <BANKID>987654321</BANKID>
          <ACCTID>123456789</ACCTID>
          <ACCTTYPE>CHECKING</ACCTTYPE>
        </BANKACCTFROM>
        <BANKTRANLIST>
          <DTSTART>20241109</DTSTART>
          <DTEND>20241208</DTEND>
          <STMTTRN>
            <TRNTYPE>POS</TRNTYPE>
            <DTPOSTED>20241206</DTPOSTED>
            <DTUSER>20241206</DTUSER>
            <TRNAMT>-100.50</TRNAMT>
            <FITID>20241206abcdefghijklmnop</FITID>
            <NAME>Name of recipient</NAME>
            <MEMO>Foo bar GIROCARD transaction</MEMO>
            <CURRENCY>
              <CURSYM>EUR</CURSYM>
            </CURRENCY>
          </STMTTRN>
        </BANKTRANLIST>
        <LEDGERBAL>
          <BALAMT>9999.99</BALAMT>
          <DTASOF>20241208000000</DTASOF>
        </LEDGERBAL>
      </STMTRS>
    </STMTTRNRS>
  </BANKMSGSRSV1>
</OFX>

@Holzhaus
Copy link
Author

Is there a OFX file supported by the Denaro importer available, that can be used as reference?

@kedder
Copy link
Owner

kedder commented Dec 10, 2024

I have never heard of Denaro before. Try asking denaro devs for an example OFX they support. Maybe we can find the difference then.

@Holzhaus
Copy link
Author

oops, sorry, asked that question in the wrong bugtracker :D

@Holzhaus
Copy link
Author

I did some digging in the Denaro code and managed to modify the ofx file so that it's accepted. Denaro uses https://github.com/afonsof/OfxSharp internally. Here's what I found to be the issue:

  1. The OFX Headers are wrapped inside XML/SGML comment blocks. OfxSharp expects them raw, not inside a comment.
  2. The OfxSharp expects CR LF line endings (Windows/DOS style), not LF (POSIX).
  3. Ofxstatement exports ENCODING:UTF-8 and CHARSET:NONE, but OfxSharp expects ENCODING:USASCII and CHARSET:1252.

@Holzhaus
Copy link
Author

Holzhaus commented Jan 11, 2025

So, according to the headers found in the generated ofx file, ofxstatement exports OFX 1.0.2.
I checked the OFX spec from https://financialdataexchange.org/FDX/FDX/About/OFX-Work-Group.aspx?a315d1c24e44=2 to see who is right.

Comments

Regarding the XML/SGML comments, the OFX 1.0.2 spec's section 2.2 "Open Financial Exchange Headers" says:

The contents of an Open Financial Exchange file consist of a simple set of headers followed by contents defined by that header

The Open Financial Exchange headers are in a simple tag:value syntax and terminated by a blank line. Open Financial Exchange always sends headers unencrypted, even if application-level encryption is used for the remaining contents. The language and character set used for the headers is the same as the preceding MIME headers.

The first entry will always be OFXHEADER with a version number. This entry identifies the contents as an Open Financial Exchange file and provides the version number of the Open Financial Exchange headers that follow (not the version number of the contents). For example:

OFXHEADER:100

Open Financial Exchange headers can contain the following tags.

DATA:OFXSGML
VERSION:102
SECURITY:
ENCODING:
CHARSET:
COMPRESSION:
OLDFILEUID:
NEWFILEUID:

A blank line follows the last header. Then (for type OFXSGML), the SGML-readable data begins with the <OFX> tag.

Also, sec. 2.3.3 "Comments Not Supported" explains that:

Open Financial Exchange files cannot contain comments.

So I guess the SGML comment delimiters written by ofxstatement are indeed wrong.

Line Endings (CR LF vs. LF)

The spec doesn't say anything about line endings. However, sec. 1.2.1 "Data Transport" states the following:

Here is a typical request:

POST http://www.fi.com/ofx.cgi HTTP/1.0        HTTP headers
User-Agent:MyApp 5.0
Content-Type: application/x-ofx
Content-Length: 1032

OFXHEADER:100                                  OFX headers
DATA:OFXSGML
VERSION:102
SECURITY:TYPE1
ENCODING:USASCII

<OFX>                                          OFX request
... Open Financial Exchange requests ...
</OFX>

A blank line defines the separation between the HTTP headers and the start of the Open Financial Exchange headers. A blank line also separates the Open Financial Exchange headers and the request. (See Chapter 2 for more information about the Open Financial Exchange headers.)

HTTP headers require CR LF (cf. RFC 9112, sec. 2.1, so I think it can be assumed that CR LF is also expected for the OFX headers as well.

The OFX 1.6 spec explicitly mentions that it needs to be CR LF:

A brief note here that a “blank line” means a carriage return and a linefeed pair – CRLF.

Therefore I think that ofxstatement should always use CR LF (\r\n), not plain CR (\n).

Encoding / Charset

Section 5.1 "Language and Encoding" of the spec states:

The Open Financial Exchange headers specify the encoding and character set, as described in Chapter 2. Current encoding values are USASCII and UNICODE. For USASCII, character set values are code pages. UNICODE ignores the character set per se although it still requires the syntax. Servers must respond with the encoding and character set requested by the client.

This means that the ENCODING:UTF-8 violates the spec because only only USASCII and UNICODE are allowed.

I tried to use UNICODE, but it looks like that OfxSharp only allows ENCODING:USASCII and CHARSET:1252. Not sure if that would work for GnuCash as well?

@Holzhaus Holzhaus linked a pull request Jan 11, 2025 that will close this issue
@kedder
Copy link
Owner

kedder commented Jan 12, 2025

@Holzhaus if the problem is that ofxstatement declares a very old OFX version, I would rather upgrade the version instead of adhering to a standard from 1997. We need to support unicode properly in the output. Going back to cp1252 is not really an option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants