[fpu] precision problem in int and floor functions

[Mo-Gul]

I am pretty sure that I encountered that specific problem several times now, so maybe it can be fixed. When fpu is enabled the floor and int functions can yield wrong results as e.g. shown in the following MWE.

Here a list were I know that this problem encountered:

• https://tex.stackexchange.com/q/505871/95441
• https://tex.stackexchange.com/a/249507/95441

---

\documentclass[border=5pt,varwidth]{standalone}
\usepackage{tikz}
    \usetikzlibrary{fpu}
\begin{document}
    \textbf{Without \texttt{fpu}:} \\
    \pgfmathparse{floor(12/4)}\pgfmathresult{} instead of 3 \\
    \pgfmathparse{int(12/4)}\pgfmathresult{} instead of 3 \\

    \textbf{With \texttt{fpu}:} \\
    \pgfkeys{
        /pgf/fpu,
        /pgf/fpu/output format=fixed,
    }
    \pgfmathparse{floor(12/4)}\pgfmathresult{} instead of 3 \\
    \pgfmathparse{int(12/4)}\pgfmathresult{} instead of 3 \\
\end{document}

[hmenke]

The problem is neither floor, nor int, but the fact that

\pgfkeys{/pgf/fpu}\pgfmathparse{12/4}\pgfmathresult

outputs 1Y2.9999e0].

[hmenke]

The two input arguments to \pgfmathfloatdivide@ are getting rescaled here:

pgf/tex/generic/pgf/libraries/pgflibraryfpu.code.tex

Lines 1297 to 1303 in 08275e3

	\edef\pgfmathfloat@arga{#1}%
	\pgfmathfloattoextentedprecision@a{\pgfmathfloat@arga}%
	\let\pgfmathfloat@arga=\pgfmathresult
	%
	\edef\pgfmathfloat@argb{#2}%
	\pgfmathfloattoextentedprecision@b{\pgfmathfloat@argb}%
	\let\pgfmathfloat@argb=\pgfmathresult

After that we have \pgfmathfloat@arga=12.000 and \pgfmathfloat@argb=40.00. Most of the rest of the function does nothing because neither number is negative, or zero, or nan, or inf. Thus we fall through to the division of the mantissae.

pgf/tex/generic/pgf/libraries/pgflibraryfpu.code.tex

Line 1385 in 08275e3

\pgfmath@basic@divide@{\pgfmathfloat@arga}{\pgfmathfloat@argb}%

So we essentially do

\dimen0=12pt
\divide\dimen0 by 40

and that is by TeX's arithmetic 0.29999pt. So I conclude that there is not really anything I can do here.

When you encounter precision problems you could instead use l3fp which has full IEEE754 precision.

[hmenke]

@Mo-Gul I'm about to close this as wont-fix because I can't actually do anything about it. Are you okay with that?

[Mo-Gul]

If there really isn't anything that you can do I/we'll have to live with it and it is fine to close it.

So please allow me to ask for one maybe possible solution/workaround:
Would it be possible to implement a special handler that checks for the result of 0.29999pt and than change that to 0.3pt?

[hmenke]

You mean in each and every of the thousands of invocations of \pgfmathfloatdivide@ check if the operands are 12 and 40 and then return 0.3? That sounds horribly inefficient.

[Mo-Gul]

Is it a guess or a statement that "this" is "horribly inefficient"?
I guess there isn't a performance test yet, right? So if you would provide a patch (here or via email) I would be happy to test compilation times for the codeexamples with and without the patch using the parallel build script.

Just to be sure: One really would need to check for 12 and 40 and cannot directly hard-code the result 0.29999pt? If yes, would it be an option to just do the calculation once and store the result which is then used?

[hmenke]

Well yes, of course there would have to be a check for the operand being 12 and 40 because not every division amounts to 0.3, right? Since it is an extra branch with two comparisons it will necessarily be slower than anything without this branch. Also I presume that 12 and 40 will not be the only exceptions, so we'd end up with a huge table of exceptions, massively slowing down the code.

[josephwright]

I'm with @hmenke here: if you need a real FPU, use the LaTeX3 one, as it's designed from the ground up to handle this sort of thing (at the cost that it is slower than the pgf one).

[Mo-Gul]

You are the experts, so I trust your opinion. So you are welcome to close this issue.

But could you state an example here (or at one of the TeX.SX questions) on how to use the l3fp package to solve/circumvent this issue!? Many thanks in advance!

[muzimuzhi]

So we essentially do

\dimen0=12pt
\divide\dimen0 by 40

and that is by TeX's arithmetic 0.29999pt. So I conclude that there is not really anything I can do here.

Using eTeX's expression may help here since

the results of divisions and scalings are rounded, whereas TeX’s \divide truncates.

according to texdoc etex, sec. 3.5 "Expressions". For example \dimen0=\dimexpr12pt/40\relax will assign \dimen0 0.3pt.

PS: I (re)read this issue when searched for possible duplicates of TeX-SX question no.611573, in which one of the causes is about the jumping result of divide.

[hmenke]

@muzimuzhi hmenke@5077895

Doesn't pass CI unfortunately.