GnuCash
Contact   Instructions
Bug 797233 - Balance Sheet, Chart of Accounts, and other reports crash when involving foreign currency stocks
Summary: Balance Sheet, Chart of Accounts, and other reports crash when involving fore...
Status: RESOLVED FIXED
Alias: None
Product: GnuCash
Classification: Unclassified
Component: Reports (show other bugs)
Version: 3.5
Hardware: PC Windows
: Normal normal
Target Milestone: ---
Assignee: reports
QA Contact: reports
URL:
Whiteboard:
Keywords:
: 797311 (view as bug list)
Depends on:
Blocks:
 
Reported: 2019-05-10 21:21 EDT by CDB-Man
Modified: 2019-07-18 23:49 EDT (History)
7 users (show)

See Also:


Attachments
Tracefile (1.32 MB, text/plain)
2019-05-10 21:49 EDT, CDB-Man
no flags Details
Tracefile after GetQuote (41.66 KB, text/plain)
2019-05-11 21:46 EDT, CDB-Man
no flags Details
#3 Tracefile (deleted)
2019-05-11 21:53 EDT, CDB-Man
no flags Details
PriceDB extract (814.58 KB, text/plain)
2019-05-11 22:23 EDT, CDB-Man
no flags Details

Description CDB-Man 2019-05-10 21:21:23 EDT
When running report:
===
Report error
An error occurred while running the report.
===

GL is in CAD, stocks being held are USD.

Account structure is:
Assets (CAD)
> Investments (CAD)
>> Trading Account (CAD)
>>> USD Holdings (USD)
>>>> MSFT (stock account with commodity type MSFT)
>>>> VTI (stock account with commodity type VTI)

When running the balance sheet or chart of accounts, the error above is displayed.  If running a reduced balance sheet account with only individual accounts selected, it generally works... unless this happens:

===
[2019.05.10 20:25:12] <CDB-Man> hmm, in the account tree view, it looks like I can use the "Present (CAD)" column to see what has no $ values assigned
[2019.05.10 20:34:01] <CDB-Man> warlord: microsoft currently doesnt have an issue, but VTI does, see screenshot: https://i.imgur.com/1FHAURk.png
[2019.05.10 20:34:37] <CDB-Man> if I retype the value in priceDB, it fixes the $ value from null/blank to the current value
[2019.05.10 20:35:56] <CDB-Man> despite correcting that one though, there still seems to be more hidden somewhere...
[2019.05.10 21:00:32] <CDB-Man> so far, it seems only USD stocks have issues (my ledger is in CAD)
[2019.05.10 21:07:00] <CDB-Man> hmm, it looks like the balance seheet report fails if "show exchange rates" is checked
[2019.05.10 21:07:06] <CDB-Man> so SOMETHING has a broken FX rate
===

Looking at tracefile, it looks like there is an invalid argument error with a backtick causing the issue, see tracefile below:

===
* 21:14:42  WARN <gnc.scm> In ice-9/eval-string.scm:
  44: 19 [read-and-eval #<input: string 115a4ee0> #:lang ...]
  37: 18 [lp (gnc:report-run 9)]
In report.scm:
 784: 17 [gnc:report-run 9]
In c-interface.scm:
  70: 16 [gnc:backtrace-if-exception #<procedure 1178e498 at report.scm:785:5 ()>]
  22: 15 [gnc:call-with-error-handling #<procedure 1178e498 at report.scm:785:5 ()> ...]
In ice-9/boot-9.scm:
 160: 14 [catch #t #<procedure 1178e480 at c-interface.scm:23:8 ()> ...]
In c-interface.scm:
  27: 13 [#<procedure 1178e480 at c-interface.scm:23:8 ()>]
In report.scm:
 788: 12 [#<procedure 1178e498 at report.scm:785:5 ()>]
 762: 11 [gnc:report-render-html # #t]
In balance-sheet.scm:
 640: 10 [balance-sheet-renderer # #f #f]
In html-acct-table.scm:
 536: 9 [gnc:make-html-acct-table/env/accts (# # # # ...) (# # # # ...)]
 928: 8 [gnc:html-acct-table-add-accounts! # #]
 773: 7 [traverse-accounts! (# # # # ...) 0 0 ...]
In srfi/srfi-1.scm:
 613: 6 [for-each #<procedure 115c6f60 at html-acct-table.scm:774:9 (acct)> #]
In html-acct-table.scm:
 775: 5 [#<procedure 115c6f60 at html-acct-table.scm:774:9 (acct)> #]
In commodity-utilities.scm:
 987: 4 [gnc:sum-collector-commodity #<procedure 11767520 at report-utilities.scm:355:4 (action commodity amount)> ...]
In srfi/srfi-1.scm:
 573: 3 [map #<procedure eeae070 at report-utilities.scm:335:7 (pair)> (# # # ...)]
In commodity-utilities.scm:
 997: 2 [#<procedure f0d1990 at commodity-utilities.scm:989:10 (curr val)> # 1]
 848: 1 [gnc:exchange-by-pricedb-nearest # # 1577854799]
In unknown file:
   ?: 0 [gnc-pricedb-convert-balance-nearest-price-t64 # 1 # ...]

ERROR: Throw to key `numerical-overflow' with args `("/" "Numerical overflow" #f #f)'.
Comment 1 CDB-Man 2019-05-10 21:28:17 EDT
The error is displayed when "show exchange rates" is selected and there's a USD stock included in the selected accounts.
Comment 2 CDB-Man 2019-05-10 21:49:37 EDT
Created attachment 373276 [details]
Tracefile

Below occurs when running a report involving USD stocks, such as balance sheet, and the checkbox to show exchange rates is enabled:
===
ERROR: Throw to key `numerical-overflow' with args `("/" "Numerical overflow" #f #f)'.
* 21:43:27  WARN <gnc.scm> In ice-9/boot-9.scm:
 160: 19 [catch #t #<procedure 1125c5e8 at c-interface.scm:23:8 ()> ...]
In c-interface.scm:
  29: 18 [#<procedure 1125c5e8 at c-interface.scm:23:8 ()>]
In unknown file:
   ?: 17 [eval-string "(gnc:report-run 93)" #<undefined>]
   ?: 16 [call-with-input-string "(gnc:report-run 93)" ...]
In ice-9/boot-9.scm:
2412: 15 [save-module-excursion #<procedure 1125c5a0 at ice-9/eval-string.scm:65:9 ()>]
In ice-9/eval-string.scm:
  44: 14 [read-and-eval #<input: string 11445cb0> #:lang ...]
  37: 13 [lp (gnc:report-run 93)]
In report.scm:
 784: 12 [gnc:report-run 93]
In c-interface.scm:
  70: 11 [gnc:backtrace-if-exception #<procedure 1125c4b0 at report.scm:785:5 ()>]
  22: 10 [gnc:call-with-error-handling #<procedure 1125c4b0 at report.scm:785:5 ()> ...]
In ice-9/boot-9.scm:
 160: 9 [catch #t #<procedure 1125c498 at c-interface.scm:23:8 ()> ...]
In c-interface.scm:
  27: 8 [#<procedure 1125c498 at c-interface.scm:23:8 ()>]
In report.scm:
 788: 7 [#<procedure 1125c4b0 at report.scm:785:5 ()>]
 762: 6 [gnc:report-render-html # #t]
In balance-sheet.scm:
 731: 5 [balance-sheet-renderer # #f #f]
In html-utilities.scm:
 748: 4 [gnc:html-make-exchangerates # ...]
In srfi/srfi-1.scm:
 613: 3 [for-each #<procedure 2890e0a8 at html-utilities.scm:749:7 (commodity)> #]
In html-utilities.scm:
 750: 2 [#<procedure 2890e0a8 at html-utilities.scm:749:7 (commodity)> #]
In commodity-utilities.scm:
 848: 1 [gnc:exchange-by-pricedb-nearest # # 1557547199]
In unknown file:
   ?: 0 [gnc-pricedb-convert-balance-nearest-price-t64 # 1 # ...]
===


Below occurs when I run Finance::Quote by clicking "get quotes" in the pricedb view
====
* 21:46:41  WARN <gnc.gui> [PrintAmountInternal()] Bad numeric: GNC_ERROR_ARG.
* 21:46:41  WARN <gnc.gui> [PrintAmountInternal()] Bad numeric: GNC_ERROR_ARG.
Comment 3 Christopher Lam 2019-05-11 02:45:57 EDT
May I suggest running the latest maint from https://code.gnucash.org/builds/win32/maint/ -- the "$" display is odd.

Secondly try adding a line in commodity-utilities.scm as follows (the + line) and reattach tracefile.

 (define (gnc:exchange-by-pricedb-nearest
          foreign domestic date)
+  (gnc:pk 'gnc:exchange-by-pricedb-nearest foreign domestic date)
   (and (record? foreign)
        (gnc:gnc-monetary? foreign)
        date
Comment 4 CDB-Man 2019-05-11 21:39:21 EDT
Added the extra line and ran Balance Sheet with "show exchange rates" enabled, tracefile outputs this. Downloading the nightly.

===
* 21:37:30  WARN <gnc.scm> In ice-9/boot-9.scm:
 160: 19 [catch #t #<procedure f3c6ea0 at c-interface.scm:23:8 ()> ...]
In c-interface.scm:
  29: 18 [#<procedure f3c6ea0 at c-interface.scm:23:8 ()>]
In unknown file:
   ?: 17 [eval-string "(gnc:report-run 13)" #<undefined>]
   ?: 16 [call-with-input-string "(gnc:report-run 13)" ...]
In ice-9/boot-9.scm:
2412: 15 [save-module-excursion #<procedure f3c6e58 at ice-9/eval-string.scm:65:9 ()>]
In ice-9/eval-string.scm:
  44: 14 [read-and-eval #<input: string fac1b60> #:lang ...]
  37: 13 [lp (gnc:report-run 13)]
In report.scm:
 784: 12 [gnc:report-run 13]
In c-interface.scm:
  70: 11 [gnc:backtrace-if-exception #<procedure f3c6d68 at report.scm:785:5 ()>]
  22: 10 [gnc:call-with-error-handling #<procedure f3c6d68 at report.scm:785:5 ()> ...]
In ice-9/boot-9.scm:
 160: 9 [catch #t #<procedure f3c6d50 at c-interface.scm:23:8 ()> ...]
In c-interface.scm:
  27: 8 [#<procedure f3c6d50 at c-interface.scm:23:8 ()>]
In report.scm:
 788: 7 [#<procedure f3c6d68 at report.scm:785:5 ()>]
 762: 6 [gnc:report-render-html # #t]
In balance-sheet.scm:
 731: 5 [balance-sheet-renderer # #f #f]
In html-utilities.scm:
 748: 4 [gnc:html-make-exchangerates # ...]
In srfi/srfi-1.scm:
 613: 3 [for-each #<procedure f47d8e8 at html-utilities.scm:749:7 (commodity)> #]
In html-utilities.scm:
 750: 2 [#<procedure f47d8e8 at html-utilities.scm:749:7 (commodity)> #]
In commodity-utilities.scm:
 849: 1 [gnc:exchange-by-pricedb-nearest # # 1557633599]
In unknown file:
   ?: 0 [gnc-pricedb-convert-balance-nearest-price-t64 # 1 # ...]

ERROR: Throw to key `numerical-overflow' with args `("/" "Numerical overflow" #f #f)'.
Comment 5 CDB-Man 2019-05-11 21:46:21 EDT
Created attachment 373277 [details]
Tracefile after GetQuote

When I run "Get Quotes" in the PriceDB view, I see this occur for 520 lines:
* 21:42:54  WARN <gnc.gui> [PrintAmountInternal()] Bad numeric: GNC_ERROR_ARG.

See attached.
Comment 6 CDB-Man 2019-05-11 21:53:53 EDT
Created attachment 373278 [details]
#3 Tracefile

This tracefile is with gnucash --debug --extra parameters.
Actions taken:
1. Start GunCash
2. Run Balance Sheet report with show exchange rate enabled (error)
3. Run Balance Sheet report with show exchange rate disabled (error)
* May have reversed the order of #2 and #3
4. Manually edit the pricedb entry for ticker SPY, to retype the value.  After doing this, the account tree's value for my SPY account goes from "$" (null) to $3864
5. Run Balance Sheet report with show exchange rate enabled (error)
6. Run Balance Sheet report with show exchange rate disabled (report works)
* May have reversed the order of #5 and #6
Comment 7 CDB-Man 2019-05-11 22:23:03 EDT
Created attachment 373279 [details]
PriceDB extract

[2019.05.11 21:43:47] -*- chris still thinks your pricedb is borked. I think gnc-pricedb-convert-balance-nearest-price-t64 where / is failing
[2019.05.11 21:44:17] <CDB-Man> when I run Get Quotes in pricedb view, I see this a million times in tracefile:
[2019.05.11 21:44:17] <CDB-Man> * 21:42:54  WARN <gnc.gui> [PrintAmountInternal()] Bad numeric: GNC_ERROR_ARG.
[2019.05.11 21:44:44] <CDB-Man> no other error or output
[2019.05.11 21:45:44] <fell> CDB-Man: if you run gnucash --debug --extra and look in the tracefile?
[2019.05.11 21:47:12] <CDB-Man> will do that next.
[2019.05.11 21:49:29] <CDB-Man> hmm, other prices are broken now...
[2019.05.11 21:50:41] <CDB-Man> uploading another tracefile
[2019.05.11 21:53:22] <CDB-Man> uploaded
[2019.05.11 21:53:40] <fell> and mime changed :-)
[2019.05.11 21:54:59] <fell> if bugzilla does not recognize the format, it sets octet stream.
[2019.05.11 21:55:11] <CDB-Man> i hope my pricedb isnt corrupted like chris thinks... and that if it is corrupted, that its recoverable
[2019.05.11 21:55:38] <CDB-Man> 99% of the entries on my pricedb are from finance quote, and the remainder 1% come form transactions where i specified a rate at the transaction level
[2019.05.11 21:56:05] <chris> well your step 4 suggests something wrong in your pricedb
[2019.05.11 21:56:50] <CDB-Man> yeah, but how to pinpoint?
[2019.05.11 21:57:29] <CDB-Man> i've thought about using the remove all old prices option, but then i would like all the historic data
[2019.05.11 21:58:38] <chris> I suggest save as XML uncompressed, open in editor, copy all pricedb into new file, save as compressed, and upload. prices won't  have your transactional data. and give us a clue which price entry was fixable in your step 4.
[2019.05.11 22:00:22] <CDB-Man> hmm, 
[2019.05.11 22:00:36] <chris> and also upload some history about which builds you were using, and approximately if/when you upgrade your build.
[2019.05.11 22:02:10] <CDB-Man> version would be 3.3 where I had no issues, and after thre upgrade to 3.5+ where I started having issues
[2019.05.11 22:03:25] <CDB-Man> regarding XML, you're saying tha tthe default .gnucash is compressed XML, so what's the best way to decompressing it for me to extract pricedb?
[2019.05.11 22:03:41] <chris> well gtg, do the above. it's gzip - use any freeware eg 7-zip
[2019.05.11 22:06:17] <CDB-Man> that worked
[2019.05.11 22:15:50] <CDB-Man> hmmm.....
[2019.05.11 22:15:51] <CDB-Man>     <price:value>690402841721574/4679745419383</price:value>

===
So it seems that some prices being pulled by Finance::Quote are overly precise, and this might be causing the issue.  Attached is the priceDB protion of my .gnucash file, with snippets below of the current prices giving me trouble (SPY, VTI).

====
  <price>
    <price:id type="guid">56864ce89913414296f3cd968117c175</price:id>
    <price:commodity>
      <cmdty:space>NYSE</cmdty:space>
      <cmdty:id>SPY</cmdty:id>
    </price:commodity>
    <price:currency>
      <cmdty:space>CURRENCY</cmdty:space>
      <cmdty:id>USD</cmdty:id>
    </price:currency>
    <price:time>
      <ts:date>2019-05-10 16:00:00</ts:date>
    </price:time>
    <price:source>Finance::Quote</price:source>
    <price:type>last</price:type>
    <price:value>1213698188994648/4212767056559</price:value>
  </price>

====

  <price>
    <price:id type="guid">50657a1b214742068ab17c2bdac082ae</price:id>
    <price:commodity>
      <cmdty:space>NYSE</cmdty:space>
      <cmdty:id>VTI</cmdty:id>
    </price:commodity>
    <price:currency>
      <cmdty:space>CURRENCY</cmdty:space>
      <cmdty:id>USD</cmdty:id>
    </price:currency>
    <price:time>
      <ts:date>2019-05-10 16:00:00</ts:date>
    </price:time>
    <price:source>Finance::Quote</price:source>
    <price:type>last</price:type>
    <price:value>690402841721574/4679745419383</price:value>
  </price>
Comment 8 CDB-Man 2019-05-12 17:55:33 EDT
[2019.05.12 05:13:52] <gncbot> chris: Sent 6 hours and 49 minutes ago: <CDB-Man> uploaded my pricedb extract to the bug report, seems your suspicion about price corruption is right, finance::quote is storing overly precise numbers such as <price:value>690402841721574/4679745419383</price:value> that it seems gnucash is choking on, see https://bugs.gnucash.org/show_bug.cgi?id=797233#c7
[2019.05.12 05:14:35] <chris> CDB-Man: you'll have to ask jralls for a price-scrubbing addition to the file-load mechanism.
[2019.05.12 05:37:11] <chris> CDB-Man: if you load a recent win32 build, new pricedb entries should be saner. I don't know what to do about your current pricedb entries; this should be answerable by jralls.

Perhaps jralls can provide some insight.
Comment 9 Frank H. Ellenberger 2019-05-12 21:06:57 EDT
Comment on attachment 373276 [details]
Tracefile

Hm, bugzilla does not understand MIME text/x-log.

CDB-Man, can you adjust the MIME Type to text/plain in the future instead of octet/stream?
Comment 10 CDB-Man 2019-05-12 22:04:36 EDT
@Frank: Noted -- I had relied on the "auto detect" to pick the right file type, looks like that didn't work so well.  Thanks for re-uploading.
Comment 11 Christopher Lam 2019-05-12 23:44:37 EDT
CDB-Man: while dev resources are in very short supply it would be perhaps best that you fix your pricedb manually meanwhile.

690402841721574/4679745419383 can be replaced to 14753/100 exactly.

There was a recent commit to fix something in quote so this bug may have been fixed already: https://github.com/Gnucash/gnucash/commit/95aed3032 but it won't fix existing pricedb entries.
Comment 12 John Ralls 2019-05-13 17:37:50 EDT
Chris is right that Mike Alexander's commit should prevent goofy numbers like from FQ going forward. Since you're on Windows you can just grab a recent nightly.

If you fix the three really big numbers does your file work again? I'm not sure that I can construct a good test to safely scrub prices like that out of a file.
Comment 13 CDB-Man 2019-05-13 20:41:02 EDT
Thanks jralls; after manually scrubbing pricedb of approx 50 entries with unusually long values, all the reports work.

I went to maint
> https://code.gnucash.org/builds/win32/maint/
And I'll download the following nightly:
> gnucash-3.900-2019-04-25-git-3.5-106-g0cf49f124+.setup.exe

I'll run Finance::Quote on that and see if it keeps pricedb clean.

===
On an unrelated matter, can we delete the tracefile attached to message #6?
> https://bugs.gnucash.org/show_bug.cgi?id=797233#c6
Since it was ran with --debug --extra it actually contains all my account names and a bunch of other data.
Comment 14 CDB-Man 2019-05-13 20:41:54 EDT
Comment on attachment 373278 [details]
#3 Tracefile

>> Deleted <<
Comment 15 CDB-Man 2019-05-13 20:44:59 EDT
More specifically https://bugs.gnucash.org/attachment.cgi?id=373278 attachment 373278 [details], some of the GnuCash account names have partial actual account numbers.
Comment 16 CDB-Man 2019-05-13 20:49:26 EDT
Nightly gnucash-3.900-2019-04-25-git-3.5-106-g0cf49f124+.setup.exe still returns goofy numbers such as the below:

<price:value>98421868895636/2267262586861</price:value>

I'll hold off on running Finance::Quote until we get a bit more clarity on this.
Comment 17 Christopher Lam 2019-05-13 20:52:48 EDT
There are newer nightlies. Check the last-modified dates.
Comment 18 CDB-Man 2019-05-13 21:06:40 EDT
I see it now, a bit misleading that the April file is 3.900-xxx where as the May file is 3.5-xxx.

I ran:
> gnucash-3.5-2019-05-10-git-3.5-170-gd62489409+.setup.exe
And it still generated this:

====
  <price>
    <price:id type="guid">fa09f2bf4759469fa6fc84d4afd17cde</price:id>
    <price:commodity>
      <cmdty:space>NASDAQ</cmdty:space>
      <cmdty:id>ATVI</cmdty:id>
    </price:commodity>
    <price:currency>
      <cmdty:space>CURRENCY</cmdty:space>
      <cmdty:id>USD</cmdty:id>
    </price:currency>
    <price:time>
      <ts:date>2019-05-13 16:00:00</ts:date>
    </price:time>
    <price:source>Finance::Quote</price:source>
    <price:type>last</price:type>
    <price:value>98421868895636/2267262586861</price:value>
  </price>
====
Looks like it's still an issue; don't think I'm missing out on a newer version this time. :)

====
> On an unrelated matter, can we delete the tracefile attached to message #6? > https://bugs.gnucash.org/show_bug.cgi?id=797233#c6
> Since it was ran with --debug --extra it actually contains all my account names and a bunch of other data.

As a regular user, it looks like I can only mark things obsolete, not delete them.
Comment 19 John Ralls 2019-05-13 23:32:13 EDT
> I see it now, a bit misleading that the April file is 3.900-xxx where as the May file is 3.5-xxx.

That's my fault. I merged a PR that was on master into maint and didn't notice.

> As a regular user, it looks like I can only mark things obsolete, not delete them.

That's not just you. BZ doesn't allow deleting anything.

Mrmph. 43.41. And
>>> from decimal import Decimal
>>> Decimal(98421868895636)/Decimal(2267262586861)
Decimal('43.40999999999999558939486853')
Comment 20 CDB-Man 2019-05-13 23:45:56 EDT
No worries re: the versioning.  Looking forward to a later nightly where the fix is in place.

Regarding deleting attachments, doing so Google searching, I found this:
https://devzing.com/blog/index.php/bugzilla-tip-deleting-attachment/

It details how administrators can be granted deletion rights on Bugzilla.
Comment 21 Christopher Lam 2019-05-14 00:38:28 EDT
to help reproduce this bug on recent windows, it *will* be useful to know the exact sources for the quotes used.
Comment 22 CDB-Man 2019-05-14 01:25:11 EDT
All stock prices come from the Yahoo JSON option.  Currency pricing from AlphaVantage.
Comment 23 Frank H. Ellenberger 2019-05-14 02:00:20 EDT
The content of attachment 373278 [details] has been deleted for the following reason:

User Request
Comment 24 John Ralls 2019-05-14 16:49:03 EDT
The problem turns out to be that Mike Alexander misunderstood what Scheme's rationalize and inexact->exact functions do. If the number is floating point the damage is already done, so the fix is to have gnc-fq-helper tag the price with #e. See inexact->exact in https://www.gnu.org/software/guile/manual/guile.html#Exactness.

I've pushed the change. If you want to test without getting a new build just change line line 189 in gnc-fq-helper from

  if($numstr =~ /^\s*(\d+(\.\d+)?([eE][+-]?\d+)?)$/o) {
    return $1;
  } else {

to

  if($numstr =~ /^\s*(\d+(\.\d+)?([eE][+-]?\d+)?)$/o) {
    return "#e" . $1;
  } else {


Chris, is there anywhere else you can think of where we might be inadvertently putting floating-point values into Scheme?
Comment 25 CDB-Man 2019-05-14 21:17:14 EDT
Jralls, that worked.  After running FQ again and checking the raw priceDB data, no goofy values were inserted.
Comment 26 Christopher Lam 2019-05-14 21:35:03 EDT
@jralls none in my travels across schemeland.
Comment 27 John Ralls 2019-05-14 23:19:15 EDT
CDB-Man, shall we declare this fixed?

Chris, OK. Something of which to be mindful.
Comment 28 CDB-Man 2019-05-14 23:21:30 EDT
Indeed, sounds fixed with the latest commit that you've pushed.
Comment 29 John Ralls 2019-07-18 23:49:58 EDT
*** Bug 797311 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.