Tools / Price Database / Currencies UI has become unusable since 3.4 It should be obvious to the eye that it isn't working though I can't quite work out what it is that has changed. The exchange rates themselves seem to be OK. I haven't checked but I think it probably affects all operating systems. This has been confirmed in the dev list where Mike Alexander also said "it seems to be impossible to edit a transaction when the price editor is open. It is permanently in front of the register window and seems to steal all keyboard events." I haven't checked that as I am staying on 3.3 because in combination with the unreliability of getting rates from Alphavantage it is more convenient for me. Mike's point may be unrelated.
I don't see any issues with using the price database on Windows, including with switching focus. What are you having trouble doing?
Ah, revisited your email on the list... I don't have any multi-currency accounts with a significant number of currency quotes; in general just one or two because they're all test files. If you create a bunch of quotes in a new test file, save, close, and reopen, do they all show up?
You're missing the point, John. MikeA sees the same thing I do. The modal stuff may be separate. Would you like me to test 3.4 again? I've gone back and forth before, it'll produce the same result. You need to give me something specific to test to make it worth the effort.
I don't think I'm missing anything except understanding the problem well enough to reproduce it. Mike isn't actively developing any more. You might be able to answer the question without further testing. What I'm looking for is whether I should create a bunch of exchange rates in an earlier version (which one and how many rates?) and then start GC 3.4 or if I can create some prices in 3.4 and immediately see the problem, or quit and restart and then see the problem.
Created attachment 373131 [details] Test file demonstrating the problem I've attached a GNuCash file that seems to demonstrate the problem. It is a compressed XML GnuCash file which contains 26 currencies and 16212 quotes for them. If you bring up the price DB dialog in 3.4 it only shows a few hundred of the quotes even though they are all there and usable. I tested this in the Quartz version of GnuCash built from the maint branch a few days ago. It was running on MacOS version 10.13.6 (17G4015). I don't see the same problem in 3.3. In that version there is a separate problem that makes it take a very long time to load the price DB, but it does load. The 3.4 version loads its partial DB very quickly.
is it possible this is connected to the Chris Carson fix?
Perfect, thanks. I wouldn't have thought that scale necessary nor had the patience to generate it. I don't think that the Chris Carson patches would have anything to do with this, but a bisect will take me to what does pretty quickly.
I have no idea if that scale is necessary, it's just what I had. That file was generated by taking the currency exchange rates from my regular file and pasting them into an almost empty file. I didn't make any effort to pare them down to the minimum.
It's a good test in that it not only shows the problem (which is https://github.com/Gnucash/gnucash/commit/e81bcf6e33bc5bcf2af8aca6931e537889e1a17a#diff-ec09a0bd9076d5f586d996c0bbebcf7fL2213) but also shows a pretty bad performance problem.
I pushed a fix for the enumeration problem that fixes the actual bug here. Wm, it's in gnucash-3.4-2019-01-20-git-3.4-38-g020bc5371+.setup.exe so you can test it. It doesn't do anything for the performance problem. Profiling shows that to be half qof_log_check and half g_sequence_append called in gtk_tree_model_filter_refilter.
I have not looked at the code but that sounds like the model has not been dropped from the tree view while prices are added and maybe while re-filtering it... The modal stuff is due to clearing transient parents warnings in version 3.x, I think the dialogues need changing to windows. If you want I could have a look?
Bob, I'm not up-to-speed on GtkTreeModels, so please do have a look at that part and see if you can speed it up. I do understand containers so I'll try to figure out how to speed up qof_log_check.
gnucash-3.4-2019-01-20-git-3.4-38-g020bc5371+.setup.exe has been running for more than 15 minutes after I tried Tools / Price Database with my own file, had to kill it. Let me try again with Mike's test file. Mike's file gives me a run time error after I try Tools / Price Database I think the puzzle is that nothing is meant to have changed wrt this bit of code between 3.3 and 3.4
rats, Mike's test file doesn't work using 3.3 for me. did anyone else use it successfully and if so on which OS?
I wish the code was less opaque so I could actually help fix the problem I am reporting :(
Yes, I've been using Mike's file, but on MacOS. I'll try it on Windows a bit later.
John, I tried Mikes file on my Linux VM and it locked up / crashed. I thought it was the number of prices and halved them but still it would crash. I ended up with 5 prices and that would crash but with no prices or 1 price it would open, with 2 it would lock up and eventually crash. Very weird as I created a new file and used the price importer to import 3 currencies each with 450 prices. This all distracted me from what I was going to do in the first place.
John, you've got some intelligent people looking at this, please indicate *where* we should be looking.
Bob, that's interesting. Can you get a stack trace from the crash? Wm, libgnucash/engine/pricedb.c (note the commit linked in comment 9), gnucash/gnome-utils/gnc-tree-model-price.c, and gnucash/gnome-utils/gnc-tree-view-price.c. You'll need to understand GtkTreeView (https://developer.gnome.org/gtk3/unstable/TreeWidgetObjects.html) for how the functions interact. The logging code that's half of the slowdown is in libgnucash/engine/qoflog.cpp.
aside: is anyone else wondering why Mike's file worked on MacOS but not Win or Linux? gnc is meant to read the same file regardless, or did that get dropped and I didn't notice?
*** Bug 797062 has been marked as a duplicate of this bug. ***
Created attachment 373139 [details] Zip file containing, gnucash file and trace files It looks like the problem for me is when there are prices for a currency/commodity in more than one currency. I created a new empty file and imported two prices... "CURRENCY","YYYY/MM/DD","AMOUNT","CURRENCY" "AFA","2015/01/02",1.5361,"GBP" "AFA","2015/01/05",2.5234,"AUD" and then saved the file. Opened this up in 2.6.21 with logging and all was OK. Tried the same with current 3.4x and Gnucash just sits there, needed to kill Gnucash to stop it. The zip file contains, the test Gnucash file, a jpg from 2.6.21 and the two log files. Looking for a difference between log files, the 3.4x log file is repeating this command gnc_tree_model_price_iter_next for ever with the same price so I think the problem may be in the representation of the pricedb in the price model. Still looking...
I think what I will do is try some windows nightly builds to find where it stops working...
Does display prices correctly... gnucash-3.3-2018-11-29-git-3.3-132-gde6c173ef+.setup.exe Does not display prices correctly... gnucash-3.3-2018-11-30-git-3.3-151-g5dcb44d99+.setup.exe
Found the problem in this commit.. https://github.com/Gnucash/gnucash/commit/e81bcf6e33bc5bcf2af8aca6931e537889e1a17a In that commit there is a change to engine/gnc-pricedb.c, all I have done is put the original back in to a current checkout and the prices are shown correctly, I can even load a smaller version of Mike's file, 8000 prices. John, can you see any mistakes in that commit? Have not looked at it much so not sure what was trying to be achieved. Regards, Bob
Yes, I did that exercise with git bisect last week. See comment 9. I fixed the mistake in https://github.com/Gnucash/gnucash/commit/020bc5371f53bdef057d3ce9fcd115b8e6511658. A full reversion regresses the array overrun that it was meant to fix, but if it's stalling out on Windows then I need to look at it some more. With it reverted can you load Mike's full 16000-price file? I tested the January 20 build on my Windows VM and it crashed for being unable to allocate a slice. There was still plenty of memory left in the system so I suspect a problem in the slice implementation. I haven't dug into that yet.
I was using that change in 'comment 9' on my Linux VM and it was still causing problems. Reverted all the changes in the commit I mentioned and can now load Mile's file of 16000 prices, it takes a while, need to get a better machine with more memory, but it gets there with a full tree of prices. So I think there is another change required.
Long before this problem started happening I noticed that the PriceDB dialog is very slow to load and slow to expand one of the categories if there are many prices for a single commodity in multiple currencies. I think that's what you're seeing now and I doubt that a faster machine or more memory will help too much. I've been meaning to look into this, but haven't. I do know that deleting the currency quotes (which is where the prices in multiple currencies are) makes the dialog load essentially instantaneously even though there are still around 23,000 prices in the PriceDB.
OK, I've reverted the rest of the changes from e81bcf6e in gnc-pricedb.c and pushed.
The code in question is indeed the multi-currency for a commodity code, but when I profile it it's the ENTER and LEAVE calls that are sucking up the time, not double-iterating through the price lists.
I think Bug 797062 is a match, I have a handful of p2p prices I maintain manually that I thought too obscure to mention before.
ff on from MikeA's comment 28 above: is it possible rusty db's could be the issue? I thought there was a rethink a while back such that we (gnc) collected minimal prices and worked out some. I certainly haven't had prices from both sides of some commodities for a while now so I sort of presumed that was happening. e.g. my home currency is GBP (I own stuff in that), I also own a commodity priced in USD and another in EUR, those are major currencies, cross rates will probably work as those are efficient markets. Cross rates (and working back from them) become less normal as you get further away from major markets.
on Comment 26 I think it worked for only one OS.
I am aware I may be overridden, I have changed the ff Hardware: All Importance: High
Neither really matters at this point, though it's certainly correct that all platforms are affected. The only thing we use the Importance fields for is to flag enhancement requests. On "minimal price collection" we decided a few years ago to collect only one a day for each commodity/currency pair; direction is significant, so you can have a price for EUR->USD and USD->EUR on the same day, and of course EUR<->GBP as well. BTW it's common in the code for "the thing being priced" to be called "commodity" and "the denomination of the price" to be called "currency" even though either can be anything.
Good work, Bob
aside: I've never had an interactive bug clash before in many years, but just got one here a minute or so ago. I think it was a system trying to decide something rather than a person.
When I get those it's usually because I forgot to refresh the page after adding/changing something. BZ doesn't like it if you comment on the acknowledgement page.
JohnR: I was just double checking on the "minimal price collection" issue. All: I suggest people should go back to the lists rather than continue here as there is an ongoing conversation about it. Throw your point in on the lists, let's see what we all think. Good news? I'm the one that gets curtailed for naughty words, etc. so go for it, your voice gets heard more! :)
Wm, did you test today's nightly?
With the full reversion the profile results change substantially, with logging no longer in the hot path. Now it's compare_prices_by_date(), with the most significant contributors being qof_instance_get_type(), boost::uuids::operator<(), and gnc::GUID::GUID(_gncGuid&).
JohnR when I look at the thread stuff I'm seeing last 5 days ago, I looked at that before. I think the question is, which file should I use :) I'm going to try to find the nightly and hope I get the right one.
Wm, try https://code.gnucash.org/builds/win32/maint/gnucash-3.4-2019-01-24-git-3.4-40-g3a4867276+.setup.exe.
(In reply to John Ralls from comment #41) I have also found I can half the load time by dropping the model from the view at setup so on my VM Mike's file loads in 5 minutes from 11 minutes. Still looking at other possibilities.
Unfortunately I don't see any leverage for improving the performance of compare_prices_by_date, but I wonder if gnc_pricedb_nth_price is even working the way Mike intended. Consider the GBP prices in his sample: There are a couple-hundred quotes from 1989-last Friday, all but two in USD; the two are in CAD and dated 22AUG2017 and 30MAY2015. I think Mike's intent was to present them all in date order, but the PriceDB tree view is showing the CAD pair first followed by all of the USD quotes in date order.
Created attachment 373140 [details] Clarify code and cache results. The major slowdown was from re-calculating the price list each time. Simply replacing the hand-rolled loops with GList functions to concatenate all of the different currency pricelists together and sort the result provided a small speed-up. But if there are n prices (and there are 12,663 USD prices in 24 "other" currencies in Mike's test file) then gnc_pricedb_nth_price gets called n times, generating and sorting the same list n times. A simple cache provided an enormous speedup, so that the dialog now loads in 6 seconds instead of 13 minutes.
A little testing with the sort order in the debugger shows that the order of the list doesn't matter. The tree view is sorted by "other" currency then descending date regardless of what order the list is in. Investigation shows that sorting is handled by gnc_tree_view_price.c's default sort, so there's no point in sorting the pricelists in gnc_pricedb_nth_price. However, even with the sorting removed it still takes over a minute to load without caching and only few seconds with it, so I've left the caching in. I've confirmed that it works on Windows and pushed. It will be in tomorrow's (i.e. 2018-01-26) nightly. Bob, it opens with the list collapsed and reloads everything when one clicks the disclosure triangle. Can we do away with the first load?
Yes, it loads much faster with this change, thanks. I've been meaning to look at this for ages, but not being able to see well has slowed me down a lot. I really think that the sort order used to be correct (date first then other currency). I gather that can't be fixed easily. The large number of prices in this file for USD is the result of the change made in 923b01e2 from Sept. 1, 2015 which caused GnuCash to get a reverse exchange rate if the requested rate is less than 1. I would suggest changing this to .1 or even .01. That's still going to give plenty of significance in results and will reduce drastically the number of reversed exchange rates. That test file has exchange rates for 24 currencies, all meant to be based on USD. Of these 24, 22 are reversed because the rate is less than 1. Of these 22, 10 are less than .1 and 6 are less than .01. Reducing the threshold to .01 wouldn't lose much significance (none of them have much to start with) and would get rid of most of the reversals.
The sort order isn't necessarily hard to fix, it just doesn't use the order that gnc_pricedb_nth_price provides so there's no point in wasting time sorting there. There are in fact several sort routines in gnc_tree_view_prices.c. I haven't looked to see how to switch them. "None of them have much to start with" is exactly the reason for the change. Users were complaining about large rounding errors because of low-precision exchange rates. FWIW, Alphavantage (the new hard-wired currency source in Finance::Quote) returns only one direction on currency quotes reqardless of what you ask for and it's the "larger" in terms of the "smaller'; e.g 1 GBP to 1.1569 EUR, 1 EUR to 1.1405109 (!) USD, 1 USD to 4.1225 MYR. Once the price is converted to a rational number it's equally precise for calculation (though not for display if the user has force-to-decimal set) regardless of what's the numerator and what's the denominator so we could consider storing it the way that makes sense for the accounts.
I know that's why you changed it to store reverse quotes. My point is that you don't need to be so aggressive to get the benefit. In fact I'm not sure you need to do it at all anymore (see below). I also realize that once you get a rational number getting a reciprocal is trivial. The hard part is converting the decimal number you get from F::Q to a rational number, in particular picking a reasonable denominator. Back when GnuCash used 32 bit numbers for numerators and denominators overflow was a problem, but now that GnuCash uses 64 bit numbers this isn't such a problem and a much wider range of numbers can be converted to rationals and used in calculations. What do you mean that AlphaVantage only returns quotes in one direction? I just tried it with IDR and USD and get { "Realtime Currency Exchange Rate": { "1. From_Currency Code": "USD", "2. From_Currency Name": "United States Dollar", "3. To_Currency Code": "IDR", "4. To_Currency Name": "Indonesian Rupiah", "5. Exchange Rate": "14160.00000000", "6. Last Refreshed": "2019-01-26 00:29:05", "7. Time Zone": "UTC" } } { "Realtime Currency Exchange Rate": { "1. From_Currency Code": "IDR", "2. From_Currency Name": "Indonesian Rupiah", "3. To_Currency Code": "USD", "4. To_Currency Name": "United States Dollar", "5. Exchange Rate": "0.00007062", "6. Last Refreshed": "2019-01-26 00:30:25", "7. Time Zone": "UTC" } } This is using a URL like https://www.alphavantage.co/query?function=CURRENCY_EXCHANGE_RATE&from_currency=USD&to_currency=IDR&apikey=**hidden** which is what F::Q uses. After a bit of experimenting I think it is probably ok to let GnuCash use quotes with very small values. I disabled the code that gets reverse quotes and had GnuCash get new quotes for QuoteTest.gnucash. For IDR->USD it got from gnc-price-helper ((IDR (symbol . IDR) (gnc:time-no-zone . 2019-01-26 00:09:48) (last . 7.062e-5) (currency . USD))) It recorded this as a price of 3531/50000000 which is .00007062. The pricedb dialog displays this as .000071 but this is really just a display problem and could be fixed in various ways. That dialog doesn't seem to respect the "Force Prices to display as decimals" option, by the way. For the heck of it I tried this with IRR (Iranian Rial) which is said to be the weakest currency in the world. F::Q gives 1 USD = 42000 IRR and 1 IRR = 2.381e-05 USD. GnuCash stored the rate (without reversing it) as 2381/100000000 which is correct. I'm going to disable the code that reverses currency quotes in my copy of GnuCash and see if I see any issues.
This is what I get: $ Argus:/Users/john> echo '(currency "IRR" "USD")' | /Applications/Gnucash.app/Contents/Resources/bin/gnc-fq-helper (("USD" (symbol . "USD") (gnc:time-no-zone . "2019-01-26 08:37:19") (last . 42000) (currency . "IRR"))) $ Argus:/Users/john> echo '(currency "USD" "IRR")' | /Applications/Gnucash.app/Contents/Resources/bin/gnc-fq-helper (("USD" (symbol . "USD") (gnc:time-no-zone . "2019-01-26 08:37:38") (last . 42000) (currency . "IRR"))) How did you get gnc-fq-helper to give you the IRR->USD quote? Otherwise we agree: The direction of the quote from F::Q doesn't matter. We can convert it to rational and store it in the direction it was asked for, thanks to GnuCash 3's wider rational numbers.
Wm, when you have time could you please test https://code.gnucash.org/builds/win32/maint/gnucash-3.4-2019-01-26-git-3.4-43-g84d1c3645+.setup.exe or later?
You were testing gnc-fq-helper, I was testing AlphaVantage. Your response said "Alphavantage ... returns only one direction on currency quotes regardless of what you ask for" so I tested AlphaVantage directly. I know that gnc-fq-helper always returns the direction that gives a number greater than 1, that's the change in 923b01e2 that I mentioned earlier. That's the source of my confusion, so I guess we agree entirely now. Shall I push the change to revert the part of 923b01e2 that forces exchange rates to be greater than 1?
Ah, I'd forgotten that. I thought that I'd changed pricedb.c to force the > 1 direction. If you can ensure that we preserve the full precision of the fractional price, sure.
The conversion code asks for 9 significant digits in the result. With a 64 bit denominator I think it can get that with values down to 1e-9 (.000000001) which I think is good enough. Especially since you probably only need about 5 significant digits since that seems to be about what AlphaVantage gives you.
Just call double_to_gnc_numeric (x, 10e18, GNC_HOW_DENOM_REDUCE | HOW_ROUND_NEVER) and it will do the right thing.
Yes, I know that the current code is not optimal. There is a fixme comment to that effect in price-quotes.scm. I was off by one in my calculation of the smallest price that can be represented with 9 significant digits, it's better than I thought. Here are the results of some tests: .123456789e-8 = 123456789/100000000000000000 .123456789e-9 = 123456789/1000000000000000000 .123456789e-10 = 6172839/500000000000000000 .123456789e-11 = 1234567/1000000000000000000 I'll revert the change to reverse quotes less than 1 and look at fixing the code to not use GNC-DENOM-SIGFIGS.
Sounds good. Hmm, we should be getting one more digit from GncNumeric/gnc_numeric. Removing GNC_DENOM_SIGFIGS entirely is on my todo list. That's a behavior that doesn't belong in an accounting program.
I think you want a denominator of GNC_DENOM_AUTO instead of 10e18 in the call to double_to_gnc_numeric. If you use 10e18 any number greater than about 9.22 gets an overflow error. When I use GNC_DENOM_AUTO I get .123456789e-8 = 123456789/100000000000000000 .123456789e-9 = 123456789/1000000000000000000 .123456789e-10 = 6172839/500000000000000000 .123456789e-11 = 1234567/1000000000000000000 123456.789 = 123456789/1000 This is with GNC_HOW_DENOM_REDUCE | HOW_ROUND_NEVER as you suggested. Does this seem right to you?
Mike, Aside from the comment on double_to_gnc_numeric about GNC_DENOM_AUTO not being allowed, yeah. But the comment is old and the new-for-3 implementation just constructs a GncNumeric and calls convert on it, so I don't think it applies anymore. 9.22e18 is INT64_MAX, so the overflow shouldn't be a surprise. I was thinking a digit off.
Just tried gnucash-3.4-2019-01-29-git-3.4-50-g62a4e73f7+.setup on a cp of my main book. Loaded fine Existing exchange rates look good Collected prices fine, (my F::Q has AV delays) Checked exchanges rates Looking good. The conversation seems to have moved on from what initially appeared to be a display issue so: 1. is there anything specific I should test? 2. is this OK to use on live data or not ? my reason for sticking with 3.3 certainly appears to be fixed Thanks for work so far, all
Wm, yes, as long as dates (see bug 797067) look OK it should be fine to use on production data.
I'm GMT at the moment so I should be safe from TZ weirdness.
I had an oddity after I collected prices very early this morning because a number of my reports didn't work, if I deleted the most recent prices in the SQLite file the reports worked again. Tried again today when I had some time to try and work out what happened and all was well. Made me think about the date bug above. Would it make sense that prices collected close to midday GMT would be OK but those collected close to midnight GMT might cause problems? Not an issue if I can only get prices (say) 08:00 to 16:00 GMT, just thought I'd mention it. -- Wm
What went wrong with the reports? Did it seem that the prices were for the wrong day?
I expect an F::Q session to get prices dated today, yesterday and sometimes even 2 or 3 days ago, not everything is priced *now* or daily; all the dates were yyyy-mm-dd and manually deleting the most recent ones collected since the previous known good collection on 2019-02-01 allowed the the reports to work again. Hmmmn, having just written that I'm wondering if it was something other than the dates; it was definitely something to do with prices as I was able to repair it simply by sorting the prices table by date and deleting the most recent entries. The reports didn't show at all, as if no parameters had been set, I didn't note the actual message (my bad). If no one else is noticing this leave it with me until / if it happens again. === Another issue is that if a price is edited (I get 2 prices that show up with the decimal wonky, e.g. VVAL arrives as 0.2553 rather than 25.53 and I need to move the decimal point) the price disappears from the UI, the table is updated fine so I'm OK with it but it will probably disconcert someone in the future and it may be best fixed while minds are still fresh about this bit of code. find a price / edit / change the price / Apply or OK; if you are seeing what I'm seeing the price isn't there until you move away and back.
stet recent posts gnucash-3.4-2019-01-29-git-3.4-50-g62a4e73f7+.setup is ok gnucash-3.4-2019-02-01-git-3.4-69-gd048caeda+.setup was troublesome. I've gone back to 3.4-50 and all seems well, I tested prices and reports before and after midnight.
I saw a similar problem in the Price Database with the initial 3.4 Windows release. The problem I was seeing was that I could not open up all the securities to see the prices. Some of the securities did not have the 'right arrow' icon to the left of the security which one needs to click to view the prices. I only have 1 currency (AUD). Installing the latest Windows Dainly Maint build (gnucash-3.4-2019-02-08-git-3.4-72-g67dbfca0e+.setup.exe) has fixed.
Yes, the Trial Balance report balances only with the Average Cost price source. See bug 775368 for a very detailed and long-running analysis. Note from that bug that Average Cost is broken in several releases of GnuCash.
Hi John, Was your comment 69 for this bug?
Chris, no, it's sort of for bug 797097 except that it makes sense only in the context of the discussion on gnucash-devel. Sorry for the noise.
I've just installed gnucash-3.4-2019-02-10-git-3.4-73-g7d7da8e2c+.setup reports seem fine (which is the important bit) the minor display problem if a price is edited manually is still there but I doubt many people will notice it, recipe: Get Quotes Edit the most recent price for something (I need to move a decimal) Press OK The price appears to disappear. Pressing Apply produces more expected behaviour. If you close and reopen the Price Database window all appears well. Not a big deal but probably best to fix it while this bit is being thought about. === Have responded in bug 797097 to other issue (haven't looked at the list yet).
The reports problem I mentioned in #66 above has happened again. e.g. the Balance Sheet returns === Report error An error occurred while running the report. === the eguile Balance Sheet works. If I delete the price below === <price> <price:id type="guid">6be93247806e4d80afbce45ed3b0d0ba</price:id> <price:commodity> <cmdty:space>CURRENCY</cmdty:space> <cmdty:id>KZT</cmdty:id> </price:commodity> <price:currency> <cmdty:space>CURRENCY</cmdty:space> <cmdty:id>GBP</cmdty:id> </price:currency> <price:time> <ts:date>2019-02-22 05:05:16</ts:date> </price:time> <price:source>Finance::Quote</price:source> <price:type>last</price:type> <price:value>2004999999999999/1000000000000000000</price:value> </price> === the Balance Sheet works again. No other change other than deleting that price. My guess is this === <price:value>2004999999999999/1000000000000000000</price:value> === is the cause of the problem. Not sure why that isn't 2005/1000000
I looked through my prices (which go back many years) and found these three that look unusual; note: I also deleted some prices when I got stuck earlier while working this out, this is what is left in my book. Possible clues: they are all recent [2019-01-30, 2019-02-21, 2019-02-05] the KZT:GBP rate mentioned above and the TRY:GBP rate below are from AV and could theoretically have a lot of precision but in both cases the numerator ends 499999 etc which makes me think there may be a rounding error of some sort. I'm not sure 499999 etc adds much precision vs 5 but IE00B50MZ724 and GB00B4PQW151 are Vanguard Mutual Funds collected from mstaruk, as far as I know these aren't normally trade-able as units beyond 2 decimals for most people though when you buy or sell you may get 4 decimals back on your contract note or statement. I thought me having to change the decimal of some my Vanguard ETFs collected from AV might be relevant but it isn't. I have hundreds of prices up to 52 chars in length including leading space, e.g. <price:value>855000000/10000000000</price:value> which gnc handles fine. the ones below stand out at 62 and 65 chars <-- that is a *lot* of probably dodgy additional precision and more than the standard Balance Sheet knows what to do with. I think the prices are wrong rather than the Balance Sheet if anyone is interested. === <price> <price:id type="guid">bb272a7dfa8d4024b1417a0f39624a8f</price:id> <price:commodity> <cmdty:space>CURRENCY</cmdty:space> <cmdty:id>TRY</cmdty:id> </price:commodity> <price:currency> <cmdty:space>CURRENCY</cmdty:space> <cmdty:id>GBP</cmdty:id> </price:currency> <price:time> <ts:date>2019-01-30 07:31:47</ts:date> </price:time> <price:source>Finance::Quote</price:source> <price:type>last</price:type> <price:value>8953124999999999/62500000000000000</price:value> </price> <price> <price:id type="guid">dc7b9e87f2974949a25dc1224fd9c4de</price:id> <price:commodity> <cmdty:space>Vanguard</cmdty:space> <cmdty:id>IE00B50MZ724</cmdty:id> </price:commodity> <price:currency> <cmdty:space>CURRENCY</cmdty:space> <cmdty:id>GBP</cmdty:id> </price:currency> <price:time> <ts:date>2019-02-21 12:00:00</ts:date> </price:time> <price:source>Finance::Quote</price:source> <price:type>last</price:type> <price:value>8637109375000001/39062500000000</price:value> </price> <price> <price:id type="guid">1864e05567b7474c85c252f1e4166a05</price:id> <price:commodity> <cmdty:space>Vanguard</cmdty:space> <cmdty:id>GB00B4PQW151</cmdty:id> </price:commodity> <price:currency> <cmdty:space>CURRENCY</cmdty:space> <cmdty:id>GBP</cmdty:id> </price:currency> <price:time> <ts:date>2019-02-05 12:00:00</ts:date> </price:time> <price:source>Finance::Quote</price:source> <price:type>last</price:type> <price:value>7632421874999999/39062500000000</price:value> </price> ===
It looks to me there are at least two things that suggest this should be reopened. 1. the actual price:value combinations being recorded are breaking that most basic of reports, the Balance Sheet. For reference I don't think the report is wrong in this case. 2. there is a smaller UI issue I mentioned in #72 above, that is still there I'm not sure what the protocol is on who marks something un-RESOLVED or INCOMPLETE or if a new bug should be opened. Advice, please.
Further thoughts I think the conversation between Mike and John (most recently #59 and #60 above) is incomplete. The good news is I think the problem might be confined to the price db, if it escapes to actual tx the peasants [1] will revolt. [1] Trump base, it is a joke, they don't seem to add and subtract much let alone understand division! Why is this good news? Because the price.db is not actually germane to the working of gnc. We have a chance to fix these assumptive sums before they get inside real life tx. I am unlikely to be affected because I generally use value to value rather than relying on a commercial rate. Someone else might not have that insight, collecting a price and using it would see to be a safe option for many people in this world. gnc is in the position of creating a bad thing at the moment, it is recording stuff in the prices.db that it can't use. if someone uses that broken price (they collected prices, they trust gnc) they then have a broken price in an actual tx. I think it is better we don't get to that point.
F::Q / AV / KZT:GBP pulled a price that broke the standard Balance Sheet (not the eguile one) and related reports again. Reflecting on what I said before the reason so many reports are affected is because the Balance Sheet is used as a skeleton for a number of other standard reports and that is compounded by me constructing Multicolumn reports which often include a cut down Balance Sheet. Good news is I knew where to look this time and just put the XE.com rate in, I'd be surprised if anyone really needs the umpteen significant digits but do understand the desire to be precise wrt securities with very high and low differences to normally-two-decimal commodities. So, not an ongoing problem for me but it will be very confusing for the next person that encounters it unless fixed and tracing it back to the price database may not be transparent. Changing the Balance Sheet report (which is overloaded generally) is, I suppose, the very scary prospect.
Balance Sheet and related reports broken again <price> <price:id type="guid">e6c13de656e1428f98763d45706b99ff</price:id> <price:commodity> <cmdty:space>CURRENCY</cmdty:space> <cmdty:id>KZT</cmdty:id> </price:commodity> <price:currency> <cmdty:space>CURRENCY</cmdty:space> <cmdty:id>GBP</cmdty:id> </price:currency> <price:time> <ts:date>2019-03-18 02:52:33</ts:date> </price:time> <price:source>Finance::Quote</price:source> <price:type>last</price:type> <price:value>1965999999999999/1000000000000000000</price:value> </price>
That would be bug 797106. Mike pushed a fix for it a month or so ago, but there seem to be cases that evade the fix.
OK, I'll follow that and see what happens.
My fix only fixed the conversion from decimal values from Finance::Quote to rational values internally. As I said in bug 797106 comment 17 the reports need fixes too. There was a discussion about how to do that in comments 14 through 18 there. I assumed that someone else would take care of that.
Mike, right, but Wm's examples here shows that something's still not working in the conversion: .001966 got turned into <price:value>1965999999999999/1000000000000000000</price:value>. That's from the data file, not a report. Wm, I guess by "broken" you mean a 1p imbalance from a rounding error. Is that right?
No, by "broken" I mean the Balance Sheet and other reports that depend on it are as per 73 above. It isn't a 1p or 1c issue, the Balance Sheet simply doesn't work. I get a tab that says === Report error An error occurred while running the report. === I'm not making this up! The fix is, as I have said above, to correct the price to something a person and the author of the Balance Sheet might understand. If I go to the price.db, find the offending price and give it a value the Balance Sheet understands it all works again. I have long supported the price.db being external to the book, it makes no sense to me for the two to be joined; we're borrowing from other projects, why reinvent the wheel? In particular, why reinvent the wheel when we do it worse than AV and F::Q combined ? It is embarrassing. Why are we converting perfectly usable values from F::Q to numbers we can't use in our own reports? That makes no sense to me. I think I am more sympathetic to Mike's POV at the moment, i.e. the reports need fixing.
Hmm, I plugged the price from comment 78 into a test file and it didn't crash the balance sheet. The tracefile should have a scheme stack trace from the error. Can you find it and either attach the tracefile or paste in the stack trace?
Hrmph. I misunderstood the guile documentation about exact, thinking that it did BCD decimals. WRONG. Decimals are always inexact, using IEEE754. So, for example, > (inexact->exact 28.03) $11 = 986217949649961/35184372088832 but > (rationalize 2803/100 1/1000000000000000) $12 = 2803/100 So what we really need to do is change gnc-fq-helper line 190 from if($numstr =~ /^\s*(\d+(\.\d+)?([eE][+-]?\d+)?)$/o) { return $1; to something like if($numstr =~ /^\s*(\d+)(\.(\d+))?([eE]([+-]?\d+))?)$/o) { and use the pieces to return the number in rational format. Mind, that still doesn't completely explain Wm's latest weird number > (inexact->exact 0.001966) $14 = 2266643678057061/1152921504606846976 but it does explain the earlier ones.
(In reply to John Ralls from comment #84) > Hmm, I plugged the price from comment 78 into a test file and it didn't > crash the balance sheet. > > The tracefile should have a scheme stack trace from the error. Can you find > it and either attach the tracefile or paste in the stack trace? If a more recent version doesn't display this problem I'm happy to try that. Let's get our differences to be as few as possible before we actually disagree
(In reply to John Ralls from comment #85) > Hrmph. I misunderstood the guile documentation about exact, thinking that it > did BCD decimals. WRONG. Decimals are always inexact, using IEEE754. So, for > example, > > > (inexact->exact 28.03) > $11 = 986217949649961/35184372088832 > but > > (rationalize 2803/100 1/1000000000000000) > $12 = 2803/100 > > So what we really need to do is change gnc-fq-helper line 190 from > if($numstr =~ /^\s*(\d+(\.\d+)?([eE][+-]?\d+)?)$/o) { > return $1; > to something like > if($numstr =~ /^\s*(\d+)(\.(\d+))?([eE]([+-]?\d+))?)$/o) { > and use the pieces to return the number in rational format. > > Mind, that still doesn't completely explain Wm's latest weird number > > (inexact->exact 0.001966) > $14 = 2266643678057061/1152921504606846976 > but it does explain the earlier ones. I think you are making a fundamental error here, John. The prices are what they are, gnc doesn't own them; it should be using them. Normalising a tx is OK so long as the actual amounts are recognised. An actual tx must take precedence, you imposing your will on the accounting view is a separate matter. gnc *MUST* record the actual tx. We may argue about accounting standards but we should *NEVER* be in doubt about the recording of actual tx. Agreed? === Presuming we have some agreement above I don't see why gnc-fq-helper needs to be examined at all. The prices are external! === I think the real problem remains JohnR thinks his view of accounting is right and won't accept that there may be another view, he appears willing to fix everything except his opinion :(
(In reply to John Ralls from comment #82) > Mike, right, but Wm's examples here shows that something's still not > working in the conversion: .001966 got turned into > <price:value>1965999999999999/1000000000000000000</price:value>. > That's from the data file, not a report. This is exactly the result I get if I go back to using double-to-gnc-numeric to do the conversion. I put this (gnc:debug "Old way .001966 -> " (gnc-numeric-to-string (double-to-gnc-numeric 0.001966 GNC-DENOM-AUTO (logior GNC-DENOM-REDUCE GNC-RND-NEVER)) )) (gnc:debug "New way .001966 -> " (gnc-numeric-to-string (gnc-scm-to-numeric (rationalize (inexact->exact 0.001966) 1/1000000000000000)) )) into price-quotes.scm and got (edited to delete the prefix) <gnc.scm> Old way .001966 -> 1965999999999999/1000000000000000000 <gnc.scm> New way .001966 -> 983/500000 I wonder if Wm's quote was fetched with an old version of price-quotes.scm?
