Created attachment 374020 [details] Screenshot of same report with different step size The "Expense Chart" or "Income Chart" reports for de period of last year (2020) with step size of 'Month' is showing November 2019 instead of November 2020 and the amounts appear as the sum of those two months (Nov/2019 and Nov/2020). Other step size I don't see the problem. For the 2019 period the reports seem ok.
Created attachment 374021 [details] Test case for "Expense chart"
Created attachment 374022 [details] try this patch to add tracer statements. Thank you Rafael; it would be really good to debug this together. I cannot reproduce this interesting bug. This requires modifying category-barchart.scm to add tracer statements, and attach the https://wiki.gnucash.org/wiki/Tracefile to this bug. These tracer statements cannot modify your data.
Created attachment 374023 [details] try this more comprehensive patch to add tracer statements.
Created attachment 374024 [details] trace of Expense Chart Report
Created attachment 374025 [details] Trace of "Expense Chart Report"
Created attachment 374026 [details] add more tracer statements Curiouser and curiouser. Add this new tracer patch#2. I suspect a bug in gnc_mktime or gnc_localtime. What's the locale?
Also if Linux $guile --version
PS your comment #4 trace is better. I'm not interested in these --debug logs.
(In reply to Christopher Lam from comment #7) > Also if Linux > > $guile --version ----------------------------------------------------------- $guile --version guile (GNU Guile) 2.2.7 Copyright (C) 2020 Free Software Foundation, Inc. License LGPLv3+: GNU LGPL 3 or later <http://gnu.org/licenses/lgpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ------------------------------------------------------------ I have two version of guile and I tried with both didn't work. ---------------------------------------------------------------------- $guile2 --version guile (GNU Guile) 2.0.14 Copyright (C) 2016 Free Software Foundation, Inc. License LGPLv3+: GNU LGPL 3 or later <http://gnu.org/licenses/lgpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. -------------------------------------------------------------------------
(In reply to Christopher Lam from comment #6) > Created attachment 374026 [details] > add more tracer statements > > Curiouser and curiouser. Add this new tracer patch#2. > > I suspect a bug in gnc_mktime or gnc_localtime. > > What's the locale? I run Gnucash with the command $LC_ALL=C gnucash to eliminate this variable. I also tried with all date formats. But I use pt_BR. ---My default locale ----------------------------- $locale LANG=pt_BR.UTF-8 LC_CTYPE="pt_BR.UTF-8" LC_NUMERIC="pt_BR.UTF-8" LC_TIME="pt_BR.UTF-8" LC_COLLATE="pt_BR.UTF-8" LC_MONETARY="pt_BR.UTF-8" LC_MESSAGES="pt_BR.UTF-8" LC_PAPER="pt_BR.UTF-8" LC_NAME="pt_BR.UTF-8" LC_ADDRESS="pt_BR.UTF-8" LC_TELEPHONE="pt_BR.UTF-8" LC_MEASUREMENT="pt_BR.UTF-8" LC_IDENTIFICATION="pt_BR.UTF-8" LC_ALL= ------------------------------
Created attachment 374027 [details] worse behavior with 'day' as step size I figure out that I can´t select November 1, 2020 in "Start Date" or "End Date". Any report before November 1, 2020 have strange behavior.
Created attachment 374028 [details] Trace with patch date-utilitties If the period of the report have the date November 1, 2002, its get crazy. If the step size is 'Day', the last patch frozen the gnucash.
https://community.powerbi.com/t5/Issues/Date-filter-issue-between-October-31-2020-and-November-1-2020/idi-p/1434271
Last test TZ=GMT-3 -- what happens when you select report-date 1-Nov-2020?
(In reply to Rafael Casali from comment #11) > Created attachment 374027 [details] > worse behavior with 'day' as step size > > I figure out that I can´t select November 1, 2020 in "Start Date" or "End > Date". Any report before November 1, 2020 have strange behavior. Was that by any chance the daylight savings time transition date last year? We've had sporadic reports of problems with that in Brazil--only in Brazil--for years.
Ok I've reproduced this bug via TZ=America/Sao_Paulo ./gnucash This causes the exact same error seen. This is caused by code which attempts to create date for: "01-Jan-2020 00:00", add 10 months, resulting in "01-Nov-2020 00:00" which apparently does not exist in Sao Paulo, and results in "Wed Dec 31 21:00:00 1969" a bogus answer. The latter date's is then sent through an algorithm to try avoid date-slips(*), finding its month December != November as expected, resulting in the previous day being sought "30-Oct-2020 00:00" whose month October != November, decrementing until 30-Nov-2019 0:00 to finally reach an answer whereby month is November as expected. I think incdate-months *can* be rewritten. (*) see Bug 639049
Or, jralls may offer a suitable boost version of incdate-months?
(In reply to John Ralls from comment #15) > (In reply to Rafael Casali from comment #11) > > Created attachment 374027 [details] > > worse behavior with 'day' as step size > > > > I figure out that I can´t select November 1, 2020 in "Start Date" or "End > > Date". Any report before November 1, 2020 have strange behavior. > > Was that by any chance the daylight savings time transition date last year? > We've had sporadic reports of problems with that in Brazil--only in > Brazil--for years. In fact, it was scheduled for that date, but it was canceled and Brazil no longer has daylight saving time. But with the previous daylight saving time, I never had a problem.
(In reply to Christopher Lam from comment #17) > Or, jralls may offer a suitable boost version of incdate-months? Christopher, Forget about this bug, it's a time zone data bug. It is not a gnucash problem. In other programs, I am also unable to select this specific day. I know that other countries also have summer time, do they not have this problem? As I never had a problem before, I believe that the cancellation of the date was the problem. Thank you for your help.
Created attachment 374030 [details] Report error with other LC_ALL I thought that changing only the timezone the report would be ok, but it only runs with LC_ALL = C. Selecting another selection (I tried 'fr_FR.utf8' and 'en_CA.utf8' (attached image). It says that there is no data.
(In reply to Rafael Casali from comment #20) > Selecting another selection (I tried 'fr_FR.utf8' and 'en_CA.utf8' (attached > image). It says that there is no data. Because changing the locale also changes the default report currency and given there's no price USD->EUR or CAD the amounts become 0EUR or 0CAD. Change report currency back to USD and it'll be fine again (with TZ-based error). Jralls may be motivated to rewrite usign boost/date_time/gregorian or std::chrono or https://stackoverflow.com/a/31508468 hopefully these work in Sao Paulo...
(In reply to Christopher Lam from comment #21) > (In reply to Rafael Casali from comment #20) > > > Selecting another selection (I tried 'fr_FR.utf8' and 'en_CA.utf8' (attached > > image). It says that there is no data. > > Because changing the locale also changes the default report currency and > given there's no price USD->EUR or CAD the amounts become 0EUR or 0CAD. > Change report currency back to USD and it'll be fine again (with TZ-based > error). > > Jralls may be motivated to rewrite usign boost/date_time/gregorian or > std::chrono or https://stackoverflow.com/a/31508468 hopefully these work in > Sao Paulo... Thanks again. I change my timezone to America/Argentina/Buenos_Aires e every thing works fine. Tanks a lot.
Created attachment 374031 [details] tracefile boost doesn't like Sao Paulo either; see snippet Tracefile. I've disabled most exception handlers in gnc-date.cpp and gnc-datetime.cpp. Here's a small patch to make this TZ bug more explicit in the tracefile: modified libgnucash/engine/gnc-date.cpp @@ -238,6 +238,11 @@ gnc_mktime (struct tm* time) } catch(std::invalid_argument&) { + char buffer [80]; + strftime (buffer, 80, "%Y-%m-%d %H:%M:%S %Z", time); + printf ("time is not valid: "); + puts (buffer); + printf ("\n"); return 0; } }
(In reply to Rafael Casali from comment #20) > Created attachment 374030 [details] > Report error with other LC_ALL > > I thought that changing only the timezone the report would be ok, but it > only runs with > LC_ALL = C. > Selecting another selection (I tried 'fr_FR.utf8' and 'en_CA.utf8' (attached > image). It says that there is no data. Not really germane to the bug, but what happens if you leave LC_ALL unset?
(In reply to John Ralls from comment #24) > (In reply to Rafael Casali from comment #20) > > Created attachment 374030 [details] > > Report error with other LC_ALL > > > > I thought that changing only the timezone the report would be ok, but it > > only runs with > > LC_ALL = C. > > Selecting another selection (I tried 'fr_FR.utf8' and 'en_CA.utf8' (attached > > image). It says that there is no data. > > Not really germane to the bug, but what happens if you leave LC_ALL unset? Christopher explained above, that setting LC_ALL changes the currency too and there were no expenses in the local currency and he had not registered the conversion of dollar to the currency corresponding to LC_ALL. Changing the timezone to another equivalent location (without daylight saving time canceled last year) works normally.
America/Sao_Paolo has no DST transitions, it's just UTC: zdump -v America/Sao_Paolo America/Sao_Paolo Fri Dec 13 20:45:52 1901 UTC = Fri Dec 13 20:45:52 1901 UTC isdst=0 America/Sao_Paolo Sat Dec 14 20:45:52 1901 UTC = Sat Dec 14 20:45:52 1901 UTC isdst=0 America/Sao_Paolo Mon Jan 18 03:14:07 2038 UTC = Mon Jan 18 03:14:07 2038 UTC isdst=0 America/Sao_Paolo Tue Jan 19 03:14:07 2038 UTC = Tue Jan 19 03:14:07 2038 UTC isdst=0 The other Brazil TZs appear to have dropped DST between 1994 (West) and 2019 (East). Rafael, what TimeZone are you in? That's the US NIST timezone database, not anyone's date implementation. There's no reason that 2020-Nov-01 00:00:00 should be invalid so I wonder if there's an issue with the struct tm. Can you crash it again and tell gdb f 8 p *tm ? The line numbers in your debug trace don't match up to current maint. What did you change? You don't need to disable catches to debug C++ exceptions, gdb has a special `catch` breakpoint for that.
(In reply to Rafael Casali from comment #25) > Christopher explained above, that setting LC_ALL changes the currency too > and there were no expenses in the local currency and he had not registered > the conversion of dollar to the currency corresponding to LC_ALL. Right, but I wondered about your using the C locale... I guess that's down to you using the USD for your base currency instead of BRL. > > Changing the timezone to another equivalent location (without daylight > saving time canceled last year) works normally. What TZ is that, to go with my question about which one do you actually use? For that TZ, what does ls -l /usr/share/zoneinfo/xxx/yyy (where xxx/yyy are the timezone name) say? Does it predate the cancellation of DST for 2020?
(In reply to John Ralls from comment #27) > (In reply to Rafael Casali from comment #25) > > > Christopher explained above, that setting LC_ALL changes the currency too > > and there were no expenses in the local currency and he had not registered > > the conversion of dollar to the currency corresponding to LC_ALL. > > Right, but I wondered about your using the C locale... I guess that's down > to you using the USD for your base currency instead of BRL. > > > > > Changing the timezone to another equivalent location (without daylight > > saving time canceled last year) works normally. > > What TZ is that, to go with my question about which one do you actually use? > > For that TZ, what does ls -l /usr/share/zoneinfo/xxx/yyy (where xxx/yyy are > the timezone name) say? Does it predate the cancellation of DST for 2020? I was using America / Sao_Paulo, but this problem also affects other programs like LibreOffice, so I ended up moving to a place that has no daylight saving time and is working perfectly. Following your command $ zdump -v America / Sao_Paulo | tail -6 America / Sao_Paulo Sun Nov 4 02:59:59 2018 UT = Sat Nov 3 23:59:59 2018 -03 isdst = 0 gmtoff = -10800 America / Sao_Paulo Sun Nov 4 03:00:00 2018 UT = Sun Nov 4 01:00:00 2018 -02 isdst = 1 gmtoff = -7200 America / Sao_Paulo Sun Feb 17 01:59:59 2019 UT = Sat Feb 16 23:59:59 2019 -02 isdst = 1 gmtoff = -7200 America / Sao_Paulo Sun Feb 17 02:00:00 2019 UT = Sat Feb 16 23:00:00 2019 -03 isdst = 0 gmtoff = -10800 America / Sao_Paulo 9223372036854689407 = NULL America / Sao_Paulo 9223372036854775807 = NULL I use the system's standard timezone database (fedora 33 use https://www.iana.org/time-zones) and two lines appear quite suspicious. But as I said, I am using a timezone that does not have daylight saving time. But the strange thing is that I never had a problem with daylight saving time before, it seems to me that this cancellation is being the problem. As for daylight saving time in 2020, it was foreseen in Brazil to know daylight saving time on November 1st, but the government ended with daylight saving time before that date. I apologize, but I am not so skilled with gdb to test your suggestion, I would have to take the steps a little more detailed. In addition, the program does not crash.
> I apologize, but I am not so skilled with gdb to test your suggestion, I would have to take the steps a little more detailed. In addition, the program does not crash. No worries, the gdb instructions are for Chris. >I use the system's standard timezone database (fedora 33 use https://www.iana.org/time-zones) and two lines appear quite suspicious. But as I said, I am using a timezone that does not have daylight saving time. But the strange thing is that I never had a problem with daylight saving time before, it seems to me that this cancellation is being the problem. The last two lines are markers in the 64-bit part of the tzinfo file. They're just there as markers to make it easy to iterate over the file. Those timestamps are for after the heat-death of the universe. It looks like there are two problems: First, the timezone reader doesn't recognize the difference between jurisdictions terminating DST and the TZ database not generating transitions because the rule hasn't changed. Second, if a date-time is invalid because it's in the middle of a begin DST gap then it should add an hour and try again; if it's ambiguous because of an end DST redundancy it should pick one. It shouldn't return 0 (that's the December 1969 timestamp Chris noticed earlier).
Created attachment 374033 [details] TZ=America/Argentina/Buenos_Aries have the problem in 15 Oct 2020. I changed my time zone to America / Argentina / Buenos_Aires and now I have a problem around October 18, 2020 (but that day changes with the year). But the problem only happens when using the 'Day' step size. Other time zones also happen, such as America / Guayaquil. I have no problems when I use TZ = GMT-3, TZ = America / Toronto, for example.
Created attachment 374034 [details] candidate patch to gnc-date (In reply to John Ralls from comment #26) > The line numbers in your debug trace don't match up to current maint. What > did you change? You don't need to disable catches to debug C++ exceptions, > gdb has a special `catch` breakpoint for that. The debug trace match 4.4 with most gnc-date and gnc-datetime exception handlers removed. > That's the US NIST timezone database, not anyone's date implementation. > There's no reason that 2020-Nov-01 00:00:00 should be invalid so I wonder if > there's an issue with the struct tm. Can you crash it again and tell gdb > f 8 > p *tm > ? (gdb) f 8 #8 0x00007ffff69afdd3 in LDT_from_struct_tm (tm=...) at /home/chris/sources/gnucash/maint-4.4/libgnucash/engine/gnc-datetime.cpp:184 184 LDT ldt(tdate, tdur, tz, LDTBase::EXCEPTION_ON_ERROR); (gdb) p *tm Cannot resolve function operator* to any overloaded instance (gdb) p tm $1 = {tm_sec = 0, tm_min = 0, tm_hour = 0, tm_mday = 1, tm_mon = 10, tm_year = 120, tm_wday = 3, tm_yday = 0, tm_isdst = 1, tm_gmtoff = -7200, tm_zone = 0x555556432760 "Unset"} (gdb) > (In reply to John Ralls from comment #29) > It looks like there are two problems: First, the timezone reader doesn't > recognize the difference between jurisdictions terminating DST and the TZ > database not generating transitions because the rule hasn't changed. Second, > if a date-time is invalid because it's in the middle of a begin DST gap then > it should add an hour and try again; if it's ambiguous because of an end DST > redundancy it should pick one. It shouldn't return 0 (that's the December > 1969 timestamp Chris noticed earlier). An idea like the attached patch? (I'll graciously defer to your datetime skills).
> An idea like the attached patch? (I'll graciously defer to your datetime skills). Eh, no. That doesn't actually do anything. Also we should avoid using "wrap" at the beginning of function names: Swig does that in its implementation files (you might have noticed it in your stack trace) and we'll confuse ourselves if we do it to. The ambiguous result is already handled (though maybe not the best way) so we just need to do something similar at https://github.com/Gnucash/gnucash/blob/1756f76079221f77904ccb9c9a504f9ebca2eb0e/libgnucash/engine/gnc-datetime.cpp#L195. The string constructor needs similar treatment.
(In reply to Rafael Casali from comment #30) > Created attachment 374033 [details] > TZ=America/Argentina/Buenos_Aries have the problem in 15 Oct 2020. > > I changed my time zone to America / Argentina / Buenos_Aires and now I have > a problem around October 18, 2020 (but that day changes with the year). But > the problem only happens when using the 'Day' step size. Other time zones > also happen, such as America / Guayaquil. > > I have no problems when I use TZ = GMT-3, TZ = America / Toronto, for > example. I think that's because of the projection problem I mentioned in comment 29. It doesn't happen in the northern hemisphere timezones because they mostly do the transition at 2:00 AM instead of midnight. The problem will happen only when a point for the chart happens to land in not-an-hour caused by starting DST, for example midnight on 1 November 2020 in Sao Paolo.
Jralls fixed in maint, 4.5 due in couple weeks. Or check tomorrow 2021-03-22 flatpak https://code.gnucash.org/builds/flatpak/maint/
