I was about to write a quick fix for this when I realized there's one additional use case that complicates it slightly.

In the past some users have expressed the need for some kind of an audit log, or a trail of all changes made to the data in the book. While the current transaction log is not a perfect fit, it does work more or less for that purpose. And in that role it would still make sense for the sqlite backend as well.

FWIW, even when using a mysql or postgresql database these transaction logs are written. But as there is no file path for the book, these are stored in the gnucash data directory (previously GNC_DOT_DIR and now GNC_DATA_HOME).

As this complicates matters, I prefer chasing more urgent bugs first.

I'll add the same information on the mailing list for people that were following this discussion.

Another use for log files in this situation may be when latest database file is corrupt or unusable say to a disk error, but the log files are still OK.
Then it would be useful to replay the transaction logs against a restored database. I vote for keeping the logs.

Absolutely, Geert. This is by no means urgent. Thanks.

I guess, in the long term, having a preference setting would satisfy both use cases.

My issue with log files is that they contain a bunch of records even when I don't make any changes to the book. And they're not very human-readable at that.

I really *wish* there was a reliable way of tracing the transactions since I was recently in a situation where, due to Dropbox sync issue, there was a conflict on the file and I had to pick one and manually recreate the transactions that have happened in between. 

@Chris, my original question in the mailing list was - is replaying the log file on top of an sql book reliable? Will it do what we expect? If the database file is corrupt, how do we know the log isn't? Which log to replay, etc. 
There's lots of questions on that road but answering them would provide an excellent way of preserving changes, just like it is currently done with the xml backend.

Mhm, I see. So, what I think happens is - every time a register is open, a blank dummy transaction (record) is created. Once the register is closed, this transaction is deleted.
The entry for the Delete operation goes into the log.

I'll observe the behaviour into more detail and perhaps use any information here to add it to the wiki at some point.

Hi Alen,

Sorry for the delay answering - I'm short of time at the moment.
I have seen Geert say on the mailing lists that the business functions (invoices etc) are not logged so that is 1 group of transactions that it is known will not be replayed.

I assume if you try to replay a corrupt log, there will be an error.
I have to assume that replaying logs will work (except for business functions) until someone logs a bug saying that it doesn't.

Which log to replay should be easy to answer: any log with a time stamp after the backup of the database which was restored. (I'm sure there is a better way to say that).

Note that if it is found there are problems replaying a log against an sql database, another option would be to restore the database, use GnuCash to save it as XML, then replay the log(s) and save it back as sql.

Hi, Chris,

Thanks for the tips. I guess one thing that would be beneficial with sqlite backend would be to create a copy of the db file, similar to what is done with an xml book.
Not sure if this is easily feasible from within GnuCash but certainly can be performed manually. 
Then the rest of the recovery process would work as you describe above.
I'll try to find a good place in the wiki to store this info.

Created an initial version at https://wiki.gnucash.org/wiki/Backup

Personally I have my doubts on the current log file as a reliable recovery mechanism even for xml files. It really only logs transactions. It won't log changes (new, rename, delete)to accounts, business objects (invoices, vendors, customers, tax tables,...), scheduled transactions, price database, securities,... Really nothing except transactions and their splits.

It's true that after an initial setup transactions and splits are the most common things to change so it can work to some extent. But I think it's important to understand the very limited scope of the log files.

As for replaying a log file on the database I wouldnet want to go there. All databases we support have a built-in of (database) transaction logging capability which would also be much more complete than just the simple log we manually keep for the xml database.

However how to configure such db logging is in the domain of the db administrator not gnucash.

Just noticed that my logs are pretty big even when not making many changes. 
This is due to the fact that there are records logged for my scheduled transactions (which are many) even when they are not modified.
This adds too many records into the logs, resulting in noise and making it more difficult to even see the real changes made.
That might be one more point on the negative side.
Having a switch to turn off this behaviour would be great because it is a bit annoying with the cloud sync services to always transfer additional megabytes of .log files which later get deleted anyway and are completely useless at that.

At home this is not a problem but when synchronizing to a mobile device while traveling, this eats away into the internet data allowance significantly. I'm using the gnucash book on my Android devices to generate a few reports using Python while away from a PC.

I'm not in favour of turning the .log files off unless we have an audit trail in place.  I think removing logging while it is known the TB isn't working fully would be irresponsible.

All of the sql db back-ends have their own versioning / backup / integrity systems.  However, db integrity isn't the same as accounting tx integrity or an audit trail.

I'm wondering what the real requirement here is?  It looks to me as though this started as (possible misunderstanding) "database integrity" and moved towards "audit trail".

The sensible response may be to not allow the replay if the replay won't work with the file format.  I can see problems with that.

===

My main point is that an accounting audit trail and a database audit trail aren't the same thing.  Which is it we want?

Hardware should be All not Windows

Component provides a head scratching choice, it definitely involves Backend stuff but can't be one of [Backend-XML | Backend-SQL] because it isn't about one or the other.

> I'm wondering what the real requirement here is?

To answer your question and clarify the original request, if it is not clear enough - I'm asking for (useless) .log files not to be created automatically.

They are fairly large (in my case) and are cluttering the folder in which they are located, which is particularly dangerous as that is the same folder where the book file is. With SQLite backend, replaying the log files seems useless, if not outright dangerous.
As noted in one of the messages, 600KB log files are created for just opening and closing a GnuCash book. That seems quite unnecessary.

The range of satisfactory solutions is relatively broad - add a configuration switch; do not generate log files for SQL backends; improve the disaster recovery so that the logs are actually useful; etc.

There are some valid points in this thread.  Some I agree with, others I do not.

I think .log files *are* necessary as they allow replay, it is a separate argument regarding where they are placed [1] [2]

[1] I have a backup and tidy strategy that means they don't bother me

[2] .log files are only useless until you need them.

I agree the backend / log file combination doesn't always work well.  If you think gnc+SQLite has a log problem try postgres or mysql.  But that isn't the issue as you can switch to xml at any time. <== that flexibility is a strength.

AlenS: has said something that chimes with me and why I am here today.  There is stuff in my .log that really shouldn't be there.

All:  I'm hunting to see if anyone else has found unnecessary crap in their .log file.  Otherwise I'll make my own bug.