(In reply to Jannick from comment #0)
> Great to have an enhanced csv importer in GC3.1 (probably earlier since
> 3.0). Many thanks for the implementation!
> 

I'm pleased you like the improvements :)

> 
> Remarks wrt the transaction csv importer:
> 
> - the buttons 'save settings' and 'delete settings' are not responsive after
> ... unless I am missing something here.

I'm not sure I understand your issue exactly. "are not responsive after ..." after what ?

The save button will save settings with the name entered in the text field next to it.
There are two default settings presets: "No settings" and "Gnucash default export format". You can't overwrite or delete these two. So unless you enter a different preset name first, the two buttons are disabled indeed. Is that what you are experiencing ?

> - account drop down: a scroll bar would be helpful

I see your point though I'm not sure that can be added easily in this type of widget (combotext). The alternative (and more common approach with a combotext widget) would allow the user to start typing part of an account name and a filtered list would be presented. However I see this is missing as well.
> 
> Separately: Would it be possible to filter out hidden accounts in account
> drop downs (e.g., register and - here - txn csv importer). Rationale: I am
> working with huge generic charts of accounts of which I unhide an account
> once it is needed.
> (Please let me know if you want to have this in a separate bug report.)   
> 
Yes, please file one suggestion/bug per report. Otherwise it makes tracking issues needlessly complicated.

(In reply to Geert Janssens from comment #1)

> > 
> > Remarks wrt the transaction csv importer:
> > 
> > - the buttons 'save settings' and 'delete settings' are not responsive after
> > ... unless I am missing something here.
> 
> I'm not sure I understand your issue exactly. "are not responsive after ..."
> after what ?

Sorry, this was incomplete apparently. 'after setting up the import configuration' I wanted to write.
 
> The save button will save settings with the name entered in the text field
> next to it.
> There are two default settings presets: "No settings" and "Gnucash default
> export format". You can't overwrite or delete these two. So unless you enter
> a different preset name first, the two buttons are disabled indeed. Is that
> what you are experiencing ?

Ahh - OK. After entering a config name I am able to successfully save and I am rewarded with a window confirming the save procedure.

BUT: When loading the configuration for another import everything but the column -> field association is restored. Is this how it should work and I am supposed to run through the columns again to associate the GC import fields? Ideally the config saves this assocation as well.

> > - account drop down: a scroll bar would be helpful
> 
> I see your point though I'm not sure that can be added easily in this type
> of widget (combotext). The alternative (and more common approach with a
> combotext widget) would allow the user to start typing part of an account
> name and a filtered list would be presented. However I see this is missing
> as well.

That'll be great. What would be helpful as well is that when entering a string the account list is restricted to accounts matching the string _anywhere_ in the account path, not only for the highest undefined account level. This would help immediately select the account when entering the account name (which is the leaf of the account tree / end of the account path).

> > Separately: Would it be possible to filter out hidden accounts in account
> > drop downs (e.g., register and - here - txn csv importer). Rationale: I am
> > working with huge generic charts of accounts of which I unhide an account
> > once it is needed.
> > (Please let me know if you want to have this in a separate bug report.)   
> > 
> Yes, please file one suggestion/bug per report. Otherwise it makes tracking
> issues needlessly complicated.

Done, see bug 796532 (just for reference).

(In reply to Geert Janssens from comment #1)
> The save button will save settings with the name entered in the text field
> next to it.
> There are two default settings presets: "No settings" and "Gnucash default
> export format". You can't overwrite or delete these two. So unless you enter
> a different preset name first, the two buttons are disabled indeed. Is that
> what you are experiencing ?

Can we put this in the tooltip and later in the docu?

(In reply to Jannick from comment #2)
> BUT: When loading the configuration for another import everything but the
> column -> field association is restored. Is this how it should work and I am
> supposed to run through the columns again to associate the GC import fields?
> Ideally the config saves this assocation as well.

These should be saved and restored correctly. I'm not sure why this isn't working for you. Can you open the gnucash meta file associated with your book and check whether the colum settings are actually being saved ?

The gnucash meta file can be found in
c:\Users\<username>\AppData\Roaming\gnucash\books\<book>.gcm

<book> is the name of your book, including the .gnucash extension.

This file will have several sections. Search for the section named
"Import csv,transactions - <save name>" with <save name> the name you gave to your saved csv settings.

That section should have a line "ColumnTypes" listing the columns you had set up when saving the settings. An example for my book:
ColumnTypes=Date;Account;Num;Description;Transfer Account;None;None;Deposit;None;

Do you have such a line ?

(In reply to Geert Janssens from comment #4)

> That section should have a line "ColumnTypes" listing the columns you had
> set up when saving the settings. An example for my book:
> ColumnTypes=Date;Account;Num;Description;Transfer
> Account;None;None;Deposit;None;
> 
> Do you have such a line ?

Actually I do. For the sake of completeness here the entire relevant meta-file section:

[Import csv,transaction - XXXX-Import]
Name=XXXX-Import
SkipStartLines=1
SkipEndLines=0
SkipAltLines=false
CsvFormat=true
Separators=;
DateFormat=1
CurrencyFormat=0
Encoding=UTF-8
MultiSplit=false
BaseAccount=Passiva:Verbindlichkeiten Sonstige:1730 Kreditkartenabrechnung
ColumnTypes=Notes;Date;None;Description;None;None;Withdrawal;

So the 'ColumnTypes' line does show up, whereas still only the first field (Notes) is allocated, the remaining field allocations (Date, Description, Withdrawal) are ignored in the dialog.

Potentially the sense check of the field types might kick in here (cf. https://github.com/Gnucash/gnucash/blob/32e799484313dc3390f97fca38f6906e7c683048/gnucash/import-export/csv-imp/gnc-imp-settings-csv-tx.cpp#L180:184), which is why I am attaching a (redacted) snap shot of the dialog window. Hope this is helpful.

Created attachment 372755 [details]
csv import dialog window: sample data causing issue

Hmm, can you attach the trace file that's generated when you load the saved settings in the csv importer ?

On windows you'll find a trace file for each time gnucash ran as described here:
http://wiki.gnucash.org/wiki/Tracefile

To get the proper trace file
1. quit gnucash
2. start gnucash
3. Start the csv importer and continue until you have selected the saved preset.
4. You can then cancel the import and quit gnucash
5. Now locate the trace files and pick the most recent one (sorted by date, not alphabetically) and post it here.

I suspect there's a sequence error in loading the settings such that the ColumnTypes are read before the Separators and/or Multisplit settings are, which could discard ColumnTypes: if parsing the file with a wrong separator setting results in fewer columns than you have ColumnTypes set, it would drop the ones that are too much. Your trace file may have more details though.

Hi Geert,

unfortunately, there was no entry regarding the csv import procedure.


BUT: Playing around a little bit with the import file, the bug disappears if from the original header line (which is to be ignored by config definition, but probably read for sense checking purposes)

Konto-/Kartennummer;Buchungsdatum;Kaufdatum;Umsatz/Ort;Fremdwährung;Kurs zu EUR;Betrag in EUR

the nasty 'ä' (= German a-umlaut) in 'Fremdwährung' is simply removed. 

Thus:
1. Are lines to be ignored for import read for sense checking purposes? (I would expect: No.)
2. Is GC confused if non-ASCII (?) chars are used in the input file? (Note that the a-umlaut is not displayed in the dialog view either (cf. the jpg attached to comment #6).)

Does this help somehow?

Good catch. That suggests some character set related issue.

What is the character set encoding of your original csv file ? You have selected UTF-8 in the csv importer but the error suggests your file is not UTF-8.

Try to select different Encodings in the importer (probably one of the Western ones) until you see the proper column headings appear.

Assuming you find a correct one, update your saved preset and see if it now loads columns properly when you restart the csv import.

If you don't find any encoding that works, I would like to receive a sample csv file (can be anonymized) in the same encoding as your original file is. But be sure not to change the encoding while anonymizing or it wouldn't be a useful test any more.

If you do find a proper encoding, please report how gnucash behaves if restart the import with a preset that uses this encoding.

(In reply to Geert Janssens from comment #9)
> What is the character set encoding of your original csv file ? You have
> selected UTF-8 in the csv importer but the error suggests your file is not
> UTF-8.

Hmmm, that setting was surely not correct. 'file input.csv' gave me ISO-8859, while notepad++ claims (as displayed in its footer line) it is ANSI. Which to reliably choose I am not sure at all.

> Try to select different Encodings in the importer (probably one of the
> Western ones) until you see the proper column headings appear.
> 
> Assuming you find a correct one, update your saved preset and see if it now
> loads columns properly when you restart the csv import.

AFTER a weird tweak (see below), the following encodings work:
1 - ISO-8859-1 (Western)
2 - Windows-1252 = 'Encoding=windows-1252'
3 - CP1252 (which is shown as 'Locale' at the bottom of the encoding drop down; I am on German Win10). Pls see side remark below.
4 - ASCII (English) = 'Encoding=ANSI_X3.4-1968'

For the sake of completeness the snip of the config meta file again:
~~~
[Import csv,transaction - ISO-8859-1]
Name=ISO-8859-1
SkipStartLines=1
SkipEndLines=0
SkipAltLines=false
CsvFormat=true
Separators=;
DateFormat=1
CurrencyFormat=2
Encoding=ISO-8859-1
MultiSplit=false
BaseAccount=LIABILITIES:CreditCard
ColumnTypes=Notes;Date;None;Description;None;None;Withdrawal;

[Import csv,transaction - ASCII]
Name=ASCII
SkipStartLines=1
SkipEndLines=0
SkipAltLines=false
CsvFormat=true
Separators=;
DateFormat=1
CurrencyFormat=2
Encoding=ANSI_X3.4-1968
MultiSplit=false
BaseAccount=LIABILITIES:CreditCard
ColumnTypes=Notes;Date;None;Description;None;None;Withdrawal;

[Import csv,transaction - Windows-1252]
Name=Windows-1252
SkipStartLines=1
SkipEndLines=0
SkipAltLines=false
CsvFormat=true
Separators=;
DateFormat=1
CurrencyFormat=2
Encoding=windows-1252
MultiSplit=false
BaseAccount=LIABILITIES:CreditCard
ColumnTypes=Notes;Date;None;Description;None;None;Withdrawal;

[Import csv,transaction - Locale_CP1252]
Name=Locale_CP1252
SkipStartLines=1
SkipEndLines=0
SkipAltLines=false
CsvFormat=true
Separators=;
DateFormat=1
CurrencyFormat=2
Encoding=CP1252
MultiSplit=false
BaseAccount=LIABILITIES:CreditCard
ColumnTypes=Notes;Date;None;Description;None;None;Withdrawal;
~~~

> If you do find a proper encoding, please report how gnucash behaves if
> restart the import with a preset that uses this encoding.

Here the funny tweak needed for files both with or without non-ASCII chars:
1. Open csv txn import dialog
2. Select any import config of the 4 mentioned above -> issue persists, i.e. only the first field (Notes) is recognized, the others aren't.
3. Select any other of the remaining 3 import configs -> surprise: everything is just fine.


Side remark regarding the local encoding (here: 'Locale:CP1252'): 
Choosing 'Locale:CP1252' as encoding is properly saved to the meta-config file (checked after closing the GC file, since only then the meta-config file is updated). 
However, after switching to this import config from another one with a different encoding, the displayed encoding in the import dialog remains unchanged, i.e. it remains set to the encoding of the previously chosen import config definition.


Wish list: Would it be possible to write the import file name (basename plus extension) to the top of the dialog box (currently it is a simple '.')?


And yes: Happy to send you the original file via private email. I just noted that notepad++ claims it saved the version with the a-umlaut removed as UTF8 (before ASCII) while 'file' says ISO-8859 (before) and ASCII (after). ... confusing.

With a long delay I have finally found time to chase this particular bug and fix it. The datafile and csv file you sent me in private were very helpful for debugging this.

For the record it had nothing to do with the encoding, but with the separators. The code that restored the separators indirectly caused the saved column types to be reset (except for the first column).

That at least should be fixed now.

Note I'm keeping this bug report open to remind me to
* eventually improve the Account selection (by adding a way to type in account names)
* add a tooltip as requested by Frank in comment 3

These last few tidbits have now been completed. Will appear in gnucash 3.8.