Skip to content
GitLab
Open
Issue created by Administrator@rootContributor

When reading non-utf8 stdin, emit a more specific warning; for Python 3.7+, use stdin.reconfigure()

Created by: dannguyen

Sorry if this issue is closely related to this one that was closed last year: Change --encoding error text if tool receives input from stdin #898

The fix for that issue was to have csvkit mention PYTHONIOENCODING in the message for an encoding error:

Your file is not "utf-8" encoded. Please specify the correct encoding with the -e flag or with the PYTHONIOENCODING environment variable. Use the -v flag to see the complete error.

I've skimmed the relevant parts in the source code but haven't yet dug in too deep, so a couple of quick comments/questions:

1. Have the error message be more explicit when stdin is used?

Is it possible/non-trivial to adjust the warning message to say something specifically about stdin when stdin is the input, especially if the user has set the -e flag? I have to admit all this time when piping into a csvkit util and getting an encoding error, I interpreted the message with the -e flag or with the PYTHONIOENCODING environment variable to mean that I should use either -e or set PYTHONIOENCODING – i.e. if -e wasn't working, it was because I hadn't figured out the proper encoding (though I guess I could have interpreted Your file is not "utf-8" encoded to mean that csvkit wasn't seeing my -e flag at all)

Something like:

Your file is not "utf-8" encoded. Please specify the correct encoding with the -e flag or with the PYTHONIOENCODING environment variable. Use the -v flag to see the complete error.

Note: if you are reading input from stdin, the -e flag is ignored and you must set the PYTHONIOENCODING variable, e.g.

 $  cat mydata.csv | PYTHONIOENCODING='windows-1252' csvformat

Or:

 $  PYTHONIOENCODING='windows-1252' csvformat < mydata.csv

2. Automatically configure the encoding for stdin for Pythons 3.7+

I saw that Python 3.7 adds a new stdin method to set its encoding:

sys.stdin.reconfigure(encoding='windows-1252')

I know the 3.7 userbase is probably still a relative minority, but is it worth adding in conditional behavior to cli.py when six detects version > 3.7?