This newCSV
parser is m17n savvy. The parser works in the Encoding of the IO or String object being read from or written to. Your data is never
transcoded (unless you ask Ruby to
transcode it for you) and will literally be parsed in the Encoding it is in. Thus CSV will return Arrays or Rows of Strings in the Encoding of your data. This is accomplished by
transcoding the parser itself into your Encoding.

Some transcoding must take place, of course, to accomplish this
multiencoding support. For example, :col_sep,:row_sep,
and :quote_char must be transcoded to match your data. Hopefully
this makes the entire process feel transparent, since CSV’s defaults should just magically work for you
data. However, you can set these values manually in the target Encoding to avoid the translation.

It’s also important to note that while all of CSV’s core parser is now Encoding agnostic, some features are not. For
example, the built-in converters will
try to transcode data to UTF-8 before making conversions. Again, you can
provide custom converters that are aware
of your Encodings to avoid this translation. It’s just too hard for me
to support native conversions in all of Ruby’s Encodings.

Anyway, the practical side of this is simple: make sure IO and String objects passed
into CSV have the proper Encoding set and everything should just work. CSV methods that allow you to openIO objects
(CSV::foreach(), CSV::open(), CSV::read(), and CSV::readlines()) do allow
you to specify the Encoding.

One minor exception comes when generating CSV into
a String with an Encoding that is not ASCII compatible. There’s
no existing data for CSV to use to prepare itself
and thus you will probably need to manually specify the desired Encoding for most of those cases. It will try to
guess using the fields in a row of output though, when using
CSV::generate_line() or Array#to_csv().

I try to point out any other Encoding issues
in the documentation of methods as they come up.

This has been tested to the best of my ability with all non-“dummy”
Encodings Ruby ships with. However,
it is brave new code and may have some
bugs. Please feel free to report any issues you find with it.