Skip to Content.
Sympa Menu

freetds - Re: [freetds] iconv madness

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: "James K. Lowden" <jklowden AT freetds.org>
  • To: freetds AT lists.ibiblio.org
  • Subject: Re: [freetds] iconv madness
  • Date: Sun, 16 Feb 2014 15:41:34 -0500

On Sat, 15 Feb 2014 17:32:53 +0000
Frediano Ziglio <freddy77 AT gmail.com> wrote:

> I'm going probably to refactor a bit the iconv stuff.

Hi Freddy,

Since the iconv mess is mostly my fault, I have a few suggestions in
hindsight.

There are only two character-set conversion requirements:

1. On the wire, to convert TDS 7+ SQL text to/from UCS-2.

2. In the client, to convert character data to/from the client's
character set for bound buffers (whether dbbind, dbrpcbind or
bcp_bind).

The policy we adopted was to convert all character data "at the wire".
The read/write functions performed the conversion, and all character
data were kept in the client's preferred encoding.

That policy was a mistake. Better not to touch the data unless and
until it's necessary, because that simplifies some things that should
be simple, like binary bindings. Only when the encoding must be
changed -- for the protocol, or for the client -- should the conversion
take place.

It's too bad the iconv standard API doesn't include I/O functions,
because for our purposes functions modelled on printf/scanf would be
easier to use. I suppose that's what tds_convert_stream is for.

Apart from protocol requirements, character-set conversion is entirely
a client library function. The user expects characters to be
faithfully rendered on the screen, no matter how they're stored in
database or transmitted on the wire. In effect, all
*client-provided* character buffers implicitly use the client's
encoding. That means dbdata() returns the unconverted buffer and
dbbind() enlists conversion (if need be).

There is an exception: bcp. Sybase's bcp utility supports a -j option
that defines the file's encoding, which may be different from the
encoding in the user's environment. The user may well have a file --
or require a file -- in a "foreign" encoding, say a UCS-2 file
destined for a Windows machine but created on a utf-8 Linux box.
There is no reason freebcp can't support that, but the current design
makes it more difficult than it should be.

It would also be nice if, as a byproduct of your work, freebcp could
read from a pipe. I created the misbegotten requirement that the
stream be seekable to avoid memory pressure. With today's larger
memories, freebcp should just realloc as needed to hold a column
buffer.

HTH & regards,

--jkl




Archive powered by MHonArc 2.6.24.

Top of Page