freetds AT lists.ibiblio.org
Subject: FreeTDS Development Group
List archive
- From: "James K. Lowden" <jklowden AT freetds.org>
- To: freetds AT lists.ibiblio.org
- Subject: Re: [freetds] iconv madness
- Date: Sun, 16 Feb 2014 15:41:34 -0500
On Sat, 15 Feb 2014 17:32:53 +0000
Frediano Ziglio <freddy77 AT gmail.com> wrote:
> I'm going probably to refactor a bit the iconv stuff.
Hi Freddy,
Since the iconv mess is mostly my fault, I have a few suggestions in
hindsight.
There are only two character-set conversion requirements:
1. On the wire, to convert TDS 7+ SQL text to/from UCS-2.
2. In the client, to convert character data to/from the client's
character set for bound buffers (whether dbbind, dbrpcbind or
bcp_bind).
The policy we adopted was to convert all character data "at the wire".
The read/write functions performed the conversion, and all character
data were kept in the client's preferred encoding.
That policy was a mistake. Better not to touch the data unless and
until it's necessary, because that simplifies some things that should
be simple, like binary bindings. Only when the encoding must be
changed -- for the protocol, or for the client -- should the conversion
take place.
It's too bad the iconv standard API doesn't include I/O functions,
because for our purposes functions modelled on printf/scanf would be
easier to use. I suppose that's what tds_convert_stream is for.
Apart from protocol requirements, character-set conversion is entirely
a client library function. The user expects characters to be
faithfully rendered on the screen, no matter how they're stored in
database or transmitted on the wire. In effect, all
*client-provided* character buffers implicitly use the client's
encoding. That means dbdata() returns the unconverted buffer and
dbbind() enlists conversion (if need be).
There is an exception: bcp. Sybase's bcp utility supports a -j option
that defines the file's encoding, which may be different from the
encoding in the user's environment. The user may well have a file --
or require a file -- in a "foreign" encoding, say a UCS-2 file
destined for a Windows machine but created on a utf-8 Linux box.
There is no reason freebcp can't support that, but the current design
makes it more difficult than it should be.
It would also be nice if, as a byproduct of your work, freebcp could
read from a pipe. I created the misbegotten requirement that the
stream be seekable to avoid memory pressure. With today's larger
memories, freebcp should just realloc as needed to hold a column
buffer.
HTH & regards,
--jkl
-
[freetds] iconv madness,
Frediano Ziglio, 02/15/2014
-
Re: [freetds] iconv madness,
James K. Lowden, 02/16/2014
- Re: [freetds] iconv madness, Frediano Ziglio, 02/17/2014
-
Re: [freetds] iconv madness,
James K. Lowden, 02/16/2014
Archive powered by MHonArc 2.6.24.