Skip to Content.
Sympa Menu

freetds - RE: Unicode - UCS2, UTF8 present and future...

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: "Lowden, James K" <LowdenJK AT bernstein.com>
  • To: "'TDS Development Group'" <freetds AT franklin.metalab.unc.edu>
  • Subject: RE: Unicode - UCS2, UTF8 present and future...
  • Date: Mon, 19 Aug 2002 17:20:19 -0400


> From: ZIGLIO Frediano [mailto:Frediano.Ziglio AT vodafoneomnitel.it]
> Sent: August 19, 2002 10:39 AM
>
> What happen if a unconvertible character is encountered? I
> think iconv
> stop at that character and C conversion return truncated bytes..
> some garbage data

iconv(3) will stop early if its input is invalid or it has no room in its
output buffer. It is possible -- very possible -- for its input to include
a character that is valid UCS-2, say, but for which an identical character
does not exist in the output codeset. In that event, it should substitute
something and keep going. What it substitutes is left to the
implementation, but it shouldn't stop and it shouldn't return an error.

A question mark is often used. A perhaps too-obvious example is in your
favorite web-browser, when reading a page composed by some ascii-challenged
yo-yo, you see something like "It?s easy" because what ought to have been a
standard ascii 39 (') was a proprietary codeset-defined 145.

(AFAICT, we're in new territory with client-side character set conversion.
Sybase's International Developer's Guide says "At the present time in Sybase
systems, automatic character set conversion occurs only on the server."
Microsoft's NT-based client platforms have built-in UCS-2 support.)

> - conversion from ucs2 to utf8 have a small problem. We
> assume conversion
> convert from a character to a byte but a UCS2 character can produce
> up to 3 utf8 bytes... (from2 byte to 3). So many time
> string are just trimmed.

If the client provides a buffer that's too small, that's not always an
error. It depends on the API function. If you call dbconvert() with a
too-small output buffer, you should see an error message and be returned a
-1. But if you use dbbind() to bind a too-small buffer to a varchar column,
I think dbnextrow() will succeed. Server-side truncations are silent.

When FreeTDS has to allocate a buffer as part of its iconv work, we could do
something simple, like always allocate 4x the size of the input buffer. Or
we could maintain a lookup table of conversion ratios, but that would be a
headache to keep up.

Probably the the most robust answer is to write tds_iconv(), such that it
automatically allocates a larger buffer if iconv(3) returns a truncation
error (E2BIG). If tds_iconv kept track of total bytes in and out during the
lifetime of the program (in local static variables, say), it would quickly
arrive at the right conversion ratio.

> Many unix do not use ucs2 but utf8 so fix for this
> would be useful more than ucs2 handling
>
> I think fix for this can't be in 0.6 (too place to
> correct...) but surely
> will be an improvement in next version...

Agreed on both points.

--jkl


The information contained in this transmission may contain privileged and
confidential information and is intended only for the use of the person(s)
named above. If you are not the intended recipient, or an employee or agent
responsible for delivering this message to the intended recipient, any
review, dissemination, distribution or duplication of this communication is
strictly prohibited. If you are not the intended recipient, please contact
the sender immediately by reply e-mail and destroy all copies of the
original message. Please note that for certain accounts we do not accept
orders and/or instructions by e-mail, and for those accounts we will not be
responsible for carrying out such orders and/or instructions. Kindly refrain
from sending orders or instructions by e-mail unless you have confirmed that
we accept such communications for your account. Please also note that to
satisfy regulatory requirements we review the outgoing and incoming e-mail
correspondence of staff members serving certain functions.






Archive powered by MHonArc 2.6.24.

Top of Page