FreeTDS Development Group

["James K. Lowden" <jklowden AT speakeasy.org>]

Hi Brian, 

Brian Bruns wrote:
> I like UTF-8 on the input side alot.  I suppose I'll have to match it on
> the bind side too...my main concerns are:
> 
> 1) UTF-8 may be larger, applications that make assumptions that a
> varchar(255) can be no longer than 255 bytes will break.  I could say we
> just truncate to the set boundary or if it's unlimited, the application
> could core dump.  If so, app is broken as designed I suppose.

You're right, it's treacherous.    

Try this in isql on NT:

        select substring( N'world', 1, 3 )

You will see (from memory; I'm at home):

------
wor

(1 row(s) affected)

SQL Server 7 returned a column six bytes wide, 2 bytes/character. 
That's the right answer and the true one; the server and the client
can't predict when the first wide character will arrive, so they reserve
the space always.  

I don't know what FreeTDS does with that today, but I'd guess it would
indicate a 3-byte column.  

(It's a little maddening, if you spend much time in Query Analyzer
looking at system tables, whose "name" columns are now nvarchar(255),
meaning they take up 512 positions on the screen.  That's a lot of
whitespace, even on a 1600x1200 screen.  Especially considering there
aren't any *fonts* installed to display non-Latin data, in the unlikely
event any such thing were returned.)

> I think having a default of UTF-8 is perhaps not a bad idea after all but
> let's add a configuration setting to convert to local 8bit where
> applicable. (We have to call iconv anyway, it'll be pretty easy to just
> substitute greek or whatever for UTF-8 where needed).

Thought experiment:  You're in the bind business anyway.  How about
leaving the Ascii/UTF-8 choice to the bind code?  That is, if the client
binds varchar to the column, return ascii.  If he binds nvarchar to it,
return UTF-8?  

As you've pointed out before, the principle of least surprise should
apply, and we don't know what perl/python/php expect/want/do with
UTF-8.  I started looking into the perl side; I'll let you know what I
find out.  
Me, I can't wrap my head around BCP.  What does Japanese text in a bcp
data file look like on a Unix box?  Depends on who's doing the looking,
no doubt.  But since Unix files are a stream of bytes, the situation
smells suspiciously like our varchar "client conspiracy", only at the OS
level.  It's a puzzlement.  

> > What does Sybase support for Unicode?
> 
> Not sure, but I'm interested in finding out. 

I looked at their online documentation a few months ago.  IIRC it looked
like a layered product of some sort that seemed to entail some
new/different APIs.  I don't recall anything about UTF-8; I remember
only Unicode.  

I don't think we've seen one question ever on this list about Unicode
data on a Sybase system, which I take as zero interest.  Just one man's
opinion, of course.  

--jkl