FreeTDS Development Group

["Lowden, James K" <LowdenJK AT bernstein.com>]

> From: Aliet Santiesteban Sifontes [mailto:alietss AT yahoo.com]
> Sent: March 17, 2003 3:54 AM
> 
> This is a really problem for me, any time a user type
> a word with acent everything ""PUM"", and I have
> tested all the charsets, this is the output of my
> locale in RedHat 8.0...
> LANG=es_ES.UTF-8
> LC_CTYPE="es_ES.UTF-8"
> LC_NUMERIC="es_ES.UTF-8"
> LC_TIME="es_ES.UTF-8"
> LC_COLLATE="es_ES.UTF-8"
> LC_MONETARY="es_ES.UTF-8"
> LC_MESSAGES="es_ES.UTF-8"
> LC_PAPER="es_ES.UTF-8"
> LC_NAME="es_ES.UTF-8"
> LC_ADDRESS="es_ES.UTF-8"
> LC_TELEPHONE="es_ES.UTF-8"
> LC_MEASUREMENT="es_ES.UTF-8"
> LC_IDENTIFICATION="es_ES.UTF-8"
> LC_ALL=
> 
> Is there any solution for RedHat 8.0 about this, or
> any way that freetds ignore this words or something...

Aliet, 

A completely fair question.  How hard are you willing to work, and which
direction do you want to go in?  Because, like it or not, you're at the
cutting edge.  :/  I realized on the way to dinner last night that Red Hat
8.0 poses a real problem to database users generally and to FreeTDS in
particular.  

Background:  You're using UTF-8, the 8-bit version of the Unicode
Transformation Format.  For more information and links to still more
infomation, see Appendix B of the FreeTDs User Guide.  UTF-8 is the standard
RH8 setup, and it's different from ASCII or ISO-8859-1 (a/k/a Latin-1, inter
alia) because the number of bytes per character varies.  For example,
suppose you want the letter 'n' with a tilde '~' above it.  ISO-8859-1 uses
0xF1 for that letter, but UTF-8 needs two bytes.  The conversion formula is
available here: http://czyborra.com/utf/#UTF-8.  FreeTDS is unaccustomed to
distinguising between the character count and the byte count of a string,
and there may be -- almost certainly are -- changes required to make it work
correctly with UTF-8.  

Data from Microsoft servers arrives in 16-bit Unicode, specifically UCS-2,
which we convert to the character set indicated in freetds.conf.  That
character set should reflect the client application's display capabilities,
most especially the set of installed fonts.  I have to believe your fonts
etc. are all coordinated around UTF-8, but that you can also display
ISO-8859-1 data with a few tweaks (one problem being, knowing what those
tweaks are).  

Broadly speaking, you have two choices apart from giving up: 

1.  Set up your system to handle ISO-8859-1 data (which is how prior Red Hat
systems were), and tell FreeTDS to use that character set.  

2.  Tell FreeTDS to use UTF-8, and bear with us while we find the bugs you
expose and endure.  To do that, set "client charset = UTF-8" in
freetds.conf, and see what happens.  

Choice #2 is the Right Thing from our perspective, because making FreeTDS
agnostic about the client character set is an immediate goal of ours.  But
it may mean more pain for you, depending on how hard #1 is, and how hard #2
is.  

Note I have no knowledge of how PHP will react to receiving UTF-8 data from
FreeTDS.  It might be fine, but I wouldn't be surprised if somewhere there's
an assumption or two that the number of characters equals the number of
bytes.  

I hope I've helped you understand the situation, Aliet.  Please let us know
what you decide. 

Regards, 

--jkl



The information contained in this transmission may contain privileged and
confidential information and is intended only for the use of the person(s)
named above.  If you are not the intended recipient, or an employee or agent
responsible for delivering this message to the intended recipient, any
review, dissemination, distribution or duplication of this communication is
strictly prohibited. If you are not the intended recipient, please contact
the sender immediately by reply e-mail and destroy all copies of the
original message. Please note that we do not accept account orders and/or
instructions by e-mail, and therefore will not be responsible for carrying
out such orders and/or instructions.