Skip to Content.
Sympa Menu

freetds - Re: [freetds] FreeTDS + SQL Server 2012 + UTF-16LE

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: Marc Abramowitz <msabramo AT gmail.com>
  • To: FreeTDS Development Group <freetds AT lists.ibiblio.org>
  • Subject: Re: [freetds] FreeTDS + SQL Server 2012 + UTF-16LE
  • Date: Sat, 30 Aug 2014 11:16:37 -1000

Interesting. John, I wonder if you can write a little C program that
illustrates the bug? I think that the FreeTDS authors are most comfortable in
C, so this will probably illustrate the problem best for them and show beyond
a shadow of a doubt that there's nothing happening in pymssql.

-Marc
http://marc-abramowitz.com
Sent from my iPhone 4S


> On Aug 29, 2014, at 7:07 AM, John Anderson <sontek AT gmail.com> wrote:
>
>> On Fri, Aug 29, 2014 at 9:56 AM, Frediano Ziglio <freddy77 AT gmail.com>
>> wrote:
>>
>> 2014-08-29 17:50 GMT+01:00 John Anderson <sontek AT gmail.com>:
>>
>>> Hello! I'm wondering if you had any tips on how to work with characters
>>> larger than 0xFFFF? We currently have the problem where we try to
>>> insert/update a row in the database and it gets truncated at the
>> character
>>> and creates an invalid syntax:
>>>
>>> For example this character: \U0001f44d
>>>
>>> We've tracked this down to the fact that FreeTDS is trying to convert
>> this
>>> to UCS-2 rather than UTF16-LE, which is what SQL Server 2012 uses.
>>>
>>> For example if we say:
>>>
>>> INSERT INTO table(name) VALUES(N' Hello \U0001f44d')
>> How are you using this string?
> Sorry, this is the python representation of the string which is a "Thumbs
> Up", It's UTF-8 hex representation is 0xf09f918d and what needs to be
> stored in the database is UTF-16LE hex 0x3dd84ddc.
>
> This is the character:
>
> http://www.fileformat.info/info/unicode/char/1F44D/index.htm
>
> But it is any character above 0xFFFF will cause the issue.
>
> So the actually query ends up looking like N'Hello <display of a thumbs up>'
>
>
>
>>
>>> We get the error:
>>>
>>> Unclosed quotation mark after the character string 'Hello ' because it
>>> stops processing the rest of the query.
>>>
>>> Is there a freetds.conf or environment setting we can use to tell it to
>> use
>>> UTF-16LE instead of UCS2 so that we can accept these larger character
>> sets?
>>>
>>> As a temporary solution we've started sending the raw UTF-16LE bytes to
>>> FreeTDS but we prefer not having to do this.
>>>
>>> Thanks,
>>> John
>>
>> Frediano
>> _______________________________________________
>> FreeTDS mailing list
>> FreeTDS AT lists.ibiblio.org
>> http://lists.ibiblio.org/mailman/listinfo/freetds
> _______________________________________________
> FreeTDS mailing list
> FreeTDS AT lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/freetds




Archive powered by MHonArc 2.6.24.

Top of Page