Skip to Content.
Sympa Menu

freetds - Re: [freetds] Conversion emoji symbols to UTF-8

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: Velichko Yuriy <velichko.yuriy AT gmail.com>
  • To: FreeTDS Development Group <freetds AT lists.ibiblio.org>
  • Subject: Re: [freetds] Conversion emoji symbols to UTF-8
  • Date: Wed, 20 Apr 2016 14:41:33 +0300

On 20 April 2016 at 14:07, Frediano Ziglio <freddy77 AT gmail.com> wrote:

> 2016-04-20 12:01 GMT+01:00 Velichko Yuriy <velichko.yuriy AT gmail.com>:
> > Your query works for me to:
> >
> > select cast(cast(0x420435044104420420003dd84ddc20007400650073007400 as
> > nvarchar(200)) as sql_variant)
> >
> > Here an example for broken result:
> >
> > create sequence s1;
> > select sys_seq.start_value FROM sys.sequences sys_seq where name = 's1';
> >
>
> Is this related to emoji/encoding ?
>
​Already no, I think. Should I create new thread?​


> Can you post a
> ​​
> TDSDUMP. I don't have a mssql 2012 to test with (it's
> tested by AppVeyor).
>
> Frediano
>
> > On 20 April 2016 at 13:13, Frediano Ziglio <freddy77 AT gmail.com> wrote:
> >
> >> 2016-04-20 9:39 GMT+01:00 Velichko Yuriy <velichko.yuriy AT gmail.com>:
> >> > One more strange thing.
> >> >
> >> > 1. As you mentioned the default property use utf-16 differs on win
> (TRUE)
> >> > and mac (FALSE). So I must to set "use utf-16 = yes" to avoid
> different
> >> > behaviour, Good that it will be fixed in 1.0
> >>
> >> no, the default is the same, just that you are using the trivial iconv
> >> provided with FreeTDS which is not recommended.
> >>
> >> > 2.
> >> >
> >> > tds version = 7.2 - has no effect for emoji
> >> > symbols
> >> > just need to use
> >> > utf-16
> >> > .
> >> >
> >>
> >> Good, so now should work for every protocol version.
> >>
> >> > 3
> >> >
> >> >
> >> >
> >> > tds version = 7.2
> >> > - brokes another kind of strings.
> >>
> >> Which ones?
> >>
> >> > After I set "
> >> >
> >> > tds version = 7.2
> >> >
> >> > "
> >> > I got few broken tests.
> >>
> >> Which tests ?
> >>
> >> > I found out that strings returned for sql_variant has the same
> problems
> >> > like for emoji symbols - string starts with
> >> >
> >> > \0 symbol and has non zero length.
> >> > Looks like similar conversion issue. Using "
> >> >
> >> > tds version = 7.
> >> > 0
> >> > "
> >> > - fixes this issue - returned value is valid.
> >>
> >> Weird. I'm trying this query on tsql
> >>
> >> select cast(cast(0x420435044104420420003dd84ddc20007400650073007400 as
> >> nvarchar(200)) as sql_variant)
> >>
> >> (which is based on your previous string) and looks the strings are
> >> always returned correctly.
> >> I tried from 7.0 to 7.3 without issues.
> >>
> >> > To test this you can create Sequence and run query:
> >> > select sys_seq.start_value FROM sys.sequences sys_seq
> >> >
> >>
> >> Can you post all queries you are using? Which exact sequence is not
> >> working?
> >> Are you trying some invalid coding?
> >>
> >> Frediano
> >>
> >> >
> >> >
> >> >
> >> > On 19 April 2016 at 13:53, Velichko Yuriy <velichko.yuriy AT gmail.com>
> >> wrote:
> >> >
> >> >> Yes, without "use utf-16" symbols are broken.
> >> >> Set
> >> >> use utf-16 = yes
> >> >> and
> >> >> tds version = 7.2
> >> >> fixes this problem on mac.
> >> >>
> >> >> If tds version = 7.0 - symbols are broken.
> >> >>
> >> >> On 19 April 2016 at 13:26, Frediano Ziglio <freddy77 AT gmail.com>
> wrote:
> >> >>
> >> >>> 2016-04-18 21:46 GMT+01:00 Daniel Fazekas <fdsubs AT t-online.hu>:
> >> >>> > It is actually odd that merely upgrading solved that emoji issue.
> >> >>> > Emojis being largely in the Supplementary Multilingual Plane of
> >> Unicode
> >> >>> require true UTF-16 support rather than the legacy UCS-2 encoding on
> >> the
> >> >>> server communication side.
> >> >>> >
> >> >>> > In FreeTDS terms, that means that in addition to version 0.95+ you
> >> >>> ought to include the following extra setting in your configuration
> >> file:
> >> >>> > use utf-16 = yes
> >> >>> >
> >> >>> > So a complete sample configuration would look something like
> >> >>> > tds version = 7.2
> >> >>> > client charset = UTF-8
> >> >>> > use utf-16 = yes
> >> >>> >
> >> >>> > Was it a Windows platform peculiarity that UTF-16 worked even with
> >> that
> >> >>> required setting missing?
> >> >>>
> >> >>> This as replacements iconv use utf16 in place of ucs2.
> >> >>> This however reminds me that "use utf-16" option default should be
> >> >>> changed to 1 (true) for 1.0 version!
> >> >>> I'm doing it right now.
> >> >>>
> >> >>> Frediano
> >> >>>
> >> >>> > In any case, I'd add that to your configuration just to be sure.
> >> >>> > It certaintly was necessary for me on Mac and Linux.
> >> >>> >
> >> >>> >> On Apr 18, 2016, at 12:09, Velichko Yuriy <
> velichko.yuriy AT gmail.com
> >> >
> >> >>> wrote:
> >> >>> >>
> >> >>> >> With version 0.95.91 all works fine. looks like this issue was
> fixed
> >> >>> >> earlier.
> >> >>> >>
> >> >>> >>>>>>>>> On 17 Apr 2016 12:59, "Velichko Yuriy" <
> >> >>> velichko.yuriy AT gmail.com> wrote:
> >> >>> >>>>>>>>> My .conf includes next rows:
> >> >>> >>>>>>>>> tds version = 7.0
> >> >>> >>>>>>>>> client charset = UTF-8
> >> >>> >
> >> >>> > _______________________________________________
> >> >>> > FreeTDS mailing list
> >> >>> > FreeTDS AT lists.ibiblio.org
> >> >>> > http://lists.ibiblio.org/mailman/listinfo/freetds
> >> >>> _______________________________________________
> >> >>> FreeTDS mailing list
> >> >>> FreeTDS AT lists.ibiblio.org
> >> >>> http://lists.ibiblio.org/mailman/listinfo/freetds
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> С уважением, Величко Юрий.
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > С уважением, Величко Юрий.
> >> > _______________________________________________
> >> > FreeTDS mailing list
> >> > FreeTDS AT lists.ibiblio.org
> >> > http://lists.ibiblio.org/mailman/listinfo/freetds
> >> _______________________________________________
> >> FreeTDS mailing list
> >> FreeTDS AT lists.ibiblio.org
> >> http://lists.ibiblio.org/mailman/listinfo/freetds
> >>
> >
> >
> >
> > --
> > С уважением, Величко Юрий.
> > _______________________________________________
> > FreeTDS mailing list
> > FreeTDS AT lists.ibiblio.org
> > http://lists.ibiblio.org/mailman/listinfo/freetds
> _______________________________________________
> FreeTDS mailing list
> FreeTDS AT lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/freetds
>



--
​С уважением, Величко Юрий.




Archive powered by MHonArc 2.6.24.

Top of Page