freetds AT lists.ibiblio.org
Subject: FreeTDS Development Group
List archive
- From: Will McQueen <wmcqueen AT athenahealth.com>
- To: "freetds AT lists.ibiblio.org" <freetds AT lists.ibiblio.org>
- Subject: [freetds] freebcp UTF8 to UCS-2LE importing
- Date: Wed, 1 Jun 2011 15:44:32 -0400
Hi, I've been searching the email lists for exactly my problem but haven't
quite found it. Let me start with my problem.
I have a UTF8 formatted data file that I'm importing into a SQL Server 2008
table which has columns typed as nvarchar and the import seems to truncate
the data to half the size it should be. Here are some tdsdump snippets:
log.c:196:Starting log file for FreeTDS 0.91RC2
...
iconv.c:349:setting up conversions for client charset "UTF8"
...
token.c:3423:adjust_character_column_size:
Server charset: UCS-2LE
Server column_size: 192
Client charset: UTF-8
Client column_size: 384
token.c:1489:tds7_get_data_info:
colname = CONTEXT_NAME (12 bytes)
type = 39 (varchar)
server's type = 231 (x UCS-2 varchar)
column_varint_size = 2
column_size = 384 (192 on server)
...
bcp.c:1141:parsing host column 2
bcp.c:1160:host column 2 uses bcpcol 2 (0xbaf3390)
bcp.c:1218:prefix_len = 0 collen = 0
bcp.c:1488:_bcp_measure_terminated_field(0xbaf8a40, 0xbaf8860, 1)
bcp.c:1249:_bcp_measure_terminated_field returned 46
bcp.c:1258:Adjusted collen is 46.
iconv.c:874:tds_iconv_fread: read 46 of 46 bytes; outbuf has 46 left.
iconv.c:891:tds_iconv_fread: error 7: Argument list too long.
bcp.c:1379:Data read from hostfile: collen is now 46, data_is_null is 0
dblib.c:2198:dbconvert(0xbab7460, SYBCHAR, 0xbaee340, 46, SYBVARCHAR,
0xbab7fe0, 384)
...
I think the error in bcp.c is line 1258 where it thinks the adjusted collen
should be 46. I think it should actually be 92, as the 46 bytes read in is
variable length UTF8 data. It could be 46 1-byte characters or it could be
12 multi-byte characters but in the worst case its 46 characters in which
case converting it to UCS-2LE will require 92 bytes in the outbuffer. The
error 7 from tds_iconv_fread seems to confirm this, as the E2BIG error is
returned from iconv when the outbuffer isn't large enough to hold the
resulting string.
Another indication this is the issue is if I switch my client charset to
ISO-8859-1. My data is almost entirely valid ISO-8859-1 data so I can
process many rows this way and in that case it properly adjusts the collen to
92.
I'll see if I can work on a patch that can be applied, but I'm curious if
anyone else has any feedback for me.
Thanks,
Will
-
[freetds] freebcp UTF8 to UCS-2LE importing,
Will McQueen, 06/01/2011
- <Possible follow-up(s)>
-
Re: [freetds] freebcp UTF8 to UCS-2LE importing,
Will McQueen, 06/02/2011
- Re: [freetds] freebcp UTF8 to UCS-2LE importing, Frediano Ziglio, 06/02/2011
Archive powered by MHonArc 2.6.24.