Skip to Content.
Sympa Menu

freetds - [freetds] freebcp UTF8 to UCS-2LE importing

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: Will McQueen <wmcqueen AT athenahealth.com>
  • To: "freetds AT lists.ibiblio.org" <freetds AT lists.ibiblio.org>
  • Subject: [freetds] freebcp UTF8 to UCS-2LE importing
  • Date: Wed, 1 Jun 2011 15:44:32 -0400

Hi, I've been searching the email lists for exactly my problem but haven't
quite found it. Let me start with my problem.

I have a UTF8 formatted data file that I'm importing into a SQL Server 2008
table which has columns typed as nvarchar and the import seems to truncate
the data to half the size it should be. Here are some tdsdump snippets:

log.c:196:Starting log file for FreeTDS 0.91RC2
...
iconv.c:349:setting up conversions for client charset "UTF8"
...
token.c:3423:adjust_character_column_size:
Server charset: UCS-2LE
Server column_size: 192
Client charset: UTF-8
Client column_size: 384
token.c:1489:tds7_get_data_info:
colname = CONTEXT_NAME (12 bytes)
type = 39 (varchar)
server's type = 231 (x UCS-2 varchar)
column_varint_size = 2
column_size = 384 (192 on server)
...
bcp.c:1141:parsing host column 2
bcp.c:1160:host column 2 uses bcpcol 2 (0xbaf3390)
bcp.c:1218:prefix_len = 0 collen = 0
bcp.c:1488:_bcp_measure_terminated_field(0xbaf8a40, 0xbaf8860, 1)
bcp.c:1249:_bcp_measure_terminated_field returned 46
bcp.c:1258:Adjusted collen is 46.
iconv.c:874:tds_iconv_fread: read 46 of 46 bytes; outbuf has 46 left.
iconv.c:891:tds_iconv_fread: error 7: Argument list too long.
bcp.c:1379:Data read from hostfile: collen is now 46, data_is_null is 0
dblib.c:2198:dbconvert(0xbab7460, SYBCHAR, 0xbaee340, 46, SYBVARCHAR,
0xbab7fe0, 384)
...

I think the error in bcp.c is line 1258 where it thinks the adjusted collen
should be 46. I think it should actually be 92, as the 46 bytes read in is
variable length UTF8 data. It could be 46 1-byte characters or it could be
12 multi-byte characters but in the worst case its 46 characters in which
case converting it to UCS-2LE will require 92 bytes in the outbuffer. The
error 7 from tds_iconv_fread seems to confirm this, as the E2BIG error is
returned from iconv when the outbuffer isn't large enough to hold the
resulting string.

Another indication this is the issue is if I switch my client charset to
ISO-8859-1. My data is almost entirely valid ISO-8859-1 data so I can
process many rows this way and in that case it properly adjusts the collen to
92.

I'll see if I can work on a patch that can be applied, but I'm curious if
anyone else has any feedback for me.

Thanks,
Will




Archive powered by MHonArc 2.6.24.

Top of Page