FreeTDS Development Group

["James K. Lowden" <jklowden AT freetds.org>]

ananthbv AT gmail.com wrote:
> Hi,
> 
> I am trying to freebcp out data from a table Sybase and freebcp it in
> to a SQL Server table. The FreeTDS installed is on a Linux box
> 
> Some of the data contain a euro symbol. The output file of freebcp
> shows the euro symbol as '~@'. When that data is loaded into the table
> with freebcp in, it is failing with the error
> 
> Starting copy...
> Msg 20088, Level 3
> I/O error while writing bcp error file
> 
> The error file (passed as -e parameter to the freebcp in command shows
> the row (and mentions the column number) which has the euro symbol.
...
> FreeTDS version: 0.82-1
> Sybase: 12.5
> SQL Server: 2008 R2

Boy, have you got a doozie.  Sybase, Microsoft, character encoding, and
bcp all in one message.  It's the toughest question I've seen here in
quite a while.  

I think it can be made to work, but it's tricky because we have to get all
the encodings right.  To make it more interesting, Sybase and Microsoft
servers have different client character encoding policies: Sybase tries to
use the client's encoding, whereas Microsoft simply describes the encoding
and leaves the rest up to you.  I'm not completely sure how well FreeTDS's
character encoding works with TDS 5.0, and I don't have an easy way to
test it.  

First things first: 

1.  What is the encoding for the column on the Sybase server?
2.  What is the encoding for the column on the Microsoft server?  
3.  What encoding are you using on your client?  

Let's look at the Sybase side first.  

> The output file of freebcp shows the euro symbol as '~@'. 

I don't doubt you saw those characters on your screen, but I can't
interpret them.  

We're going to need a TDSDUMP log of the session.  We need to know the
bytes the server sent, what conversion (if any) FreeTDS tried to apply,
and what bytes were written to your file.  The log should show all that.  

At the top of the TDSDUMP log you should also see the client and server
encodings in use for the session.  (When we get to the Microsoft side,
it's important to remember that individual columns can use distinct
encodings.)  

To verify the bytes written to the file, we need a hex dump.  Something
like this might do the trick:

        grep '@' datafile | hexdump -C

Once we establish the encoding of the data as received from Sybase, we
have a fighting chance of uploading it on the Microsoft server.  The
target column can use any encoding that supports the Euro symbol.  Unicode
(UCS-2) will work, so will Code Page 1252.  

--jkl