FreeTDS Development Group

["James K. Lowden" <jklowden AT freetds.org>]

ananthbv AT gmail.com wrote:
> 
> $ hexdump -C tmpreq.txt
> 
> 00000000 31 31 39 31 35 35 37 21 23 80 34 34 2e 33 30 0a
> |1191557!#.44.30.|
                                      ^^ kaching!  

That's a Euro sign in Windows 1252:

http://unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1252.TXT

> I have attached the file for the freebcp out+in session
> (freetds_varchar.log.txt)

Wasn't attached to the mail, sorry.  

I can't be sure why datacopy is working for you without a TDSDUMP log.  I
can tell you your file appears encoded as cp1252.  How the data are
encoded on your servers we still don't know.  

> I read in the sybase manuals that iso_1 does not
> have euro symbol. But when I did a sybase bcp out with -J iso_1, I
> could see the euro symbol in the output file. How does this happen?

Sybase wouldn't lie to you!  

Sometimes data are misencoded on the server and work by happy accident
until something comes along expecting it to be right.  In your case, *if*
the Sybase server really is using iso_1, that might be what's happening. 
iso_1 is ISO 8859-1, and ISO 8859-1 has neither a definition for the value
0x80 nor for a  Euro symbol.  That doesn't always prevent applications
from stashing non-iso_1 data in the colunn (although, really, it should). 


http://en.wikipedia.org/wiki/ISO/IEC_8859-1 has a nice picture of 8859-1. 
Note the gray area on the 8x and 9x lines.  For extra credit, find out
why.  ;-) 

There are a two ways that 0x80 could have gotten there:

1.  FreeTDS (or Sybase) converted it from the server's format because it
decided the client wants cp1252.  That would be nice: both side correctly
configured and functioning.  

2.  Neither FreeTDS nor Sybase converted it, meaning either:

2a.  Both server and client are (supposed to be cp1252), or 
2b.  Neither server nor client are cp1252, but are instead ISO 8859-1, and
the bad data on the server are simply being stored verbatim by freebcp.  

The 2b case seems most likely to me because iso_1 and 8859-1 are common
arrangements, and because it conforms to the second half of your
experience: if your client isn't announcing itself as cp1252 to either
server, yet is receiving (in effect) data with cp1252 encoding, then you
will see errors at upload time, when FreeTDS attempts to convert what it
thinks is 8859-1 data to the encoding used on the Microsoft server.  

That a theory, at least.  The log should make a lot of this more definite.
 

HTH.  

--jkl