FreeTDS Development Group

[Peter Deacon <peterd AT iea-software.com>]

On Tue, 8 Jan 2019, Deepa Umesh wrote:

> Hi Peter,
> I set ClientCharset=CP1252 in odbc.ini (unixODBC)
> But the result is as previous :

> [FreeTDS][SQL Server]Some character(s) could not be converted into
> client's character set.  Unconverted bytes were changed to question
> marks ('?')

Hi Deepa,

Normally this encoding causes everything to be mapped across so above 
would not occur.  Recommend checking FreeTDS logs to confirm this is 
actually encoding used by FreeTDS.

Also make sure FreeTDS uses libiconv when compiling.

regards,
Peter

> So do I need to reinstall freetds with this configuration in freetds.conf ?


> Request you to provide additional info on installation stuff.
> Also is there any config flags to be set?
>
> Thanks,
> Deepa
>
> -----Original Message-----
> From: FreeTDS <freetds-bounces AT lists.ibiblio.org> On Behalf Of Peter Deacon
> Sent: Tuesday, January 8, 2019 1:50 PM
> To: FreeTDS Development Group <freetds AT lists.ibiblio.org>
> Subject: Re: [freetds] UTF-8 issue
>
> On Tue, 8 Jan 2019, Deepa Umesh wrote:
>
> > Hi ,
> > I am trying to rephrase my question with more clarity,
>
> > We are using UTF-8 as encoding in our product, which uses single byte
> > of character data, But when we attempt to use non US ASCII UTF-8
> > charecters , we are having encoding problems similar to the following,
>
> > Insert query : insert into altitem values ('あなたの犬を散歩したいと思います');
>
> > [FreeTDS][SQL Server]Some character(s) could not be converted into
> > client's character set.  Unconverted bytes were changed to question
> > marks ('?')
>
> > > [sqlprog1]
> > > Driver=/usr/local/freetds-1.00.97/lib/libtdsodbc.so
> > > Server=172.16.127.134
> > > Database=sqlprog1_catdev19
> > > Port=1433
> > > TDS_Version=7.3
> > > Client Charset = UTF-8.
>
> Hi Deepa,
>
> Set ClientCharset=CP1252
>
> regards,
> Peter
> _______________________________________________
> FreeTDS mailing list
> FreeTDS AT lists.ibiblio.org
> https://lists.ibiblio.org/mailman/listinfo/freetds
> This message contains information that is confidential and/or may be 
> privileged. If you are not the intended recipient, you are hereby notified 
> that any use, dissemination, distribution or copy of this message and/or its 
> attachments is strictly prohibited. If you have received this message in 
> error, please advise the sender immediately by replying to this email and 
> delete this message. We are committed to your privacy. For more information, 
> please see our Privacy Policy at http://www.aptean.com/privacy-statement
> _______________________________________________
> FreeTDS mailing list
> FreeTDS AT lists.ibiblio.org
> https://lists.ibiblio.org/mailman/listinfo/freetds
>
> From Deepa.Umesh AT aptean.com  Wed Jan  9 01:52:08 2019
Return-Path: <Deepa.Umesh AT aptean.com>
X-Original-To: freetds AT lists.ibiblio.org
Delivered-To: freetds AT lists.ibiblio.org
Received: from mx0b-0014cc01.pphosted.com (mx0a-0014cc01.pphosted.com
[208.84.65.236])
by lists.ibiblio.org (Postfix) with ESMTP id 9749620141EE
for <freetds AT lists.ibiblio.org>; Wed,  9 Jan 2019 01:52:08 -0500 (EST)
Received: from pps.filterd (m0041843.ppops.net [127.0.0.1])
by mx0a-0014cc01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id
x096nl5E026442
for <freetds AT lists.ibiblio.org>; Wed, 9 Jan 2019 01:52:07 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aptean.com;
h=from : to : subject :
date : message-id : references : in-reply-to : content-type :
content-transfer-encoding : mime-version; s=own;
bh=3X5oPfbAIc2iUR9VqoichAbKTPb9zpPDCyFzTTijKhE=;
b=JcIPlBSIInzR74jbQlznfGAKn0HBF1DvqTHXyE4DldwfH4PsoduhdF/uFq8NrppVJ9/4
X1wZ0y26HEvvJirHlRrIxvwow51F/cIhFqgv+YwP8x9/ysK+JfH7jJd5yrUOfnnilyDd
cjhVxlVR5s9CzsbJTfZv38TUsZeDJ96+CMC9FtzJUTIwWv0y4eCdTbIUeKokE+tTIc0h
x+tFhbnAyDsApxzJGyhazxHCWmj6Vn/65FwOLrlJ3hAa5c9UxfGkUS219LeZGd+oVgSP
YWoUyB0DVqTy+YQW9dtrOoFZHUpBtGKC6vzzwroU6kRooBA/lQ4kHPyqEha+wyy8VjlC 7w== 
Received: from nam04-co1-obe.outbound.protection.outlook.com
(mail-co1nam04lp2054.outbound.protection.outlook.com [104.47.45.54])
by mx0a-0014cc01.pphosted.com with ESMTP id 2pu41yupfe-1
(version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT)
for <freetds AT lists.ibiblio.org>; Wed, 09 Jan 2019 01:52:07 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=ApteanOnline.onmicrosoft.com; s=selector1-aptean-com;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=3X5oPfbAIc2iUR9VqoichAbKTPb9zpPDCyFzTTijKhE=;
b=Klz91rLo8+74HwqAmd9qRP7UIT2nqczCuuPW/rB0xFlXB6g5m6J9w2FTCi5fPSULO3IZbn5IJYkKald/5JQbUfiwao5zMFScqzqs/ReyVYB/+hFwbYMaPmALVTXXGW9lS75Oji0vQXbYugqnXoN4qnM9hr/qky/4eCsbSqmj91Y=
Received: from SN1PR18MB2272.namprd18.prod.outlook.com (52.132.199.146) by
SN1PR18MB2125.namprd18.prod.outlook.com (52.132.196.33) with Microsoft SMTP
Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
15.20.1495.6; Wed, 9 Jan 2019 06:52:05 +0000
Received: from SN1PR18MB2272.namprd18.prod.outlook.com
([fe80::19f6:93d5:72ab:2016]) by SN1PR18MB2272.namprd18.prod.outlook.com
([fe80::19f6:93d5:72ab:2016%2]) with mapi id 15.20.1495.011; Wed, 9 Jan 2019
06:52:05 +0000
From: Deepa Umesh <Deepa.Umesh AT aptean.com>
To: FreeTDS Development Group <freetds AT lists.ibiblio.org>
Thread-Topic: [freetds] UTF-8 issue
Thread-Index: 
AdSiYVwEraYquvosRHi6rL0N1eOqrgBrMAuAAMLsXxAABEe5AAAEybXQAA5aPoAAGWr8AA==
Date: Wed, 9 Jan 2019 06:52:05 +0000
Message-ID: 
<SN1PR18MB22725F961072602990782C1C878B0 AT SN1PR18MB2272.namprd18.prod.outlook.com>
References: 
<SN1PR18MB227294BE18628A9B0A8AC372878C0 AT SN1PR18MB2272.namprd18.prod.outlook.com>
<CAHt6W4eenkvZhuxHbb=z0nF8u507ZLa+Jz8_ODPQwhn1jtgT3g AT mail.gmail.com>
<BL0PR18MB2259BB651E5A3E778A42AE05878A0 AT BL0PR18MB2259.namprd18.prod.outlook.com>
<alpine.WNT.2.21.1.1901080014290.2516@smurf>
<SN1PR18MB22722370CC38E6F669BD77C4878A0 AT SN1PR18MB2272.namprd18.prod.outlook.com>
<alpine.WNT.2.21.1.1901080918570.2516@smurf>
In-Reply-To: <alpine.WNT.2.21.1.1901080918570.2516@smurf>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [115.110.248.130]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1; SN1PR18MB2125;
20:v1DkaCdGSnRUWoXx4f9jVZeBdiVCoeW2RKlmKVSyXDxFnrLTYuyuTncHaXZFKgw7OXSBNRI18CvavhO38AkQsSe018NbMWZe7MNEydB2S3GSBYPILdsWrw17ePphEm+LFAfSSsGeVmGvx6WpuCjTBKKCV82eTIHH7ke55/rlYqg=
x-ms-exchange-antispam-srfa-diagnostics: SOS;
x-ms-office365-filtering-correlation-id: c4fa5104-27a0-4bad-fa54-08d675fef771
x-microsoft-antispam: BCL:0; PCL:0;
RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600109)(711020)(2017052603328)(7153060)(7193020);
SRVR:SN1PR18MB2125; 
x-ms-traffictypediagnostic: SN1PR18MB2125:
x-microsoft-antispam-prvs: 
<SN1PR18MB2125EA23144A2CD3C596A766878B0 AT SN1PR18MB2125.namprd18.prod.outlook.com>
x-forefront-prvs: 0912297777
x-forefront-antispam-report: SFV:NSPM;
SFS:(10009020)(346002)(376002)(366004)(39860400002)(136003)(396003)(55514002)(13464003)(189003)(199004)(476003)(229853002)(53936002)(7696005)(76176011)(26005)(86362001)(106356001)(93886005)(8936002)(25786009)(102836004)(55016002)(6306002)(9686003)(186003)(305945005)(68736007)(7736002)(316002)(3846002)(19627235002)(6116002)(66066001)(2906002)(486006)(256004)(14444005)(71200400001)(71190400001)(14454004)(5024004)(74316002)(478600001)(99286004)(6916009)(5660300001)(11346002)(446003)(6436002)(6506007)(81166006)(81156014)(6246003)(53546011)(8676002)(105586002)(72206003)(966005)(33656002)(97736004);
DIR:OUT; SFP:1101; SCL:1; SRVR:SN1PR18MB2125;
H:SN1PR18MB2272.namprd18.prod.outlook.com; FPR:; SPF:None; LANG:en;
PTR:InfoNoRecords; MX:1; A:1; 
received-spf: None (protection.outlook.com: aptean.com does not designate
permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: 
W6V/i1/mbx8pC1H1AiJ7Nv5ASv0mhx4rDuZnD5MFN3XgnqUNA34XGwn6rTmHEt4KEN15Wbx6XM7RcBP3WP+89WnRj9ETHvk00mmLmZ1LngryEWM2nyak7elTnbDg3osfay6pZR0e+bBKwLhDySLs1dOelLJt5G+yT5L9fw8McKGm3dZ2LyisJYV7YXq/Q1XF5EDhLtmwJKOOQmHYm5iyOYPNHXyEYEDttfNyEoxfbwqLUhtPSfAp3q6DTz3IBcKMWKbd3dJUj7GcUCbbV+fiKMoXNyt1sFOyuNCFTnnrkRkXZezYSQfEvMxfvYr9Ngjk
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="iso-2022-jp"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: aptean.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 
c4fa5104-27a0-4bad-fa54-08d675fef771
X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Jan 2019 06:52:05.1995 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 560ec2b0-df0c-4e8c-9848-a15718863bb6
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR18MB2125
X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9130
signatures=668680
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0
priorityscore=1501 malwarescore=0
suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015
lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0
classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000
definitions=main-1901090058
Subject: Re: [freetds] UTF-8 issue
X-BeenThere: freetds AT lists.ibiblio.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: FreeTDS Development Group <freetds.lists.ibiblio.org>
List-Unsubscribe: <https://lists.ibiblio.org/mailman/options/freetds>,
<mailto:freetds-request AT lists.ibiblio.org?subject=unsubscribe>
List-Archive: <https://lists.ibiblio.org/sympa/arc/freetds/>
List-Post: <mailto:freetds AT lists.ibiblio.org>
List-Help: <mailto:sympa AT lists.ibiblio.org?subject=HELP>
List-Subscribe: <https://lists.ibiblio.org/mailman/listinfo/freetds>,
<mailto:freetds-request AT lists.ibiblio.org?subject=subscribe>
X-List-Received-Date: Wed, 09 Jan 2019 06:52:08 -0000

Hi Peter,

Let me summarize my installation and configuration settings :

We are using FreeTDS(freetds-1.00.99) with UnixODBC(unixODBC-2.2.13) in Lin=
ux Platform and provided the following configuration in
Odbc.ini in UnicODBC is as below:

[sqlprog1]
Driver=3D/usr/local/freetds-1.00.97/lib/libtdsodbc.so
Server=3D172.16.127.134
Database=3Dsqlprog1_catdev19
Port=3D1433
TDS_Version=3D7.3
Client Charset =3D CP1252

CP1252 is used in Windows Platform , is there anything similar to CP1252 in=
Linux?

Thanks,
Deepa

-----Original Message-----
From: FreeTDS <freetds-bounces AT lists.ibiblio.org> On Behalf Of Peter Deacon
Sent: Tuesday, January 8, 2019 10:58 PM
To: FreeTDS Development Group <freetds AT lists.ibiblio.org>
Subject: Re: [freetds] UTF-8 issue

On Tue, 8 Jan 2019, Deepa Umesh wrote:

> Hi Peter,
> I set ClientCharset=3DCP1252 in odbc.ini (unixODBC) But the result is as
> previous :

> [FreeTDS][SQL Server]Some character(s) could not be converted into
> client's character set.  Unconverted bytes were changed to question
> marks ('?')

Hi Deepa,

Normally this encoding causes everything to be mapped across so above would=
not occur.  Recommend checking FreeTDS logs to confirm this is actually en=
coding used by FreeTDS.

Also make sure FreeTDS uses libiconv when compiling.

regards,
Peter

> So do I need to reinstall freetds with this configuration in freetds.conf=
?


> Request you to provide additional info on installation stuff.
> Also is there any config flags to be set?
>
> Thanks,
> Deepa
>
> -----Original Message-----
> From: FreeTDS <freetds-bounces AT lists.ibiblio.org> On Behalf Of Peter
> Deacon
> Sent: Tuesday, January 8, 2019 1:50 PM
> To: FreeTDS Development Group <freetds AT lists.ibiblio.org>
> Subject: Re: [freetds] UTF-8 issue
>
> On Tue, 8 Jan 2019, Deepa Umesh wrote:
>
> > Hi ,
> > I am trying to rephrase my question with more clarity,
>
> > We are using UTF-8 as encoding in our product, which uses single byte
> > of character data, But when we attempt to use non US ASCII UTF-8
> > charecters , we are having encoding problems similar to the
> > following,
>
> > Insert query : insert into altitem values ('=1B$B$"$J$?$N8$$r;6Jb$7$?$$$=
H;W$$$^$9=1B(B');
> > [FreeTDS][SQL Server]Some character(s) could not be converted into
> > client's character set.  Unconverted bytes were changed to question
> > marks ('?')
>
> > > [sqlprog1]
> > > Driver=3D/usr/local/freetds-1.00.97/lib/libtdsodbc.so
> > > Server=3D172.16.127.134
> > > Database=3Dsqlprog1_catdev19
> > > Port=3D1433
> > > TDS_Version=3D7.3
> > > Client Charset =3D UTF-8.
>
> Hi Deepa,
>
> Set ClientCharset=3DCP1252
>
> regards,
> Peter
> _______________________________________________
> FreeTDS mailing list
> FreeTDS AT lists.ibiblio.org
> https://lists.ibiblio.org/mailman/listinfo/freetds
> This message contains information that is confidential and/or may be
> privileged. If you are not the intended recipient, you are hereby
> notified that any use, dissemination, distribution or copy of this
> message and/or its attachments is strictly prohibited. If you have
> received this message in error, please advise the sender immediately
> by replying to this email and delete this message. We are committed to
> your privacy. For more information, please see our Privacy Policy at
> http://www.aptean.com/privacy-statement
> _______________________________________________
> FreeTDS mailing list
> FreeTDS AT lists.ibiblio.org
> https://lists.ibiblio.org/mailman/listinfo/freetds
_______________________________________________
FreeTDS mailing list
FreeTDS AT lists.ibiblio.org
https://lists.ibiblio.org/mailman/listinfo/freetds
This message contains information that is confidential and/or may be privil=
eged. If you are not the intended recipient, you are hereby notified that a=
ny use, dissemination, distribution or copy of this message and/or its atta=
chments is strictly prohibited. If you have received this message in error,=
please advise the sender immediately by replying to this email and delete =
this message. We are committed to your privacy. For more information, pleas=
e see our Privacy Policy at http://www.aptean.com/privacy-statement