FreeTDS Development Group

[Peter Deacon <peterd AT iea-software.com>]

On Tue, 8 Jan 2019, Deepa Umesh wrote:

> Hi ,
> I am trying to rephrase my question with more clarity,

> We are using UTF-8 as encoding in our product, which uses single byte of 
> character data, But when we attempt to use non US ASCII UTF-8 charecters 
> , we are having encoding problems similar to the following,

> Insert query : insert into altitem values ('あなたの犬を散歩したいと思います');

> [FreeTDS][SQL Server]Some character(s) could not be converted into 
> client's character set.  Unconverted bytes were changed to question 
> marks ('?')

> > [sqlprog1]
> > Driver=/usr/local/freetds-1.00.97/lib/libtdsodbc.so
> > Server=172.16.127.134
> > Database=sqlprog1_catdev19
> > Port=1433
> > TDS_Version=7.3
> > Client Charset = UTF-8.

Hi Deepa,

Set ClientCharset=CP1252

regards,
Peter
> From Deepa.Umesh AT aptean.com  Tue Jan  8 05:40:24 2019
Return-Path: <Deepa.Umesh AT aptean.com>
X-Original-To: freetds AT lists.ibiblio.org
Delivered-To: freetds AT lists.ibiblio.org
Received: from mx0b-0014cc01.pphosted.com (mx0b-0014cc01.pphosted.com
[208.86.201.194])
by lists.ibiblio.org (Postfix) with ESMTP id 4E1E92014209
for <freetds AT lists.ibiblio.org>; Tue,  8 Jan 2019 05:40:24 -0500 (EST)
Received: from pps.filterd (m0102391.ppops.net [127.0.0.1])
by mx0b-0014cc01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id
x08Ab4tg021524
for <freetds AT lists.ibiblio.org>; Tue, 8 Jan 2019 05:40:23 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aptean.com;
h=from : to : subject :
date : message-id : references : in-reply-to : content-type :
content-transfer-encoding : mime-version; s=own;
bh=dD99LT7JsqmCEktmTYyo64mB2+UIx79E+Aj4hXnkZLg=;
b=J0GUpaFS02UxyRZHVBIfNizrswSDdUWQptNcnP0kyCO6DXqgyH+F4uxeBktvXsRzKdtM
JQNDhRu3f4bLrPynPslmTImtUD7Wb6hrG2NWRe5wJNay75Kr8POIcATm2Zs7kw4z1bbU
pE7ad7bTnF3qZh/f1DTrYY/lVGFzh9BV9QL9WHu4Oz0570GassYkqX8Lmx0q4HYPlTKX
gxctz6VhTOQuWADDt4dKmbQyh2T9Mpgib0D2RhRxPifrL5pyI/J5HcUTYrqXw5T4V3il
j4KF2EqIPMtbYg3vMbSkR2/I+7SJm4AEjlSYhCFWDviA5v7DC2x+Vs5Ou9R21HgrJTEK Dw== 
Received: from nam04-sn1-obe.outbound.protection.outlook.com
(mail-sn1nam04lp2058.outbound.protection.outlook.com [104.47.44.58])
by mx0b-0014cc01.pphosted.com with ESMTP id 2pttdh1pfs-1
(version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT)
for <freetds AT lists.ibiblio.org>; Tue, 08 Jan 2019 05:40:23 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=ApteanOnline.onmicrosoft.com; s=selector1-aptean-com;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=dD99LT7JsqmCEktmTYyo64mB2+UIx79E+Aj4hXnkZLg=;
b=b7NxE+qdycSR8tYtqjz4tHSZT1bYH9j3epnsbQsows5UKrm7PTB7mBCkQV8e1nNwaNv+GN2Axa8n+23szk9Aouo3WN5Mr01eirWhtgT3Eae8kQVKN487+GzD6CZqaUdP9xnYF0NgD6pSM8Woa+l1E/ap2/UYTydVI2tnwf4eCYM=
Received: from SN1PR18MB2272.namprd18.prod.outlook.com (52.132.199.146) by
SN1PR18MB2319.namprd18.prod.outlook.com (52.132.199.157) with Microsoft SMTP
Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
15.20.1495.6; Tue, 8 Jan 2019 10:40:21 +0000
Received: from SN1PR18MB2272.namprd18.prod.outlook.com
([fe80::19f6:93d5:72ab:2016]) by SN1PR18MB2272.namprd18.prod.outlook.com
([fe80::19f6:93d5:72ab:2016%2]) with mapi id 15.20.1495.011; Tue, 8 Jan 2019
10:40:20 +0000
From: Deepa Umesh <Deepa.Umesh AT aptean.com>
To: FreeTDS Development Group <freetds AT lists.ibiblio.org>
Thread-Topic: [freetds] UTF-8 issue
Thread-Index: AdSiYVwEraYquvosRHi6rL0N1eOqrgBrMAuAAMLsXxAABEe5AAAEybXQ
Date: Tue, 8 Jan 2019 10:40:20 +0000
Message-ID: 
<SN1PR18MB22722370CC38E6F669BD77C4878A0 AT SN1PR18MB2272.namprd18.prod.outlook.com>
References: 
<SN1PR18MB227294BE18628A9B0A8AC372878C0 AT SN1PR18MB2272.namprd18.prod.outlook.com>
<CAHt6W4eenkvZhuxHbb=z0nF8u507ZLa+Jz8_ODPQwhn1jtgT3g AT mail.gmail.com>
<BL0PR18MB2259BB651E5A3E778A42AE05878A0 AT BL0PR18MB2259.namprd18.prod.outlook.com>
<alpine.WNT.2.21.1.1901080014290.2516@smurf>
In-Reply-To: <alpine.WNT.2.21.1.1901080014290.2516@smurf>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [115.110.248.130]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1; SN1PR18MB2319;
20:nM+b0EFpReFHUZoaZ8+HU9ZOKpevyQ6icd1GlayqYknC7ACyx8jI+HJ3kXxzAfQqiHJW81T52XWb3Jv0tk/n399fxSPpEb49NMjEiDzgEI3jnulBXm4FV6F6tJLajOK585zGsZ0Zs4wGq9uTAR3EA/GIyAtFfoQR8mfmyBdUKcg=
x-ms-exchange-antispam-srfa-diagnostics: SOS;
x-ms-office365-filtering-correlation-id: 5e183e96-9d9d-4d97-abd6-08d67555b045
x-microsoft-antispam: BCL:0; PCL:0;
RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600109)(711020)(2017052603328)(7153060)(7193020);
SRVR:SN1PR18MB2319; 
x-ms-traffictypediagnostic: SN1PR18MB2319:
x-microsoft-antispam-prvs: 
<SN1PR18MB231926CFA760F990AAA52115878A0 AT SN1PR18MB2319.namprd18.prod.outlook.com>
x-forefront-prvs: 0911D5CE78
x-forefront-antispam-report: SFV:NSPM;
SFS:(10009020)(396003)(346002)(136003)(366004)(376002)(39860400002)(199004)(13464003)(55514002)(189003)(99286004)(14454004)(2906002)(478600001)(55016002)(6436002)(486006)(229853002)(6246003)(6916009)(33656002)(9686003)(6306002)(966005)(53936002)(72206003)(97736004)(3846002)(316002)(5660300001)(74316002)(6116002)(68736007)(7696005)(93886005)(76176011)(102836004)(476003)(81166006)(81156014)(8676002)(53546011)(6506007)(71200400001)(71190400001)(186003)(26005)(305945005)(7736002)(86362001)(256004)(66066001)(5024004)(14444005)(25786009)(19627235002)(105586002)(106356001)(11346002)(446003)(8936002);
DIR:OUT; SFP:1101; SCL:1; SRVR:SN1PR18MB2319;
H:SN1PR18MB2272.namprd18.prod.outlook.com; FPR:; SPF:None; LANG:en;
PTR:InfoNoRecords; A:1; MX:1; 
received-spf: None (protection.outlook.com: aptean.com does not designate
permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: 
sTaiJc3QTWo6d+LZ+joltGCDfBDsWCcHPsyvmT61KiBlamx5wqvQArHj5vuSzywlOKKtqLUL9+PRRzkxw1XzcueQv1DINFj63mmf28qQBnAWnEh3n8Fk9S/PLtQukNetx03lxDJF6H15tSu9NSNvKYePLwf4c/isU3tNFXmTLShzeKD0EDefeR5DatF7CcXXyjR78qM8F8E2+wjizWBmdC1M/dFPFuKmhLsmM5Asp6gr0RMtK5LIxE44yeF6q0PthUV5xMqpbRXgQ9uktSHzRy/plwLVEckw/m3xLcXOmnkKZOW1RqAYNqSOUUaBOcya
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="iso-2022-jp"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: aptean.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 
5e183e96-9d9d-4d97-abd6-08d67555b045
X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Jan 2019 10:40:20.8436 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 560ec2b0-df0c-4e8c-9848-a15718863bb6
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR18MB2319
X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9129
signatures=668680
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0
priorityscore=1501 malwarescore=0
suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015
lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0
classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000
definitions=main-1901080088
Subject: Re: [freetds] UTF-8 issue
X-BeenThere: freetds AT lists.ibiblio.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: FreeTDS Development Group <freetds.lists.ibiblio.org>
List-Unsubscribe: <https://lists.ibiblio.org/mailman/options/freetds>,
<mailto:freetds-request AT lists.ibiblio.org?subject=unsubscribe>
List-Archive: <https://lists.ibiblio.org/sympa/arc/freetds/>
List-Post: <mailto:freetds AT lists.ibiblio.org>
List-Help: <mailto:sympa AT lists.ibiblio.org?subject=HELP>
List-Subscribe: <https://lists.ibiblio.org/mailman/listinfo/freetds>,
<mailto:freetds-request AT lists.ibiblio.org?subject=subscribe>
X-List-Received-Date: Tue, 08 Jan 2019 10:40:24 -0000

Hi Peter,
I set ClientCharset=3DCP1252 in odbc.ini (unixODBC)

But the result is as previous :

[FreeTDS][SQL Server]Some character(s) could not be converted into
client's character set.  Unconverted bytes were changed to question
marks ('?')

So do I need to reinstall freetds with this configuration in freetds.conf ?

Request you to provide additional info on installation stuff.
Also is there any config flags to be set?

Thanks,
Deepa

-----Original Message-----
From: FreeTDS <freetds-bounces AT lists.ibiblio.org> On Behalf Of Peter Deacon
Sent: Tuesday, January 8, 2019 1:50 PM
To: FreeTDS Development Group <freetds AT lists.ibiblio.org>
Subject: Re: [freetds] UTF-8 issue

On Tue, 8 Jan 2019, Deepa Umesh wrote:

> Hi ,
> I am trying to rephrase my question with more clarity,

> We are using UTF-8 as encoding in our product, which uses single byte
> of character data, But when we attempt to use non US ASCII UTF-8
> charecters , we are having encoding problems similar to the following,

> Insert query : insert into altitem values ('=1B$B$"$J$?$N8$$r;6Jb$7$?$$$H=
;W$$$^$9=1B(B');

> [FreeTDS][SQL Server]Some character(s) could not be converted into
> client's character set.  Unconverted bytes were changed to question
> marks ('?')

> > [sqlprog1]
> > Driver=3D/usr/local/freetds-1.00.97/lib/libtdsodbc.so
> > Server=3D172.16.127.134
> > Database=3Dsqlprog1_catdev19
> > Port=3D1433
> > TDS_Version=3D7.3
> > Client Charset =3D UTF-8.

Hi Deepa,

Set ClientCharset=3DCP1252

regards,
Peter
_______________________________________________
FreeTDS mailing list
FreeTDS AT lists.ibiblio.org
https://lists.ibiblio.org/mailman/listinfo/freetds
This message contains information that is confidential and/or may be privil=
eged. If you are not the intended recipient, you are hereby notified that a=
ny use, dissemination, distribution or copy of this message and/or its atta=
chments is strictly prohibited. If you have received this message in error,=
please advise the sender immediately by replying to this email and delete =
this message. We are committed to your privacy. For more information, pleas=
e see our Privacy Policy at http://www.aptean.com/privacy-statement