Skip to Content.
Sympa Menu

freetds - Re: [freetds] Truncated data for real data type in Sybase.

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: "Lowden, James K" <LowdenJK AT bernstein.com>
  • To: "FreeTDS Development Group" <freetds AT lists.ibiblio.org>
  • Subject: Re: [freetds] Truncated data for real data type in Sybase.
  • Date: Tue, 22 Nov 2005 17:33:03 -0500

> From: manoranjan_sahu AT agilent.com
> Sent: Tuesday, November 22, 2005 2:14 PM
>
> I am using unixODBC with freetds v0.63RC10.
>
> The issue I have is that my application is truncating the real
> number data.
>
> I was able to see this behavior through tsql as well. Here is the
> test case.
>
[modified to use temporary table]

create table #t1 (c1 integer, c2 real)
go
delete from #t1
go
insert into #t1 values(1, 123456789.123456789)
insert into #t1 values(2, 12345678.123456789)
insert into #t1 values(3, 1234567.123456789)
insert into #t1 values(4, 123456.123456789)
insert into #t1 values(5, 12345.123456789)
go

select c1, c2 from #t1 order by c1
go

Here are the results in sqsh:

[30] mpquant.testdb.1> select c1, c2 from #t1 order by c1
[30] mpquant.testdb.2> select cast(123456789.123456789 as real)
[30] mpquant.testdb.3> go
c1 c2
----------- --------------------
1 123456792.000000
2 12345678.000000
3 1234567.125000
4 123456.125000
5 12345.123047

(5 rows affected)

--------------------
123456792.000000

In Microsoft's isql:

2> select c1, c2 from #t1 order by c1
3> go
c1 c2
----------- --------------------
1 1.234567920000e+008
2 1.234567800000e+007
3 1.234567125000e+006
4 123456.125000
5 12345.123047

> Please note that the actual data is truncated.

Right. The real datatype holds 7 digits of precision, irrespective of
where the decimal is. That's why "select cast(123456789.123456789 as
real)" produces a rounded result; it also explains all of your results.


cf. http://msdn.microsoft.com/library/en-us/tsqlref/ts_fa-fz_6r3g.asp

As you can see, this isn't a library issue. It's a simple question of
how many bits you're using to represent your number, and there's no
configuration option to stuff more than one bit into a bit. ;-)

If you need more precision, I recommend float instead.

--jkl

-----------------------------------------
The information contained in this transmission may be privileged and
confidential and is intended only for the use of the person(s) named
above. If you are not the intended recipient, or an employee or agent
responsible
for delivering this message to the intended recipient, any review,
dissemination,
distribution or duplication of this communication is strictly prohibited. If
you are
not the intended recipient, please contact the sender immediately by reply
e-mail
and destroy all copies of the original message. Please note that we do not
accept
account orders and/or instructions by e-mail, and therefore will not be
responsible
for carrying out such orders and/or instructions. If you, as the intended
recipient
of this message, the purpose of which is to inform and update our clients,
prospects
and consultants of developments relating to our services and products, would
not
like to receive further e-mail correspondence from the sender, please "reply"
to the
sender indicating your wishes. In the U.S.: 1345 Avenue of the Americas, New
York,
NY 10105.




Archive powered by MHonArc 2.6.24.

Top of Page