FreeTDS Development Group

["Lowden, James K" <LowdenJK AT bernstein.com>]

> From: manoranjan_sahu AT agilent.com
> Sent: Tuesday, November 22, 2005 2:14 PM
> 
> I am using unixODBC with freetds v0.63RC10.
> 
> The issue I have is that my application is truncating the real
> number data.
> 
> I was able to see this behavior through tsql as well. Here is the
> test case.
> 
[modified to use temporary table]

        create table #t1 (c1 integer, c2 real)
        go
        delete from #t1
        go
        insert into #t1 values(1, 123456789.123456789)
        insert into #t1 values(2, 12345678.123456789)
        insert into #t1 values(3, 1234567.123456789)
        insert into #t1 values(4, 123456.123456789)
        insert into #t1 values(5, 12345.123456789)
        go

        select c1, c2 from #t1 order by c1
        go

Here are the results in sqsh:

[30] mpquant.testdb.1> select c1, c2 from #t1 order by c1
[30] mpquant.testdb.2> select cast(123456789.123456789 as real)
[30] mpquant.testdb.3> go
 c1          c2
 ----------- --------------------
           1     123456792.000000
           2      12345678.000000
           3       1234567.125000
           4        123456.125000
           5         12345.123047

(5 rows affected)

 --------------------
     123456792.000000

In Microsoft's isql:

2> select c1, c2 from #t1 order by c1
3> go
 c1          c2
 ----------- --------------------
           1  1.234567920000e+008
           2  1.234567800000e+007
           3  1.234567125000e+006
           4        123456.125000
           5         12345.123047

> Please note that the actual data is truncated.

Right.  The real datatype holds 7 digits of precision, irrespective of
where the decimal is.  That's why "select cast(123456789.123456789 as
real)" produces a rounded result; it also explains all of your results.


cf. http://msdn.microsoft.com/library/en-us/tsqlref/ts_fa-fz_6r3g.asp

As you can see, this isn't a library issue.  It's a simple question of
how many bits you're using to represent your number, and there's no
configuration option to stuff more than one bit into a bit.  ;-)

If you need more precision, I recommend float instead.  

--jkl

-----------------------------------------
The information contained in this transmission may be privileged and
confidential and is intended only for the use of the person(s) named
above. If you are not the intended recipient, or an employee or agent 
responsible
for delivering this message to the intended recipient, any review, 
dissemination,
distribution or duplication of this communication is strictly prohibited. If 
you are
not the intended recipient, please contact the sender immediately by reply 
e-mail
and destroy all copies of the original message. Please note that we do not 
accept
account orders and/or instructions by e-mail, and therefore will not be 
responsible
for carrying out such orders and/or instructions.  If you, as the intended 
recipient
of this message, the purpose of which is to inform and update our clients, 
prospects
and consultants of developments relating to our services and products, would 
not
like to receive further e-mail correspondence from the sender, please "reply" 
to the
sender indicating your wishes.  In the U.S.: 1345 Avenue of the Americas, New 
York,
NY 10105.