FreeTDS Development Group

[Frediano Ziglio <freddy77 AT gmail.com>]

2011/5/31 Thompson, William <bill.d.thompson AT baml.com>:
> Hi All,
>
> I've had a new issue reported to me on freebcp.
> This has come from someone migrating from Sybase to SQL Server, using 
> freetds as their "enabler" - an increasing trend in the use of freetds, I'm 
> told.
> So we have some solid evidence - "This works using Sybase bcp, but it 
> doesn't work with freebcp" - and although it's an edge case, I think it 
> should be fixed.
>
> So the problem occurs with freebcp using a format file. It evidences itself 
> as a dbconvert() overflow error...
>
> The problematic column is CHAR(6) on the database
> The line from the format file for the column looks like this:
>
> 22      SYBCHAR 0       6       "~"     22      cost_center
>
> and a snippet from the problematic line from the data file, showing the 
> problematic host file data looks like this:
>
> ...~123456  ~...
>
> So in the format file, we've given a host file data length as 6 (the fourth 
> token in the format file), but we've also said the data is terminated  (by 
> a "~")
> However, there's a couple of extra spaces on the end of that data in the 
> data file, giving it a total length of 8 characters.
>
> Well I took a look at this, and the Sybase manual pages for format files 
> tells me the following about the host file data length:
>
>
> "Host file data length refers to the maximum number of bytes to copy for 
> the field. To decide how much data to copy in or out, bcp uses one of:
>
> - The maximum field length
> - The prefix length, if any
> - The field terminator string, if any
>
> If more than one method of field length specification is given, bcp chooses 
> the one that copies the least amount of data."

The field length is taken from database information or from format
file ?? It we truncate to database length we will just silently
truncate data. I remember I saw a truncation warning/error using bcp
so I think truncating always is not correct.

In your example data was "123456  ", perhaps dblib just trim spaces so
it does not give error. I don't know relationship between format file
and bcp.

> This caused me to delve into _bcp_read_hostfile()  (in src/dblib/bcp.c). To 
> be honest, it shows the signs of having "evolved" to deal with the myriad 
> of possibilities available to the innocent or pragmatic user. I'm probably 
> to blame for much of that ad-hoc evolution, and here's another mutation to 
> add to the mix. It works, and doesn't make things significantly worse than 
> they they currently are:
> Add the following lines to the function after approximately line 1421 of 
> bcp.c (some context given):
>
>               /*
>                * FIXME bcpcol->bcp_column_data->data && bcpcol->column_size 
> ??
>                * It seems a buffer overflow waiting...
>                */
>
> +              /* If column length specified (in a format file), then it 
> *might*   */
> +              /* be less than the data read from file (if the column was 
> also     */
> +              /* described as a terminated field...) in this case we need 
> to only */
> +              /* take the number of characters specified in the column 
> length     */
> +
> +              if (hostcol->column_len > 0) {
> +                     collen = (hostcol->column_len < collen) ? 
> hostcol->column_len : collen;
> +              }
>
>               bcpcol->bcp_column_data->datalen =
>                    dbconvert(dbproc, hostcol->datatype, (const BYTE *) 
> coldata, collen, desttype,
>                              bcpcol->bcp_column_data->data, 
> bcpcol->column_size);
>
>               if (bcpcol->bcp_column_data->datalen == -1) {
>
>
> In penance, I may re-work the function at some point soon. The code is 
> going to keep me awake at nights otherwise...
>
> Bill
>

Frediano