FreeTDS Development Group

["Lowden, James K" <LowdenJK AT bernstein.com>]

All, 

As we add functionality to libtds, some of its spindlier and more
provisional legs are becoming unsteady.  I think the fundamental data
structures -- that hold data and meta data for result sets, including
compute rows and paramaters -- should be restructured along the lines of the
attached.  

If you're interested, please review the attached design and tell me what you
think.  If you think its good but infeasible, let's see if we can come up
with a way to take advantage of some of it, on the road to a better
tomorrow.  If you think it's pointless, don't worry, we have too many
pointful things already in TODO.  

I have these goals:

1.  16-bit clean: no strlen(), use counted buffers.  
2.  separate data from meta data.
3.  support zerocopy.
4.  base structures on protocol packets where appropriate. 
5.  cleaner names via nested structures. 
6.  better consistency.
7.  good structures lead to good code.
8.  good code engenders participation and satisfaction.  

In the end, I want to make libtds more accessible, to make it easier to add
to the client libraries, and easier to spot goofs in libtds itself.  

There are basically four kinds of data, for purposes of this discussion:

1.  Column metadata
2.  Row metadata
3.  Binding metadata
4.  Data

(#4 being what the user cares about.)

These can be combined in many (!4 ?) ways.  At present, they're munged into
TDSCOLINFO, except for some (inconsistent) parts that are in TDSRESULTINFO.
The thing is unwieldy; it lacks even the concept of a row.  

I propose to nest things in much the way we've done, but with structures to
delineate the parts:

        row ::
                row meta
                column array ::
                        column meta
                        column ::
                                binding meta
                                data

That is, a row consists metadata and columns, and each column consists of
metadata (from the server), bind metadata (from the client), and data.  The
latter two are nested for convenient communication with the client.  

Note there is practically no state information, no notion of a "current" row
or column or size or offset.  The row metadata tell you how many columns
there are.  You merely index into that array; the column metadata tell you
the size of the column, the data.cb tells you the size of the data.  

The metadata arrive first, and are provisioned along with their
corresponding buffers.  The data arrive in rows; each field gets put in its
own little buffer, and the whole shebang is passed back to the client.  

I also considered another design, one that more closely reflects the
protocol, where the data and metadata are sharply segregated:

        row ::
                meta ::
                        row meta
                        column meta array
                column array ::
                        binding meta
                        data

but that requires parallel arrays and more pointers, both of which I prefer
to avoid.  

Notes on the attached C structures.

1.  The term "pbcb" stands for "pointer to buffer count of bytes", and I
didn't invent it.  It's a little like a Pascal string.  Since each buffer
needs a length, it's helpful to combine the two tersely.  It also puts the
pointer at the very end of nested structures, which is very convenient to
use.  

2.  I've used my naming conventions while trying to respect the FreeTDS
style.  I don't insist on them if you don't like them.  

3.  Every field is accounted for, else I made an error.  There are some new
ones (order by, schema name, column label).  Might as well make room;
they're defined in the protocol.  

4.  I haven't tried to account for multibyte character sets.  The protocol
and the APIs deal mainly in byte counts.  Where character counts are
distinct, I think it will be up to the client library to compute them.  

5.  I think these structures handle array binding.  If
TDS_COLUMN_DATA::bind.type indicates array binding, then
TDS_COLUMN_DATA::data.pb points to a buffer that is in fact an array of
TDS_PBCB structures.  TDS_COLUMN_DATA::data.cb retains its definition, too:
it is the size of the buffer, not the count of elements.  

6.  Every buffer and every size is unsigned.  I think this will reduce
casts.  

7.  I wasn't sure what to do with column_text_sqlgetdatapos and vicinity.  

8.  See zerocopy comment.  My idea is that the client allocates a buffer
into which libtds may place incoming wire data.  Should the buffer be too
small, libtds mustn't fail; it must allocate a buffer, receive the data, and
indicate to the client what happened.  This way, good behavior is rewarded
and bad behavior is accomodated.  

Many thanks for your time and interest.  

--jkl

typedef struct _TDS_PBCB
{
        TDS_INT cb;
        TDS_UCHAR       *pb;
} TDS_PBCB;

typedef struct _TDS_PS
{
        TDS_TINYINT     precision;
        TDS_TINYINT     scale;
} TDS_PS;

/**
 * Metadata about regular and compute columns 
 */ 
typedef struct _TDS_COLUMN_META
{
        TDS_PBCB        label;  // NEW: "select name as label from table"
        TDS_PBCB        name;   // TDS_TINYINT column_namelen;
                                // TDS_CHAR column_name[256];
        TDS_UINT        status;// TDS_INT column_flags
                                // Do the bit fields duplicate this status? 
                                // unsigned int column_nullable:1;
                                // unsigned int column_writeable:1;
                                // unsigned int column_identity:1;
                                // unsigned int column_unicodedata:1;
                                // unsigned int column_output:1;
        TDS_UINT        user_type;      // TDS_INT column_usertype;
        TDS_TINYINT     data_type;      // TDS_SMALLINT column_type;
        TDS_TINYINT     wire_type;      // TDS_SMALLINT column_type_save;
        TDS_TINYINT     length_size;    // TDS_TINYINT column_varint_size;
        TDS_UINT        length; // TDS_INT column_size;
        TDS_PS          ps;             // TDS_TINYINT column_prec;
                                        // TDS_TINYINT column_scale;
        TDS_PBCB        tds5_locale_info
        TDS8_COLLATION tds8_collation;  // TDS_UCHAR column_collation[5];
        TDS_PBCB        blob_class_id;

        struct _compute_meta
        {
                TDS_USMALLINT   id;
                TDS_TINYINT     type;   // TDS_TINYINT  column_operator;
                TDS_SMALLINT    operand;        // TDS_SMALLINT
column_operand;
                                                
        } compute_operator;
} TDS_COLUMN_META;

typedef enum  { regular_row, compute_row, paramater_row } TDS_ROW_TYPE;

/**
 * Metadata about regular and compute rows 
 */ 
typedef struct _TDS_ROW_META
{
        TDS_ROW_TYPE  row_type; /* regular or compute (or parameter?) */
        TDS_USMALLINT   compute_id;
        TDS_UINT        ncolumns;
        TDS_PBCB        order_by_columns;       // NEW 
        TDS_PBCB        compute_by_columns;     // TDS_TINYINT *bycolumns;
                                                // TDS_SMALLINT  by_cols;
} TDS_ROW_META;

/**
 * Data about regular and compute columns, and binding
 */
typedef struct _TDS_COLUMN_DATA
{
        TDS_PBCB        catalog
        TDS_PBCB        schema
        TDS_PBCB        table   // TDS_TINYINT table_namelen;
                                // TDS_CHAR table_name[256];
        struct _bind
        {       TDS_USMALLINT type;     // TDS_SMALLINT column_bindtype;
                TDS_USMALLINT format;// TDS_SMALLINT column_bindfmt;
                TDS_UINT  length;       // TDS_UINT column_bindlen;
        } bind;
        /* 
         * To promote zerocopy, the client provides a buffer, "data".  
         * If data.cb is large enough, fine, set pdata to &data.
         * Else allocate a new PBCB for pdata, and use it.  
         * 
         * The client can free pdata "if (pdata != &data)".
         */
        TDS_PBCB  data;
        TDS_PBCB  *pdata        // TDS_CHAR *column_nullbind
                                // TDS_CHAR *column_varaddr;
                                // TDS_CHAR *column_lenbind;
                                // TDS_INT column_textpos;
                                // TDS_INT column_text_sqlgetdatapos;
} TDS_COLUMN_DATA;

/**
 * Data for regular and compute columns
 */ 
typedef struct _TDS_COLUMN
{
        TDS_COLUMN_META meta;
        TDS_COLUMN_DATA data;
} TDS_COLUMN;

/**
 * Data about regular and compute rows 
 */ 
typedef struct _TDS_ROW
{
        TDS_INT         irow;
        TDS_ROW_META            meta;
        TDS_COLUMN              *pdata;
        // meta.ncolumns tells count of data pointers.  
} TDS_ROW;



The information contained in this transmission may contain privileged and
confidential information and is intended only for the use of the person(s)
named above.  If you are not the intended recipient, or an employee or agent
responsible for delivering this message to the intended recipient, any
review, dissemination, distribution or duplication of this communication is
strictly prohibited. If you are not the intended recipient, please contact
the sender immediately by reply e-mail and destroy all copies of the
original message. Please note that we do not accept account orders and/or
instructions by e-mail, and therefore will not be responsible for carrying
out such orders and/or instructions.