Skip to Content.
Sympa Menu

freetds - [freetds] basic data structures

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: "Lowden, James K" <LowdenJK AT bernstein.com>
  • To: "'TDS Development Group'" <freetds AT lists.ibiblio.org>
  • Subject: [freetds] basic data structures
  • Date: Thu, 6 Mar 2003 17:20:15 -0500

All,

As we add functionality to libtds, some of its spindlier and more
provisional legs are becoming unsteady. I think the fundamental data
structures -- that hold data and meta data for result sets, including
compute rows and paramaters -- should be restructured along the lines of the
attached.

If you're interested, please review the attached design and tell me what you
think. If you think its good but infeasible, let's see if we can come up
with a way to take advantage of some of it, on the road to a better
tomorrow. If you think it's pointless, don't worry, we have too many
pointful things already in TODO.

I have these goals:

1. 16-bit clean: no strlen(), use counted buffers.
2. separate data from meta data.
3. support zerocopy.
4. base structures on protocol packets where appropriate.
5. cleaner names via nested structures.
6. better consistency.
7. good structures lead to good code.
8. good code engenders participation and satisfaction.

In the end, I want to make libtds more accessible, to make it easier to add
to the client libraries, and easier to spot goofs in libtds itself.

There are basically four kinds of data, for purposes of this discussion:

1. Column metadata
2. Row metadata
3. Binding metadata
4. Data

(#4 being what the user cares about.)

These can be combined in many (!4 ?) ways. At present, they're munged into
TDSCOLINFO, except for some (inconsistent) parts that are in TDSRESULTINFO.
The thing is unwieldy; it lacks even the concept of a row.

I propose to nest things in much the way we've done, but with structures to
delineate the parts:

row ::
row meta
column array ::
column meta
column ::
binding meta
data

That is, a row consists metadata and columns, and each column consists of
metadata (from the server), bind metadata (from the client), and data. The
latter two are nested for convenient communication with the client.

Note there is practically no state information, no notion of a "current" row
or column or size or offset. The row metadata tell you how many columns
there are. You merely index into that array; the column metadata tell you
the size of the column, the data.cb tells you the size of the data.

The metadata arrive first, and are provisioned along with their
corresponding buffers. The data arrive in rows; each field gets put in its
own little buffer, and the whole shebang is passed back to the client.

I also considered another design, one that more closely reflects the
protocol, where the data and metadata are sharply segregated:

row ::
meta ::
row meta
column meta array
column array ::
binding meta
data

but that requires parallel arrays and more pointers, both of which I prefer
to avoid.

Notes on the attached C structures.

1. The term "pbcb" stands for "pointer to buffer count of bytes", and I
didn't invent it. It's a little like a Pascal string. Since each buffer
needs a length, it's helpful to combine the two tersely. It also puts the
pointer at the very end of nested structures, which is very convenient to
use.

2. I've used my naming conventions while trying to respect the FreeTDS
style. I don't insist on them if you don't like them.

3. Every field is accounted for, else I made an error. There are some new
ones (order by, schema name, column label). Might as well make room;
they're defined in the protocol.

4. I haven't tried to account for multibyte character sets. The protocol
and the APIs deal mainly in byte counts. Where character counts are
distinct, I think it will be up to the client library to compute them.

5. I think these structures handle array binding. If
TDS_COLUMN_DATA::bind.type indicates array binding, then
TDS_COLUMN_DATA::data.pb points to a buffer that is in fact an array of
TDS_PBCB structures. TDS_COLUMN_DATA::data.cb retains its definition, too:
it is the size of the buffer, not the count of elements.

6. Every buffer and every size is unsigned. I think this will reduce
casts.

7. I wasn't sure what to do with column_text_sqlgetdatapos and vicinity.

8. See zerocopy comment. My idea is that the client allocates a buffer
into which libtds may place incoming wire data. Should the buffer be too
small, libtds mustn't fail; it must allocate a buffer, receive the data, and
indicate to the client what happened. This way, good behavior is rewarded
and bad behavior is accomodated.

Many thanks for your time and interest.

--jkl

typedef struct _TDS_PBCB
{
TDS_INT cb;
TDS_UCHAR *pb;
} TDS_PBCB;

typedef struct _TDS_PS
{
TDS_TINYINT precision;
TDS_TINYINT scale;
} TDS_PS;

/**
* Metadata about regular and compute columns
*/
typedef struct _TDS_COLUMN_META
{
TDS_PBCB label; // NEW: "select name as label from table"
TDS_PBCB name; // TDS_TINYINT column_namelen;
// TDS_CHAR column_name[256];
TDS_UINT status;// TDS_INT column_flags
// Do the bit fields duplicate this status?
// unsigned int column_nullable:1;
// unsigned int column_writeable:1;
// unsigned int column_identity:1;
// unsigned int column_unicodedata:1;
// unsigned int column_output:1;
TDS_UINT user_type; // TDS_INT column_usertype;
TDS_TINYINT data_type; // TDS_SMALLINT column_type;
TDS_TINYINT wire_type; // TDS_SMALLINT column_type_save;
TDS_TINYINT length_size; // TDS_TINYINT column_varint_size;
TDS_UINT length; // TDS_INT column_size;
TDS_PS ps; // TDS_TINYINT column_prec;
// TDS_TINYINT column_scale;
TDS_PBCB tds5_locale_info
TDS8_COLLATION tds8_collation; // TDS_UCHAR column_collation[5];
TDS_PBCB blob_class_id;

struct _compute_meta
{
TDS_USMALLINT id;
TDS_TINYINT type; // TDS_TINYINT column_operator;
TDS_SMALLINT operand; // TDS_SMALLINT
column_operand;

} compute_operator;
} TDS_COLUMN_META;

typedef enum { regular_row, compute_row, paramater_row } TDS_ROW_TYPE;

/**
* Metadata about regular and compute rows
*/
typedef struct _TDS_ROW_META
{
TDS_ROW_TYPE row_type; /* regular or compute (or parameter?) */
TDS_USMALLINT compute_id;
TDS_UINT ncolumns;
TDS_PBCB order_by_columns; // NEW
TDS_PBCB compute_by_columns; // TDS_TINYINT *bycolumns;
// TDS_SMALLINT by_cols;
} TDS_ROW_META;

/**
* Data about regular and compute columns, and binding
*/
typedef struct _TDS_COLUMN_DATA
{
TDS_PBCB catalog
TDS_PBCB schema
TDS_PBCB table // TDS_TINYINT table_namelen;
// TDS_CHAR table_name[256];
struct _bind
{ TDS_USMALLINT type; // TDS_SMALLINT column_bindtype;
TDS_USMALLINT format;// TDS_SMALLINT column_bindfmt;
TDS_UINT length; // TDS_UINT column_bindlen;
} bind;
/*
* To promote zerocopy, the client provides a buffer, "data".
* If data.cb is large enough, fine, set pdata to &data.
* Else allocate a new PBCB for pdata, and use it.
*
* The client can free pdata "if (pdata != &data)".
*/
TDS_PBCB data;
TDS_PBCB *pdata // TDS_CHAR *column_nullbind
// TDS_CHAR *column_varaddr;
// TDS_CHAR *column_lenbind;
// TDS_INT column_textpos;
// TDS_INT column_text_sqlgetdatapos;
} TDS_COLUMN_DATA;

/**
* Data for regular and compute columns
*/
typedef struct _TDS_COLUMN
{
TDS_COLUMN_META meta;
TDS_COLUMN_DATA data;
} TDS_COLUMN;

/**
* Data about regular and compute rows
*/
typedef struct _TDS_ROW
{
TDS_INT irow;
TDS_ROW_META meta;
TDS_COLUMN *pdata;
// meta.ncolumns tells count of data pointers.
} TDS_ROW;



The information contained in this transmission may contain privileged and
confidential information and is intended only for the use of the person(s)
named above. If you are not the intended recipient, or an employee or agent
responsible for delivering this message to the intended recipient, any
review, dissemination, distribution or duplication of this communication is
strictly prohibited. If you are not the intended recipient, please contact
the sender immediately by reply e-mail and destroy all copies of the
original message. Please note that we do not accept account orders and/or
instructions by e-mail, and therefore will not be responsible for carrying
out such orders and/or instructions.




  • [freetds] basic data structures, Lowden, James K, 03/06/2003

Archive powered by MHonArc 2.6.24.

Top of Page