FreeTDS Development Group

[Brian Bruns <brian AT bruns.com>]

My 0.02 USD. (Sorry if it's a bit incoherent, fighting a cold, and overly 
tired)

The fundamental unit of the TDS protocol is the token stream.  Aside
from a few oddballs, like the login packet other client side things,
everything is a token stream.  So, one part of libtds' job is simply
to unpack the token streams from the outer netlib packets and
reassemble them if they span.

Easy-peasy.  Except the ROW token stream requires knowledge of what
result set it belongs to because we need to calculate lengths based on
the columns.  Everything else is either fixed length or calculable
internally.  The minimal amount of knowledge that is necessary to
compute the row token stream length is a jump table to variable field
length entries.  Not great, but not awful but it requires the
otherwise pure networking code to maintain a little bit of state.

If we had a lower rung of libtds, (let's make up a name and call it,
oh I don't know, "netlib" maybe?), then the upper rung could called
tds_get_next_token_stream() and be handed a fully self contained blob
of memory that it could read randomly instead of parsed sequentially.
Now the upper rung is dealing only with whole token streams and apply
state machine rules on them, have separate state machines for
different protocol versions and everyones life gets easier.

MARSy type stuff would be easier (can have a separate state machine
for TDS+MARS), SSL/TDS isn't such a kludge, and deferred/asynchronous
I/O gets a whole lot easier.  This also makes something like TDS over
RPC or named pipes much more approachable.

I think everyone agrees the byte-by-byte read from the wire idiom that
libtds is written in is a disaster.

Is this along the lines of what you were thinking Jim?

On Thu, Aug 18, 2011 at 6:01 PM,  <jklowden AT schemamania.org> wrote:
> I would like to re-engineer libtds from the ground up as a finite
> state machine.
>
> Fundamentally, libtds is a TDS parser.  Packets arrive in
> pre-determined sequences.  Packets, even variable-length fields
> within packets, are well defined.  I don't know if TDS is a *regular*
> language, but I suspect so.  It's definitely a parseable one.
>
> Why do this?  libtds is quite ad hoc; it reflects its origins as a
> discovery platform for understanding the TDS protocol.  Consequently,
> it's hard to understand and hasn't attracted a new developer in five
> years.  Clear principles and architecture would benefit the library
> and the project.
>
> Technically, I would put forward these goals:
>
> 0.  Reduce TDS description to tabular form, to be used as input the
> parser-generator.  Can also be used to generate structs for each
> packet type (see #5).  Can also be used to improve TDS documentation.
>
> 1.  More predictable and rational flow of control, essential for
> thread safety.
>
> 2.  Clear factoring of protocol versions.
>
> 3.  Better RAII, clearer memory ownership.
>
> 4.  Removal of iconv from TDS layer.  libtds should be transparent.
> Charset issues are binding issues; client libraries need to support
> access to the raw data.  Cf. dbdata() and nvarchar columns.
>
> 5.  Packet-based access to the network.  Data exchange between the
> client libraries and libtds should be by packet.
>
>    I do not mean that each client library should chase every
> packet 1:1 on the wire.  In particular, libtds has to scoop up the
> DONE_IN_PROC and compute packets after e.g. the final dbnextrow().
>
> 6.  The same state machine can inform a server implementation.
>
> There are non-technical goals, too:
>
> 7.  Attract more developers by adopting a classic, computer-science
> approach.  I venture to say most programmers never get to work on a
> finite-state machine.
>
> 8.  Make FreeTDS portable to other RDBMSs.  Cf.
> http://freedb.schemamania.org.
>
> 9.  Make client library maintenance easier.  Consider this from
> dbnextrow():
>
> const int mask = 
> TDS_STOPAT_ROWFMT|TDS_RETURN_DONE|TDS_RETURN_ROW|TDS_RETURN_COMPUTE;
> ...
> switch (tds_process_tokens(tds, &res_type, NULL, mask)) {
>
> What would your mother say if she saw you writing C like that?
>
> Asynchronous calls comport with a by-packet interface design.  To
> implement dbpoll()/dbdataready(), db-lib would call something like:
>
>        int tds_next_packet_type(TDS*);
>
> which would return the TDS marker if a packet is available, or -1 if
> not.  The blocking calls would be:
>
>        int tds_get_XXX(TDS*, struct XXX *);
>
> I once thought C++ was the answer to libtds's problems, and I could
> still be convinced to use C++.  But I think the real answer is a
> formal approach.
>
> I've been reluctant to undertake all this myself, though.  I see no
> point in writing a new libtds unless the client libraries are updated
> to use it, and I don't want to take on more than db-lib.  If I *did*
> write libtds2 *and* change db-lib to use it, and that's all that
> happened, I'd have split the project, probably making things worse,
> not better.  For the sake of the project, I really need agreement
> before I can start.
>
> --jkl
> _______________________________________________
> FreeTDS mailing list
> FreeTDS AT lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/freetds
>