Skip to Content.
Sympa Menu

freetds - Re: [freetds] Optimizing memory allocation when using dynamic SQL

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: Sebastien FLAESCH <sf AT 4js.com>
  • To: FreeTDS Development Group <freetds AT lists.ibiblio.org>
  • Subject: Re: [freetds] Optimizing memory allocation when using dynamic SQL
  • Date: Mon, 21 Jan 2008 10:25:33 +0100

Thank you Frediano for these tips...

Actually we have not much places where to use static memory, because
in our database driver all must be dynamic.

As you pointed out, SQL statements can be very long some time...

Also, we have no idea how many result set columns or or how many
SQL parameters will be used, and we can't guess the data types either.

You know all of that...

I was just wondering if modern systems / c runtime libs do now
handle small pieces calls of malloc()/free() in a more efficient
way as older systems...

I did not find much on the net about malloc/free optimization.

We have wrappers to allocate/free dynamic memory, so it's not a big
deal to write our own optimization algorithm and allocate blocks
of 512 bytes or whatever.

The question is: Does this still make sense on recent systems?
I mean, everybody has this problem, and I wonder if malloc/free
is not already optimized, to avoid the real system calls for
small pieces of memory...

About memory allocation on the stack, we used to have some places
where alloca() was used, but we removed it because (man alloca):

"The alloca() function returns a pointer to the beginning of the allo‐
cated space. If the allocation causes stack overflow, program behavior
is undefined.

The inlined code often consists of a single instruction adjusting the
stack pointer, and does not check for stack overflow. Thus, there is
no NULL error return.
"

Not quite reliable...

Cheers,
Seb

ZIGLIO, Frediano, VF-IT wrote:
Dear all,

Frediano Ziglio wrote:
http://msdn2.microsoft.com/en-us/library/ms709306(VS.85).aspx
Interesting, I knew about alignment issues, but from an optimization point of
view I have always asked myself if doing several malloc() of little pieces of
memory is really an issue...

We used to have our own mem allocation wrapper to optimize this in our VM but
we have removed it in latest version to just use malloc() as is... allocating
any size of memory...

I really wonder if C runtime library providers (on any platform) did not yet
optimize malloc() to allocate large blocks of memory (I mean, if you allocate
4 bytes, does it really just allocate 4 bytes or does it pick 4b in a memory pool
or whatever and just increment an internal offset)?

This is a real issue for use, as we have different sort of database interfaces
and need to allocate buffers for dynamic SQL result sets...

Note also that such kind of memory optimizations is IMHO not significant if
you compare with database communication... am I wrong?

Any comment/advice is welcome!

Thanks a lot.

;-)
Seb

Mmmm... how many words have I ?? :)

It's very difficult to say what's the best way. There are many things to
consider
1- memory fragmentation
2- thread considerations
3- speed
4- security
5- data spent

1. allocation/deallocationg in different order cause memory
fragmentation which if very fragmented can lead to slow programs cause
you use more cache and more memory pages
2. multithread can cause slower allocation or excessive fragmentation.
Allocating on stack don't have these problems
3. malloc is slower than automatic. A single malloc instead of multiple
(think at a struct with pointers and a struct with embedded arrays) is
faster
4. stack corruption is worst than heap one. Also having dynamic
allocation on stack is difficult to manage (stack is limited and if you
allocate too much memory program core! linux kernel don't use much stack
for stability problems)
5. well.. a malloc(1) occupy more that a char data[1] cause you have to
store
- the pointer (4 or 8 bytes)
- heap structure (to be able to free)
- data have to be aligned (usually 16 or 32 byte but for 1 byte this can
be avoided)
However you have to consider that embedding static arrays is a big
limit! For instance sql statements can be very long but usually are
quite short. In this case a dynamic allocated buffer is the better way.

I have changed your patch changing
static const struct {
const char *value;
unsigned char to_return;
} boolean_values[] = {
{ "yes", 1 },
{ "no", 0 },
{ "on", 1 },
{ "off", 0 },
{ "true", 1 },
{ "false", 0 }
};

to
static const struct {
char value[7];
unsigned char to_return;
} boolean_values[] = {
{ "yes", 1 },
{ "no", 0 },
{ "on", 1 },
{ "off", 0 },
{ "true", 1 },
{ "false", 0 }
};

and

static const char *odbc_param_Servername = "Servername";

to
static const char odbc_param_Servername[] = "Servername";

you may ask how much is the gain. Taking into account that
- you don't have a pointer to these buffers
- you don't have relocation records for these pointers
- you don't have code to deference that pointers
The gain can be even 20 bytes for a single structure!
Oh... someone could also ask why value is 7 byte long instead of 6
(which is sufficient)... alignment!!

freddy77
_______________________________________________
FreeTDS mailing list
FreeTDS AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/freetds






Archive powered by MHonArc 2.6.24.

Top of Page