Skip to Content.
Sympa Menu

freetds - Re: [freetds] Connection Pool crashing problems?

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: Brian Bruns <brian.bruns AT gmail.com>
  • To: craigs <s.craig AT andronics.com>, freetds AT lists.ibiblio.org
  • Cc:
  • Subject: Re: [freetds] Connection Pool crashing problems?
  • Date: Tue, 14 Dec 2004 08:29:32 -0500

Tests went fine. I'm convinced we have no problems with normal
queries. There were some issues with client disconnects and such.
The other thing to look for would be RPCs/placeholder stuff and more
esoteric features like that. What did you say your frontend was?

Also if we could look at the last few queries from the log before a
crash that might be helpful. The new version removed some of the
excessive messages to stderr, so the log should be more intelligible.

Brian


On Tue, 14 Dec 2004 13:20:42 +0000, craigs <s.craig AT andronics.com> wrote:
> Brian,
>
> Im flat out at the minute doing system admin so cant get a chance to
> test the new stuff, how did your tests go with ms sql
>
> Shaun.
>
> Brian Bruns wrote:
>
> >Checked in some more changes. When a client exits in the middle of a
> >query it can leave the member in a confused state, so now we just
> >close it to be safe.
> >
> >So please try the latest from CVS and let me know.
> >
> >Brian
> >
> >On Mon, 13 Dec 2004 15:02:00 +0000, craigs <s.craig AT andronics.com> wrote:
> >
> >
> >>Brian,
> >>
> >>the only error message we get from our clients is that we cannot connect
> >>to database server, this happens in dbopen() which makes sense seen as
> >>the pool is down, your tests sound encouraging so maybe if i get a
> >>chance later today i will write a test program in a safe environment to
> >>hammer the pool, will run in gdb with tdsdump on and see how we go.
> >>
> >>Shaun.
> >>
> >>
> >>
> >>Brian Bruns wrote:
> >>
> >>
> >>
> >>>Ok, I'm running now. This is what I did to test:
> >>>
> >>>1. Built a query file with 'select * from <table>' for every table in
> >>>the database (about a 500 meg database) Good mix of small and large
> >>>tables.
> >>>2. Set up the pool for 2 connections
> >>>3. Spun off tsql to run the queries in parallel, averaging about 150
> >>>client connections to the pool at any one time.
> >>>4. Ran for half an hour so far and holding.
> >>>5. Ran clients up to 1000 and now letting it run for a bit.
> >>>
> >>>One thing I did notice is the need for a better scheduling algorithm
> >>>under heavy load to prevent starvation.
> >>>
> >>>Other than that, it seems we are having an issue with some sort of
> >>>abnormal case. I did notice the pool go down when a "cancel" was
> >>>issued by the client. I'm looking into that, but does that ring any
> >>>bells?
> >>>
> >>>Brian
> >>>
> >>>On Mon, 13 Dec 2004 13:28:22 +0000, craigs <s.craig AT andronics.com> wrote:
> >>>
> >>>
> >>>
> >>>
> >>>>Brian,
> >>>>
> >>>>we only had the new pool running for a short time and had to revert to
> >>>>an older version pretty quick when we seen there was problems so we
> >>>>never got a chance to get a stack trace, its very dangerous for us to
> >>>>put it back on just to get the trace as we have alot of input to our
> >>>>system. Let me know how the tests go using SQL Server, thanks for your
> >>>>time.
> >>>>
> >>>>Shaun.
> >>>>
> >>>>
> >>>>
> >>>>Brian Bruns wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>Shaun,
> >>>>>
> >>>>>I ran it continuously for about 3 hours using a simulated workload.
> >>>>>This was against a Sybase 12.5.2 database, so that may be the issue.
> >>>>>I can try it from here against SQL Server, but didn't have that access
> >>>>>over the weekend. I'll let you know if anything turns up, but so far
> >>>>>I'm not seeing it.
> >>>>>
> >>>>>BTW, I made two distinct fixes, one of which I was almost certain was
> >>>>>what you were seeing based on your description...oh well, we'll keep
> >>>>>plugging away. Was the stack trace the same on the new code?
> >>>>>
> >>>>>Brian
> >>>>>
> >>>>>
> >>>>>On Mon, 13 Dec 2004 12:23:01 +0000, craigs <s.craig AT andronics.com>
> >>>>>wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>hi Brian,
> >>>>>>
> >>>>>>got the latest version off CVS, put it on our server and things didnt
> >>>>>>go
> >>>>>>to well! will test it more ourselfs but its really hard for us to
> >>>>>>duplicate live data but will try, did everything go ok when you
> >>>>>>tested it?
> >>>>>>
> >>>>>>Shaun.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>Brian Bruns wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>I just applied a one liner (and your patch as well) to CVS. Please
> >>>>>>>give it a try and let me know if it works. As a bonus, this should
> >>>>>>>make it much easier to add TDS7/8 support to the pool.
> >>>>>>>
> >>>>>>>Cheers,
> >>>>>>>
> >>>>>>>Brian
> >>>>>>>
> >>>>>>>On Fri, 10 Dec 2004 11:24:59 +0000, craigs <s.craig AT andronics.com>
> >>>>>>>wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>thats one hell of a film! thanks for looking into it, next year we
> >>>>>>>>have
> >>>>>>>>predicted that we will average over 3000 clients hitting the pool
> >>>>>>>>every
> >>>>>>>>hour so it would be good if we could track it down sooner rather
> >>>>>>>>than later.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>Brian Bruns wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>The good: I was able to get it to crash on the same line.
> >>>>>>>>>The bad: Looks like a buffer overflow
> >>>>>>>>>The ugly: This means it originated elsewhere in the code and is
> >>>>>>>>>therefore hard to track down.
> >>>>>>>>>
> >>>>>>>>>Still plugging away at it, i'll let you know what i find.
> >>>>>>>>>
> >>>>>>>>>On Fri, 10 Dec 2004 10:30:24 +0000, craigs <s.craig AT andronics.com>
> >>>>>>>>>wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>here is a detailed list of what we are running! we have a suse
> >>>>>>>>>>9.2 linux
> >>>>>>>>>>server, using tds version 4.2 for connection pooling, snapshot
> >>>>>>>>>>freetds-0.64.dev.20041207, client programs written in C using
> >>>>>>>>>>db-lib. the
> >>>>>>>>>>pool goes down at least a few times every hour, we are using 3 max
> >>>>>>>>>>connections and receiving approx 1200 clients an hour and each
> >>>>>>>>>>client does
> >>>>>>>>>>at least 2 querys and an insert with few doing 1 or 2 more.
> >>>>>>>>>>
> >>>>>>>>>>we have written a process watcher that watchs the connection pool
> >>>>>>>>>>and
> >>>>>>>>>>restarts it if it crashes, before we added a patch it would take
> >>>>>>>>>>the
> >>>>>>>>>>connection pool nearly a minute to come back up, it wouldnt
> >>>>>>>>>>release the
> >>>>>>>>>>socket, or the socket had to wait to time out, now the pool comes
> >>>>>>>>>>back up
> >>>>>>>>>>again within a few seconds.
> >>>>>>>>>>
> >>>>>>>>>>Brian Bruns wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>Craig, Wanna send along the patch for that? I'll merge it. I've
> >>>>>>>>>>been
> >>>>>>>>>>planning on spending some time on this, and may get around to it
> >>>>>>>>>>this
> >>>>>>>>>>weekend. What is your server/version? I'll try to do some load
> >>>>>>>>>>testing and
> >>>>>>>>>>see if I can reproduce your error locally, and failing that scan
> >>>>>>>>>>the code
> >>>>>>>>>>for something. Brian On Thu, 09 Dec 2004 09:17:32 +0000, craigs
> >>>>>>>>>><s.craig AT andronics.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>Frediano Ziglio wrote:
> >>>>>>>>>>Il mer, 2004-12-08 alle 13:52, craigs ha scritto:
> >>>>>>>>>>Brian Bruns wrote: Program received signal SIGSEGV, Segmentation
> >>>>>>>>>>fault.
> >>>>>>>>>>0x0804b5b0 in pool_process_users (pool=0x804d008, fds=0xbfffdbd0)
> >>>>>>>>>>at
> >>>>>>>>>>user.c:131 131 if (FD_ISSET(puser->tds->s, fds)) { is this any
> >>>>>>>>>>help? Shaun.
> >>>>>>>>>>Yes, probably puser->tds == NULL or puser->tds->s < 0. In other
> >>>>>>>>>>words
> >>>>>>>>>>connection closed freddy77 Any ideas why the connection is
> >>>>>>>>>>closing, would it
> >>>>>>>>>>be on my C clients side or maybe something to do with ms sql.
> >>>>>>>>>>when it does
> >>>>>>>>>>crash it takes nearly a minute to come up again which can cause
> >>>>>>>>>>problems for
> >>>>>>>>>>ourselfs, we have added a line in pool_main_loop which releases
> >>>>>>>>>>the socket
> >>>>>>>>>>as soon it crashes, our up time again has been greatly reduced
> >>>>>>>>>>because of
> >>>>>>>>>>this line, would this be of any use to anyone?
> >>>>>>>>>>_______________________________________________ FreeTDS mailing
> >>>>>>>>>>list
> >>>>>>>>>>FreeTDS AT lists.ibiblio.org
> >>>>>>>>>>http://lists.ibiblio.org/mailman/listinfo/freetds
> >>>>>>>>>>_______________________________________________ FreeTDS mailing
> >>>>>>>>>>list
> >>>>>>>>>>FreeTDS AT lists.ibiblio.org
> >>>>>>>>>>http://lists.ibiblio.org/mailman/listinfo/freetds
> >>>>>>>>>>_______________________________________________ FreeTDS mailing
> >>>>>>>>>>list
> >>>>>>>>>>FreeTDS AT lists.ibiblio.org
> >>>>>>>>>>http://lists.ibiblio.org/mailman/listinfo/freetds
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>




Archive powered by MHonArc 2.6.24.

Top of Page