Good points - response inline... On Fri, Apr 24, 2009 at 9:49 PM, Ned T. Crigler <crigler@gmail.com> wrote:
You may have trouble with operating systems that emulate threads in userspace.
At this point I'm focusing on two operating systems only - linux and freebsd. I'd like to support Solaris as well, but the vast majority of our base is in the first two categories.
This design uses too many threads that need to talk to each other. For example, when there is new data to be read on a socket, this is what will happen: <snip> Each step will require a context switch to transfer control to a different thread. This will be especially bad if not all threads are on the same processor/core in a SMP machine. In order to finish processing activity on a socket, we will be bouncing back and forth between threads before we are done.
We could have an accept thread which does nothing else other than wait for activity on all the listening sockets and accept connections. When a new connection arrives, it can assign a connection to a worker thread.
Each worker thread then waits for activity on only the connections that it has been assigned, and handles all the work itself.
I like this idea - however, it does introduce a slightly complicated behavior - if you have "ownership" of a client by one thread, then messages triggered by other clients (which is 90% of things) gets a bit more tricky. What do you think about combining the two methods - ownership of clients for reading and polling purposes, but still having a separate writing thread (set)? This way you're not actually waking up the somewhat busy read/poll thread for that client on every write queue insert. As follows: 1. Client is accepted by the accept thread, and assigned to a reading/polling/parsing thread. 2. A message is sent by the client to a channel, for which there are 2 clients involved. 3. A shared buffer is created by the parsing thread, and inserted into the other two client queues. 4. The write thread is then woken up, checks the current write state for the clients, and sends if possible. This way the primary poll/read/parse thread is free to continue without actually having to manage write queues. In your originally suggested method, the context switch is extremely likely anyway, since there is a good chance that a target socket is "owned" by a different read thread. This way at least the read/parse thread can focus on reading and parsing, rather than trying to do everything. Thoughts? -epi