EAGAIN on a blocking socket?

Per Jessen

2 Mar 2006 2 Mar '06

13:29

OK, for the benefit of others listening in, this is what I've briefly described to Steve and Anders on suse-linux-e - who both said "it's impossible" - which I agree with. Reality however seems to be of a different opinion. I'm writing a simple, multi-threaded SMTP proxy. New connections are received/accepted, packed into a unit-of-work, then queued for one of X worker-threads to pick up. Each UOW has the socket returned by the accept call. I refer to this as the "left side". When a UOW is picked up for processing, the thread will connect to the outbound side (the "right side"), and then start processing. Here's roughly what happens: (code attached) check that the socket is still valid - the client could have gone away whilst the UOW was queued. connect to right side (create socket etc.) create a copy of the socket - new one is called right2. The original is set nonblocking. Now create streams, one on each socket. Similar thing for the left side. The main code (not attached) now uses select() on the left and right inbound, nonblocking sockets, and when e.g. the left side has some data for us,: if ( FD_ISSET(left,&readfds) ) { closed=1; while( NULL!=fgets( buffer, sizeof(buffer), left_in ) ) { closed=0; while( EOF==(rc=fputs( buffer, right_out )) ) { if ( errno!=EAGAIN ) break; log_error( "[%02u.%04u] %d.2: unable to write to right; error %d: %s\n", c->id, work->id, stage, errno, strerror(errno) ); sleep(1); } if ( EOF==rc ) { stage=99; break; } } fflush( right_out ); // if the left side closed the connection if ( closed ) { log_error( "right side closed??\n"); stage=99; } } /Per Jessen, Zürich

Attachments:

ibwd_wrk_stage0.c (text/x-objcsrc — 3.4 KB)

Show replies by date

Anders Johansson

2 Mar 2 Mar

17:57

New subject: [suse-programming-e] EAGAIN on a blocking socket?

Per Jessen wrote:

...

right2=dup(right); fcntl( right, F_SETFL, O_NONBLOCK );

File status flags are shared between dup()licated file descriptors, so this operation sets both to O_NONBLOCK. Could this answer your problem?

Per Jessen

18:49

New subject: [suse-programming-e] EAGAIN on a blocking socket?

Anders Johansson wrote:

...

Per Jessen wrote:

...
right2=dup(right); fcntl( right, F_SETFL, O_NONBLOCK );

File status flags are shared between dup()licated file descriptors, so this operation sets both to O_NONBLOCK. Could this answer your problem?

Yeah, that would explain it, except when I do fcntl( GET_FL ) to read the setting of right2 it doesn't report nonblocking? Are you sure the flags are shared? They should be _copied_ with dup(), but the a change of flags on one filedescriptor is copied to another, that doesn't sound right - to me anyway. /Per Jessen, Zürich

Anders Johansson

18:53

New subject: [suse-programming-e] EAGAIN on a blocking socket?

Per Jessen wrote:

...

Anders Johansson wrote:

...
Per Jessen wrote:

...
right2=dup(right); fcntl( right, F_SETFL, O_NONBLOCK );

File status flags are shared between dup()licated file descriptors, so this operation sets both to O_NONBLOCK. Could this answer your problem?

Yeah, that would explain it, except when I do fcntl( GET_FL ) to read the setting of right2 it doesn't report nonblocking? Are you sure the flags are shared? They should be _copied_ with dup(), but the a change of flags on one filedescriptor is copied to another, that doesn't sound right - to me anyway.

No, see "man dup". The file status flags are shared between the two, so a change to one means both are affected. The file descriptor flags are not. At what point do you do the GET_FL? I don't see it in the code you sent

Per Jessen

19:07

New subject: [suse-programming-e] EAGAIN on a blocking socket?

Anders Johansson wrote:

...

No, see "man dup". The file status flags are shared between the two, so a change to one means both are affected. The file descriptor flags are not.

Hmm. Any suggestion on how to do nonblocking reads and blokcing writes on one socket?

...

At what point do you do the GET_FL? I don't see it in the code you sent

I think I removed it again. I basically did a check just before fputs(). Something like this: fcntl( fd, F_GETFL, &flag ); if ( flag & O_NONBLOCK ) log_error("nonblocking when it shudnt be"); Anyway, yours and Steves comments, particularly about fputs() and fprintf() not setting errno in case of error, has made me think. /Per Jessen, Zürich

Anders Johansson

19:20

New subject: [suse-programming-e] EAGAIN on a blocking socket?

Per Jessen wrote:

...

Hmm. Any suggestion on how to do nonblocking reads and blokcing writes on one socket?

A few ideas spring to mind, such as changing the flag right before the write, or implementing your own write that blocks inside your own program even if the stream doesn't.

...

...
At what point do you do the GET_FL? I don't see it in the code you sent

I think I removed it again. I basically did a check just before fputs(). Something like this:

fcntl( fd, F_GETFL, &flag ); if ( flag & O_NONBLOCK ) log_error("nonblocking when it shudnt be");

and you're saying you never saw this error message in the log? That is strange, that would definitely be worth a bug report. If you could write a minimal program that exhibits the problem, and include the glibc version and kernel you use, I don't think it would take the developers long to fix

Per Jessen

19:44

New subject: [suse-programming-e] EAGAIN on a blocking socket?

Anders Johansson wrote:

...

...
I think I removed it again. I basically did a check just before fputs(). Something like this:

fcntl( fd, F_GETFL, &flag ); if ( flag & O_NONBLOCK ) log_error("nonblocking when it shudnt be");

and you're saying you never saw this error message in the log? That is strange, that would definitely be worth a bug report. If you could write a minimal program that exhibits the problem, and include the glibc version and kernel you use, I don't think it would take the developers long to fix

I'll now be double- and triple-checking it, but I'm fairly certain that's the behaviour I saw. Thanks for your time guys - I knew a third and fourth pair of eyes would come in handy. /Per Jessen, Zürich

Anders Johansson

20:03

New subject: [suse-programming-e] EAGAIN on a blocking socket?

Per Jessen wrote:

...

Anders Johansson wrote:

...
No, see "man dup". The file status flags are shared between the two, so a change to one means both are affected. The file descriptor flags are not.

Hmm. Any suggestion on how to do nonblocking reads and blokcing writes on one socket?

...
At what point do you do the GET_FL? I don't see it in the code you sent

I think I removed it again. I basically did a check just before fputs(). Something like this:

fcntl( fd, F_GETFL, &flag ); if ( flag & O_NONBLOCK ) log_error("nonblocking when it shudnt be");

by the way, this isn't a valid call. flag doesn't get modified by this. Correct would be flag=fcntl(fd, F_GETFL);

Per Jessen

20:19

New subject: [suse-programming-e] EAGAIN on a blocking socket?

Anders Johansson wrote:

...

...
fcntl( fd, F_GETFL, &flag ); if ( flag & O_NONBLOCK ) log_error("nonblocking when it shudnt be");

by the way, this isn't a valid call. flag doesn't get modified by this. Correct would be

flag=fcntl(fd, F_GETFL);

Ahhh ... ok, that does it. Takker. I guess the easiest is to go nonblocking throughout - except why can't I rely on fputs()/fprintf() sticking something useful into errno? It seems reasonable that an fputs() could return -1 and errno==EAGAIN. Sorry, just thinking out loud. Maybe I'll do what you suggested - change blocking whenver I need to - just sounds a bit kludgy. /Per Jessen, Zürich

Steve Graegert

19:18

New subject: [suse-programming-e] EAGAIN on a blocking socket?

On 3/2/06, Per Jessen wrote:

...

Anders Johansson wrote:

...
Per Jessen wrote:

...
right2=dup(right); fcntl( right, F_SETFL, O_NONBLOCK );

File status flags are shared between dup()licated file descriptors, so this operation sets both to O_NONBLOCK. Could this answer your problem?

Yeah, that would explain it, except when I do fcntl( GET_FL ) to read the setting of right2 it doesn't report nonblocking? Are you sure the flags are shared?

Yes, both descriptors share the same file table entry. For example: fd1 = open(..., O_NONBLOCK); fd2 = dup(fd1); fd2 shares the same file table entry as fd1 and therefore the same file status flags. Only close-on-exec is unset. \Steve

Steve Graegert

18:04

New subject: [suse-programming-e] EAGAIN on a blocking socket?

On 3/2/06, Per Jessen wrote:

...

OK, for the benefit of others listening in, this is what I've briefly described to Steve and Anders on suse-linux-e - who both said "it's impossible" - which I agree with. Reality however seems to be of a different opinion.

<snip> Per, did not have time to take a closer look at your code, but I think that fputs(3) itself is not the culprit since it does not necessarily set errno when EOF is encountered (write(2) on the other hand, which is called on behalf of fputs(2), does). Instead you're duplicating the FDs before associating it with a stream and the call to fcntl(2) call causes the other (new) FD to be put into non-blocking (O_NONBLOCK) as well since the data structures in the kernel are shared among them. \Steve

Jerry Feldman

18:25

New subject: [suse-programming-e] EAGAIN on a blocking socket?

On Thursday 02 March 2006 1:04 pm, Steve Graegert wrote:

...

On 3/2/06, Per Jessen wrote:

...
OK, for the benefit of others listening in, this is what I've briefly described to Steve and Anders on suse-linux-e - who both said "it's impossible" - which I agree with. Reality however seems to be of a different opinion.

...

did not have time to take a closer look at your code, but I think that fputs(3) itself is not the culprit since it does not necessarily set errno when EOF is encountered (write(2) on the other hand, which is called on behalf of fputs(2), does).

I agree with Steve, Both fgets(3) and fputs(3) are buffered streams, but you are using select(2). I would avoid the use of streams even though an C stream is attached to a valid file descriptor (or socket). I would use either write(2), read(2), send(2), or recv(2) directly. -- Jerry Feldman Boston Linux and Unix user group http://www.blu.org PGP key id:C5061EA9 PGP Key fingerprint:053C 73EC 3AC1 5C44 3E14 9245 FB00 3ED5 C506 1EA9

Anders Johansson

18:54

New subject: [suse-programming-e] EAGAIN on a blocking socket?

Jerry Feldman wrote:

...

I would use either write(2), read(2), send(2), or recv(2) directly.

Why?

Jerry Feldman

20:56

New subject: [suse-programming-e] EAGAIN on a blocking socket?

...

Jerry Feldman wrote:

...
I would use either write(2), read(2), send(2), or recv(2) directly.

Why? The issue is one of buffering, timing and control, especially when doing network programming.

On Thursday 02 March 2006 1:54 pm, Anders Johansson wrote: part of the problem is that fputs(3) writes the string to a user space buffer. This buffer does not get flushed until you either fill the buffer or flush the buffer using fflush(3). In this case, you may not detect I/O problems until well after the fact. It is not wrong to use C streams, but you will see these delays in things, like receiving the EOF on fputs(3). You certainly can handle it, but you are at a different level of abstraction. -- Jerry Feldman Boston Linux and Unix user group http://www.blu.org PGP key id:C5061EA9 PGP Key fingerprint:053C 73EC 3AC1 5C44 3E14 9245 FB00 3ED5 C506 1EA9

Per Jessen

19:00

New subject: [suse-programming-e] EAGAIN on a blocking socket?

Jerry Feldman wrote:

...

Both fgets(3) and fputs(3) are buffered streams, but you are using select(2).

But I'm only using select for the read status, not the write. Besides, using select() in combination with reading from a buffered stream should be perfectly fine.

...

I would avoid the use of streams even though an C stream is attached to a valid file descriptor (or socket). I would use either write(2), read(2), send(2), or recv(2) directly.

Well, yeah - except it just complicates things unnecessarily. /Per Jessen, Zürich

Per Jessen

18:56

New subject: [suse-programming-e] EAGAIN on a blocking socket?

Steve Graegert wrote:

...

did not have time to take a closer look at your code, but I think that fputs(3) itself is not the culprit since it does not necessarily set errno when EOF is encountered

Hmmm, interesting - good point. Does anyone know how to go about examining the error-condition in such a case then? where fputs() returns EOF. I also notice on re-reading the man-page for fprintf that it doesn't return EOF in error-situations, but a negative value.

...

Instead you're duplicating the FDs before associating it with a stream and the call to fcntl(2) call causes the other (new) FD to be put into non-blocking (O_NONBLOCK) as well since the data structures in the kernel are shared among them.

That's what Anders said too, yet I'm pretty certain (I'll obviously have to check again) I checked the the flags of the second fd to make sure it was blocking. But that leads me to this question - how to I have one socket where I want reads to be nonblocking and writes to block? Or am I asking for something impossible? /Per Jessen, Zürich

Steve Graegert

19:25

New subject: [suse-programming-e] EAGAIN on a blocking socket?

On 3/2/06, Per Jessen wrote:

...

Steve Graegert wrote:

...
did not have time to take a closer look at your code, but I think that fputs(3) itself is not the culprit since it does not necessarily set errno when EOF is encountered

Hmmm, interesting - good point. Does anyone know how to go about examining the error-condition in such a case then? where fputs() returns EOF. I also notice on re-reading the man-page for fprintf that it doesn't return EOF in error-situations, but a negative value.

Everytime EOF is returned you should (a) check if the EOF indicator has been set (see feof(3)) and (b) check for errors with ferror(3).

...

...
Instead you're duplicating the FDs before associating it with a stream and the call to fcntl(2) call causes the other (new) FD to be put into non-blocking (O_NONBLOCK) as well since the data structures in the kernel are shared among them.

That's what Anders said too, yet I'm pretty certain (I'll obviously have to check again) I checked the the flags of the second fd to make sure it was blocking.

But that leads me to this question - how to I have one socket where I want reads to be nonblocking and writes to block? Or am I asking for something impossible?

Maybe send(2) and recv(2) would help. Both allow for individual I/O operations to be done in either blocking or non-blocking mode. \Steve

Per Jessen

19:42

New subject: [suse-programming-e] EAGAIN on a blocking socket?

Steve Graegert wrote:

...

Maybe send(2) and recv(2) would help. Both allow for individual I/O operations to be done in either blocking or non-blocking mode.

Yeah, I was just hoping to avoid having to do my own buffering etc. When you're doing SMTP, using streams is pretty optimal. /Per Jessen, Zürich

6623

Age (days ago)

6623

Last active (days ago)

List overview

Download

17 comments

4 participants

participants (4)

Anders Johansson
Jerry Feldman
Per Jessen
Steve Graegert