having a daemon monitor its own shared libraries?
I've got a daemon that uses a library that is occasionally updated - not very frequently, perhaps once per month. What I'm thinking about is having the daemon monitor this particular library for changes using inotify. I could perhaps do an automatic restart or something like that - or maybe there is a way of reloading/refreshing a library whilst running? Another question is - is there a way of listing libraries in use from within my daemon? I could just hardcode the name, but ... /Per Jessen, Zürich
On Tuesday 08 August 2006 04:41, Per Jessen wrote:
I've got a daemon that uses a library that is occasionally updated - not very frequently, perhaps once per month. What I'm thinking about is having the daemon monitor this particular library for changes using inotify. I could perhaps do an automatic restart or something like that - or maybe there is a way of reloading/refreshing a library whilst running?
Another question is - is there a way of listing libraries in use from within my daemon? I could just hardcode the name, but ...
I assume you're discussing a shared library linked by the system at run time, not an object library linked at compile time. My understanding is that if the update is overwriting the library in use you would see the process core dump, because the file is memory mapped to the process. I'm also assuming you're not seeing the process core dump, because the update renames or unlinks the share library file in use before installing the new library. Assuming all the assumptions are correct ;-) ... 1) simple(er) Monitor the library with a shell script. If the file changes, the kill and restart the process. OR 2) (Another assumption -- You control the process source code and can modify and build it.) Instead of letting the system dynamic link the library change the process to link the library itself [ using dlopen() ] and monitor changes to the file [ using stat() ]. When the file changes then the process would dlclose() the library and then reopen it and then the process would need to reattach itself to the library's symbols with dlsym(). I've been brainstorming a C library at work (for Solaris) during my "copious spare time" to do option 2 more or less automatically for a process, so that we can do updates in production without bringing down an application. But, spare time what it is, I only got around to drawing squiggles on Post-It notes. Thanks, Ken Jennings
Ken Jennings wrote:
I assume you're discussing a shared library linked by the system at run time, not an object library linked at compile time.
Correct.
My understanding is that if the update is overwriting the library in use you would see the process core dump, because the file is memory mapped to the process.
That doesn't happen. I can update the library without the process noticing at all.
I'm also assuming you're not seeing the process core dump, because the update renames or unlinks the share library file in use before installing the new library.
Ah, possible. But all I do is an scp from a development system into the target system - if I attempt to update a running binary, I get "text file busy", but not for a library.
2) (Another assumption -- You control the process source code and can modify and build it.)
Yes, I can.
Instead of letting the system dynamic link the library change the process to link the library itself [ using dlopen() ] and monitor changes> to the file [ using stat() ]. When the file changes then the process would dlclose() the library and then reopen it and then the process would need to reattach itself to the library's symbols with dlsym().
I like that - presumably I would need to change something in my build such that I don't need to link to this library?
I've been brainstorming a C library at work (for Solaris) during my "copious spare time" to do option 2 more or less automatically for a process, so that we can do updates in production without bringing down an application.
I'm not too concerned with the downtime, it's more of a management thing. I have this daemon running on N geographically separate systems, and it would be nice to have the restart done automatically. Thanks for all your input Ken - much appreciated. /Per Jessen, Zürich
Per, On Tuesday 08 August 2006 05:38, Per Jessen wrote:
...
My understanding is that if the update is overwriting the library in use you would see the process core dump, because the file is memory mapped to the process.
That doesn't happen. I can update the library without the process noticing at all.
That's because the software development tools don't overwrite existing files. They move / rename or delete those files and create new ones. This prevents failure of either the tools or the programs using them. You might want to look into "fam" the File Alteration Monitor, which now has kernel support and has low overhead compared to earlier, user-based implementations.
...
/Per Jessen, Zürich
Randall Schulz
On Tue, 8 Aug 2006, Randall R Schulz wrote:
That doesn't happen. I can update the library without the process noticing at all.
That's because the software development tools don't overwrite existing files. They move / rename or delete those files and create new ones. This prevents failure of either the tools or the programs using them.
Ummm... As I recall the Unix filesystem does this: If a file is opened by one or more processes and the file gets deleted, the file is still maintained on the system until the last process using it closes the file or dies. At that point the space is free'd up for use again. Until that time it is just hanging around, invisible to new processes but still usable by those that had it open. This "trick" is used often to create temporary files that other processes can't touch: Create it, open it, then delete it but keep it open. That's probably why the daemon isn't noticing the change: It's still using the original file it was dynamically linked to when it was started up. -- Curt, WE7U. APRS Client Comparisons: http://www.eskimo.com/~archer "Lotto: A tax on people who are bad at math." -- unknown "Windows: Microsoft's tax on computer illiterates." -- WE7U "The world DOES revolve around me: I picked the coordinate system!"
Curt, On Tuesday 08 August 2006 05:50, Curt, WE7U wrote:
On Tue, 8 Aug 2006, Randall R Schulz wrote:
That doesn't happen. I can update the library without the process noticing at all.
That's because the software development tools don't overwrite existing files. They move / rename or delete those files and create new ones. This prevents failure of either the tools or the programs using them.
Ummm... As I recall the Unix filesystem does this: If a file is opened by one or more processes and the file gets deleted, the file is still maintained on the system until the last process using it closes the file or dies. At that point the space is free'd up for use again. Until that time it is just hanging around, invisible to new processes but still usable by those that had it open.
That _is_ how the kernel behaves. But the tools could be written to overwrite existing files, but it would just reduce their utility.
This "trick" is used often to create temporary files that other processes can't touch: Create it, open it, then delete it but keep it open.
That's probably why the daemon isn't noticing the change: It's still using the original file it was dynamically linked to when it was started up.
Quite likely.
-- Curt
RRS
Randall R Schulz wrote:
My understanding is that if the update is overwriting the library in use you would see the process core dump, because the file is memory mapped to the process.
That doesn't happen. I can update the library without the process noticing at all.
That's because the software development tools don't overwrite existing files. They move / rename or delete those files and create new ones. This prevents failure of either the tools or the programs using them.
That sounds like a nice explanation, but "scp" does it differently for libraries and executables. If I scp a library, the library is just replaced, if I scp an executable I get "text file is busy". Also - would you call scp a "software development tool" ? :-) (I use scp to copy the libraries from a development system to the target system).
You might want to look into "fam" the File Alteration Monitor, which now has kernel support and has low overhead compared to earlier, user-based implementations.
That's what I mentioned earlier - the code is already using inotify(), so adding another one is no big deal. Then I could just trigger an execv(myself) off that. But that still leaves the question of - which file to monitor without hardcoding the name? Is there some library equivalent of "ldd" that I could call from within my code? /Per Jessen, Zürich
Randall R Schulz wrote:
That's because the software development tools don't overwrite existing files. They move / rename or delete those files and create new ones. This prevents failure of either the tools or the programs using them.
Just a quick update in case someone's interested - I've set up the daemon to use inotify() and when I use 'scp' to copy in a new library from my development system, I see the following events: (not monitoring for IN_ACCESS and IN_ATTRIB) IN_OPEN IN_MODIFY . . . . IN_CLOSE_WRITE So 'scp' certainly doesn't do any move, rename or delete, but plainly overwrites the library file. I guess shared libraries are just read in by the loader, and then forgotten about. Also, by default inotify will follow symlinks by default, which is quite handy. The actual library in this case is "/usr/lib/libclamav.so.1.0.17", but of course it has a few symlinks, so I just monitor "/usr/lib/libclamav.so" - I would still prefer not having to hardcode "/usr/lib", but I can live with it :-) /Per Jessen, Zürich
On Wednesday 09 August 2006 3:14 am, Per Jessen wrote:
Just a quick update in case someone's interested - I've set up the daemon to use inotify() and when I use 'scp' to copy in a new library from my development system, I see the following events: (not monitoring for IN_ACCESS and IN_ATTRIB)
IN_OPEN IN_MODIFY . . . . IN_CLOSE_WRITE
So 'scp' certainly doesn't do any move, rename or delete, but plainly overwrites the library file. I guess shared libraries are just read in by the loader, and then forgotten about.
Also, by default inotify will follow symlinks by default, which is quite handy. The actual library in this case is "/usr/lib/libclamav.so.1.0.17", but of course it has a few symlinks, so I just monitor "/usr/lib/libclamav.so" - I would still prefer not having to hardcode "/usr/lib", but I can live with it :-) What happens in the case where you change the location of a symbol in the shared library.
--
Jerry Feldman
Jerry Feldman wrote:
Also, by default inotify will follow symlinks by default, which is quite handy. The actual library in this case is "/usr/lib/libclamav.so.1.0.17", but of course it has a few symlinks, so I just monitor "/usr/lib/libclamav.so" - I would still prefer not having to hardcode "/usr/lib", but I can live with it :-)
What happens in the case where you change the location of a symbol in the shared library.
I'm probably not quite understanding your question, but any change in the library leads to a new library being copied to the target system. When my daemon detects a change to the library file, it'll close up shop, do some minimal clean-up and restart using exec(argv[0],argv). /Per Jessen, Zürich
participants (5)
-
Curt, WE7U -
Jerry Feldman -
Ken Jennings -
Per Jessen -
Randall R Schulz