Since I've only been lurking on this list until now, but I'd like to become actively involved, let me start with a quick introduction... My name is Deven Corzine, and I've been seriously programming computers for over 20 years. (Since I'm just under 31 years old, that's a bug chunk of my lifetime!) I've been using Unix, free software and the Internet for at over 13 years now. I haven't released any free software of significance, mostly for circumstantial reasons. I'm looking to start doing so now, but finding the time is always difficult. One project I've had in the back of my mind for a while now is a CD-based backup system. All the existing tools for doing this seem clumsy, poorly integrated and very limited. Solutions seem to generally revolve around using mkisofs and cdrecord to burn a CD, containing files or archives. While this can work as an ad-hoc solution, I want something better. My ideal goal is a system that uses variable-length packet writing and has the CD-recording functionality, the filesystem, and the backup application highly integrated. I'd also like a mountable write-once (WORM) filesystem for CD-R's or CD-RW's, and possibly a rewritable one (maybe needing fixed packets) for CD-RW's only. (I don't know if the filesystem would depend on the integrated backup application, or the other way around, or neither...) I want to be able to burn daily incremental backups without wasting 30 Meg daily for multisession. (Unless it's necessitated by other factors; multisession could be a fallback mode.) I want the backup system to be able to restore the original filesystem as perfectly as it can, possibly requiring custom metadata storage to do so. I want to be able to burn to CD-R disks as easily as CD-RW. I want the burned CD's to be as useful and accessible as possible; I'd like the final disk (when full) to be "closed out" to ISO-9660 format, with both Rock Ridge and Joliet extensions to allow VERY easy access to any given file stored on the CD even when the backup software is unavailable for "perfect" recovery. (The directly-accessed files might or might not be compressed, probably at the user's option.) It should be possible to "close out" the disk to ISO 9660 format more than once, paying the inevitable 30 Meg multisession cost to do so -- this would be done if packet writing isn't available, or to access a non-full CD on another system for data transfer. Using variable-length packets could ensure that any given file doesn't have to span packets, which would greatly improve interoperability with older ISO filesystem implementations. Before the disk is "closed out" to ISO 9660 format, it might be in UDF format (which is designed to be compatible with ISO 9660), or it might be some custom format; I don't know yet. There is a strong argument for UDF, since there is independent demand for a UDF filesystem, and an existing implementation in progress. However, it might add more complexity, so it's a possible tradeoff. Although it's tempting to use the version numbers in the ISO filesystem, there is an interoperability question, since virtually every ISO 9660 file ever seen has a ";1" version on it -- some ISO filesystem implementations might choke on multiple version numbers and/or version numbers other than ";1" -- the best solution might be to use modified filenames for multiple versions. (This could be optional.) Another option, of course, is to make the ISO filesystem contain one or more archives in a fairly standard format (.tar, .tar.gz, .zip) for each backup time; in some ways this might be cleaner (and would probably save some space), compared to top-level files with Rock Ridge/Joliet extensions; this could again be optional. Another likely feature of the high-level backup application would be the ability to buffer backup data in one or more locations on local or remote hard drives until it comes time to burn a CD. This could be waiting on user intervention to load the backup CD, to use fewer multisession writes if packet writing is unavailable, or even to burn all CD's as full disks. Obviously, the project I'm contemplating has the potential to be large, complex and quite difficult to implement the way I'd like to see it done. I'm seriously considering at least *starting* this project, although I'm not making any promises at all -- I may change my mind, choose higher priorities, get tired or bored with this, or quit if it's working well enough for my immediate purposes. I might even stick with the generic mkisofs/cdrecord solutions. I just don't know yet, and I'm not making any specific commitment to implement this; don't hold your breath waiting for it. On the other hand, I haven't seen any sign that anyone is doing anything much like this, and it's something I'd like to have -- that might turn out to be motivation enough to work on it. (We'll see.) While most of this could be done without packet writing, it really becomes much less interesting or valuable; the efficiency to be gain from packet writing (enabling daily incremental backups *and* good interoperability) is what really makes this project most interesting to me. Otherwise, I would probably come up with a much less integrated solution for simplicity... Anyway, that explains my interest in packet writing, and why I signed up for this mailing list. This context may supply background information to make it more clear why I'm asking some of the questions below. On with the reply! On Fri, 8 Sep 2000, Jens Axboe wrote:
On Mon, Aug 07 2000, Alexander V. Trotsay wrote:
Hi
Can I get packet-writing patches for kernel like 2.4.0-test5 or newest?
Not yet, this weekend there will be a 0.0.2c version against test7 however.
Is there any way to do packet writing without recompiling the kernel? Can it be done in userspace with the sg* SCSI (or ide-scsi) generic devices? The usefulness of packet-writing software will be limited if it requires a custom or brand-spanking-new kernel to use; a userspace fallback for older kernels would be VERY useful, if possible. The backup system I'd like to have would be useful for ANYONE with a CD burner, not just the few who are willing to run bleeding-edge kernels. Userspace code would also make it harder to crash or hang the system... Where can I find documentation on the SCSI-level commands for packet writing (fixed and variable, CD-R and CD-RW) and other CD-ROM commands, and is this information freely available? (By the way, why were the kdb patches merged with the packet-writing patches instead of in a separate file? I don't see much use for the patches if I don't bother to connect to the serial port as a console...)
And I have few questions:
1. When I read this list I see that formated disk must be 650Mb, but I have only 550. Why?
For fixed packet writing, space is wasted. So the capacity will be considerably less than 650MB, 550MB or so sounds ok:
Sep 8 12:46:38 bart kernel: pktcdvd: writer sr0 sucessfully registered Sep 8 12:47:15 bart kernel: pktcdvd: inserted media is CD-RW Sep 8 12:47:15 bart kernel: pktcdvd: Fixed packets, 32 blocks / packet, Mode-1 disc Sep 8 12:47:15 bart kernel: pktcdvd: speed (R/W) 6/4 Sep 8 12:47:15 bart kernel: pktcdvd: 551616KB available on disc
I'm particularly interested in variable-length packet writing; what does it take to do it? UDF is designed to work with fixed or variable packets, and on CD-RW or CD-R media; must we narrow our options?
2. When I try wrote files all work Ok, but when I try to delete file my system locks up and only unclean shutdown help me. Files do not be deleted.
You may have bumped into a deadlock that I have fixed. 0.0.2b and earlier version were all affected. So I'll encourage you to watch the list for the announcement.
I haven't started experimenting with the code yet; should I simply wait for the 0.0.2c release? Have you considered setting up CVS access?
3. Can I format and write CDR disk (on my CD-RW drive). I know than I can't delete information written once but I wan't incrimentally write files to disk
Not yet.
Is the formatting process writing all the fixed packets in advance, with the intent of replacing them? Obviously this wouldn't work for CD-R media, but is it even necessary in the first place? If UDF can work for CD-R's, shouldn't it be possible to access a CD-RW the same way?
4. Why format procedure so long. When I use cdrecord, I have option -blank=fast, may be use same methods for disk formating.
Don't confuse format and blank. cdrwtool can do fast blanking as well, just use cdrwtool -b fast. Once the disc has been used with the -q quick setup, when you need to start from scracth just do a fast blank and manually mkudf the device afterwards.
Is there such a thing as packet blanking, or do you just rely on overwriting the packets with new data? What do I need to do to get more involved in this packet-writing project? Deven