Non-RW media; WORM hopes
I apologize if I'm missing the bus here, but for some reason I recently came awake and noticed packet-writing for what it could be. In reading the history and archives I can find, it seems most people are concentrated on using RW media for temporary storage - "floppy replacement" kind of stuff. My interest is in using it with R media as a poor-man's WORM drive substitute for storing log files, but I'm not finding much reference to anyone doing so other than claims that it "can be done." I've tried a superset of all the instructions I can find, but thus far have succeeded in nothing more than making a couple of coasters - dvd+rw-format seems counter-intuitive on write-once media. This list seems to have really slowed down in the past couple of years, but I hope there are still enough knowledgeable people watching this list that can answer my [hopefully] elementary questions: - It has been stated that CD-R (and presumably CD+R) media is not supported for use; is that due to lack of software or that the physical medium itself cannot be abused in that manner? - Has anyone actually ever successfully used DVD-R (or DVD+R) media to store 'immutable' data? If so, can you list the precise steps, drive, and media (if relevant) you used to do so? The ability to actually [physically] delete things from RW media defeats my intended purpose. - Does pktcdvd gracefully handle concurrent, incremental file writes (read: log files), i.e., can blocks be interleaved or must an entire file's worth of blocks be allocated at once? Note - this question is due more to my lack of understanding of the UDF format than anything. Whatever the answers, I'm certainly willing to learn & get my hands dirty trying to get this working, I'd just like a few things cleared up so I know what I'm getting into. Thanks for your time! RB -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
I apologize if I'm missing the bus here, but for some reason I recently came awake and noticed packet-writing for what it could be. In reading the history and archives I can find, it seems most people are concentrated on using RW media for temporary storage - "floppy replacement" kind of stuff. My interest is in using it with R media as a poor-man's WORM drive substitute for storing log files, but I'm not finding much reference to anyone doing so other than claims that it "can be done." I've tried a superset of all the instructions I can find, but thus far have succeeded in nothing more than making a couple of coasters - dvd+rw-format seems counter-intuitive on write-once media.
This list seems to have really slowed down in the past couple of years, but I hope there are still enough knowledgeable people watching this list that can answer my [hopefully] elementary questions:
- It has been stated that CD-R (and presumably CD+R) media is not supported for use; is that due to lack of software or that the physical medium itself cannot be abused in that manner?
- Has anyone actually ever successfully used DVD-R (or DVD+R) media to store 'immutable' data? If so, can you list the precise steps, drive, and media (if relevant) you used to do so? The ability to actually [physically] delete things from RW media defeats my intended purpose.
- Does pktcdvd gracefully handle concurrent, incremental file writes (read: log files), i.e., can blocks be interleaved or must an entire file's worth of blocks be allocated at once? Note - this question is due more to my lack of understanding of the UDF format than anything.
Whatever the answers, I'm certainly willing to learn & get my hands dirty trying to get this working, I'd just like a few things cleared up so I know what I'm getting into.
Thanks for your time!
RB There was a MS Windows program called "Direct CD" that provided "letter access" access to a CD-RW media. I do not believe it worked with CD-R, as
On Thu May 31 2007 07:38, RB wrote: the media would be getting smaller and smaller as files are deleted, and with the media costing $0.15 for CD-R and about $0.24 for CD-RW, it is not worth it anyway. In the mean time re: "Worm", I would seriously consider DVD-RAM. This is a $50 drive with $8.00 media, that reads/writes as if it were a hard disk. The media is perported to be of better quality than the other CD DVD media. Some drives use a cartridge, the newer ones do not. You can format (via MS Windows only, I believe) as FAT32, UDF 1.5 or UDF 2.0. UDF 2.0 has larger "clusters" so it is more efficient with very large files (multimedia). I use FAT32, not because I use MS Windows, as I use Suse Linux, but because I can use it easily in Dosemu to run DOS programs (Foxpro mainly) under Suse. Hope this input helps, -- John R. Sowden AMERICAN SENTRY SYSTEMS, INC. Residential & Commercial Alarm Service UL Listed Central Station Serving the San Francisco Bay Area Since 1967 mail@americansentry.net www.americansentry.net -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
Unfortunately, just like other RW media DVD-RAM won't work for this purpose - they all allow complete deletion/overwrite of files & inodes, which is unacceptable for WORM (write-once, read-many). The drawback of the media "getting smaller and smaller as files are deleted" would be considered a feature, especially considering many people *never* want certain files (esp. logs) deleted, regardless of who attempts to do so. Imagine, if you will, the following scenario: 1. Mount DVD-R via pktcdvd under /var/log/WORM 2. Direct syslog daemon to write a copy of all log files to /var/log/WORM/somefilename 3. System is compromised, attacker attempts to [clumsily] erase their tracks by removing log files 4. Logs are recoverable by adding '-o unhide,undelete' when mounting the DVD from another system during post-mortem. Of course, a simple cron job similar to tape-change reminders would have to be set up to aid in managing the space available, but that would be the easiest part. If someone tried to be more subtle and just modify part of a file, it would be terribly obvious as well, since an inode out of the middle of a previously linear file would suddenly leapfrog forward. I'm sure there are additional use scenarios, but this is the specific one I'm trying to address and am bit surprised that no one seems to have brought it up before. I can see & understand everyone's fascination with rewritable media as cheap temporary storage, but a readily available, cheap storage medium that is _physically_ immutable short of destruction has equally fascinating applications, especially if we were able to write to it incrementally. RB -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
On 2007.06.06 04:57, John R. Sowden wrote:
On Thu May 31 2007 07:38, RB wrote:
I apologize if I'm missing the bus here, but for some reason I recently came awake and noticed packet-writing for what it could be. In reading the history and archives I can find, it seems most people are concentrated on using RW media for temporary storage - "floppy replacement" kind of stuff. My interest is in using it with R media
[snip] John, Packet writing to -R media is not yet supported in Linux. Regards, Roy Bamford -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
On 6/6/07, Roy Bamford
Packet writing to -R media is not yet supported in Linux.
After looking at all the dates on the docs I can find, that seems to be the latest consensus. However, having browsed through pktcdvd.c, it would seem there are several acknowledgments of/facilities for -R media, so again I ask - what lacks? Additions to the pktcdvd driver, userspace tools, or something else entirely? RB -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
RB wrote:
After looking at all the dates on the docs I can find, that seems to be the latest consensus. However, having browsed through pktcdvd.c, it would seem there are several acknowledgments of/facilities for -R media, so again I ask - what lacks? Additions to the pktcdvd driver, userspace tools, or something else entirely?
IIRC, what is lacking is filesystem support for WORM media. In otherwords, the UDF filesystem driver tries to rewrite blocks on the disc. For your purposes, I would suggest that you don't use a filesystem on the disk and just write the raw log file out, or maybe a tar data stream. If you pipe it though dd to reblock you should be able to write it directly to the cd-r without pktcdvd. -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
Testing your the
On 6/7/07, Phillip Susi
For your purposes, I would suggest that you don't use a filesystem on the disk and just write the raw log file out, or maybe a tar data stream. If you pipe it though dd to reblock you should be able to write it directly to the cd-r without pktcdvd.
Unfortunately, that doesn't seem to work in 2.6.21.3. In testing, 'dd if=/var/log/messages of=/dev/hdc bs=2048' results in dd's immediate failure with the message: 'dd: opening `/dev/hdc': Read-only file system', regardless of what the permissions are on the device. Attempting to pass the data through 'cdrecord -packet -noclose' has some interesting results (i.e., next writable addr. increments as remaining writable decrements), but I can't read anything past the first 2048 bytes using dd. Physically, the track is obviously growing, but if I can't read that it's pretty useless. Through pretty exhaustive testing, it would seem the only method one can [right now] packet-write to -R media is via 'cdrecord -packet', but I have yet to read data written in that manner. RB -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
RB wrote:
Unfortunately, that doesn't seem to work in 2.6.21.3. In testing, 'dd if=/var/log/messages of=/dev/hdc bs=2048' results in dd's immediate failure with the message: 'dd: opening `/dev/hdc': Read-only file system', regardless of what the permissions are on the device. Attempting to pass the data through 'cdrecord -packet -noclose' has some interesting results (i.e., next writable addr. increments as remaining writable decrements), but I can't read anything past the first 2048 bytes using dd. Physically, the track is obviously growing, but if I can't read that it's pretty useless.
Through pretty exhaustive testing, it would seem the only method one can [right now] packet-write to -R media is via 'cdrecord -packet', but I have yet to read data written in that manner.
RB
The media must be formatted for packet writing mode and you must use a block size equal to the packet size ( 32k by default ). -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
On 6/12/07, Phillip Susi
The media must be formatted for packet writing mode and you must use a block size equal to the packet size ( 32k by default ).
I wish I weren't so dense. I thought the default packet size for optical media was 2k - where do you get 32k? Regardless, I can't get any of the UDF formatting tools to do anything anyway 'mkudffs -r 0x0150 --media-type=worm /dev/hdc 359845' fails silently with EROFS and EBADF; 'cdrwtool -d /dev/hdd -u 359845' runs a bit, then fails with 'bad sector' IDE errors. Is the comment about formatting the media an esoteric one or do you know something about one of the tools I'm unaware of? -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
On 2007.06.12 23:07, RB wrote:
On 6/12/07, Phillip Susi
wrote: The media must be formatted for packet writing mode and you must use a block size equal to the packet size ( 32k by default ).
I wish I weren't so dense. I thought the default packet size for optical media was 2k - where do you get 32k?
Regardless, I can't get any of the UDF formatting tools to do anything anyway 'mkudffs -r 0x0150 --media-type=worm /dev/hdc 359845' fails silently with EROFS and EBADF; 'cdrwtool -d /dev/hdd -u 359845' runs a bit, then fails with 'bad sector' IDE errors.
Is the comment about formatting the media an esoteric one or do you know something about one of the tools I'm unaware of? -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
RB, The packet/block size on CDs is indeed 2k. On DVDs, its 32kB, however, the logical block size is faked by the drive to be 2k to get standards compliance. That means a DVD drive does read/modifiy/writes 16 times for every 2k packet you write. This reuslts in a huge slowdown. Writing 32kB blocks is much faster. For CD-RW media you must format the media before your mount it. A format involves writing the entire surface of the disk. I haven't tried DVD-RW media. With DVD+RW media, packet writing is not required. DVD+RW supports proper random access and can support the filesystem of your choice. However, UDF is still a good choice. With the limited write life of DVD +RW, journaling and defualt -o rw mounts must be avoided. Its all at http://fy.chalmers.se/~appro/linux/DVD+RW/ Regards, Roy Bamford -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
RB wrote:
On 6/12/07, Phillip Susi
wrote: The media must be formatted for packet writing mode and you must use a block size equal to the packet size ( 32k by default ).
I wish I weren't so dense. I thought the default packet size for optical media was 2k - where do you get 32k?
No, the sector size is 2k. The default packet size that cdrwtool formats the track with is 32k. I uploaded some patches to Ubuntu documenting the hidden parameter to change the packet size, and fixing a bug that prevented sizes other than 32k from working. If you download the source package from packages.ubuntu.com, you will find these patches in the debian/dpatch directory. Larger packets allow you to use more of the disk, smaller ones require that you write less at a time.
Regardless, I can't get any of the UDF formatting tools to do anything anyway 'mkudffs -r 0x0150 --media-type=worm /dev/hdc 359845' fails silently with EROFS and EBADF; 'cdrwtool -d /dev/hdd -u 359845' runs a bit, then fails with 'bad sector' IDE errors.
Is the comment about formatting the media an esoteric one or do you know something about one of the tools I'm unaware of?
Again, you can not put a udf filesystem on it. You need to use cdrwtool to only format the track, which I believe was the -m option. Once the track is formatted, you should be able to write directly to it with dd using 32kb blocks. -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
Again, you can not put a udf filesystem on it. You need to use cdrwtool to only format the track, which I believe was the -m option.
I wish I had good news... Reading through the source of cdrwtool (udftools-1.0.0b3) and using strace, I'm able to follow the execution of '-m' up to the point that it actually tries to send the packet containing the format to the CD (cdrwtool.c:424), which fails immediately with a return value of EIO. I thought it was the fact that cdrwtool.h tries to reference HZ in it's timeouts, but hardcoding those to sane values did nothing for me. In three different drives from three different manufacturers, I get the following with virgin Imation CD-R media: [root@tst ~] cdrwtool -d /dev/hdc -m 359846 using device /dev/hdc formatting 359846 blocks 1751KB internal buffer setting write speed to 12x wait_cmd: Input/output error Command failed: 04 17 00 00 00 00 00 00 00 00 00 00 - sense 05.30.06 format disc: Illegal seek Afterwards, using 'cdrwtool -i' the discs then [usually] show 295264 free blocks, as opposed to the prior 359846, but 'dd if=/dev/zero of=/dev/hdc bs=32k' gives EROFS and fails. I swear I see a 'dotted' burn just around the center of the media (inside the area usually burned during fixation), but the usually visible "ring o' death" I've seen after my 'cdrecord --packet' experiments is not evident. I've got ide-cd.ko and cdrom.ko compiled with all the debugging options turned up, but don't see anything in the kernel logs that give me any indication as to why it's failing. The sense failure is as cryptic as anything, and while I'm sure the 0x17 command is significant (0x10, or '1 << 4' seeming to be IMMED), the 0x07 portion seems like a magic number - I guess it came out of one of the rainbow books, but none of the source documents it. I'm learning kernel code as I go here, so somebody smack me if I'm being dumb. Any other ideas? RB -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
El Sábado, 16 de Junio de 2007, RB escribió:
Again, you can not put a udf filesystem on it. You need to use cdrwtool to only format the track, which I believe was the -m option.
I wish I had good news... Reading through the source of cdrwtool (udftools-1.0.0b3) and using strace, I'm able to follow the execution of '-m' up to the point that it actually tries to send the packet containing the format to the CD (cdrwtool.c:424), which fails immediately with a return value of EIO. I thought it was the fact that cdrwtool.h tries to reference HZ in it's timeouts, but hardcoding those to sane values did nothing for me. In three different drives from three different manufacturers, I get the following with virgin Imation CD-R media:
[root@tst ~] cdrwtool -d /dev/hdc -m 359846 using device /dev/hdc formatting 359846 blocks 1751KB internal buffer setting write speed to 12x wait_cmd: Input/output error Command failed: 04 17 00 00 00 00 00 00 00 00 00 00 - sense 05.30.06 format disc: Illegal seek
Afterwards, using 'cdrwtool -i' the discs then [usually] show 295264 free blocks, as opposed to the prior 359846, but 'dd if=/dev/zero of=/dev/hdc bs=32k' gives EROFS and fails. I swear I see a 'dotted' burn just around the center of the media (inside the area usually burned during fixation), but the usually visible "ring o' death" I've seen after my 'cdrecord --packet' experiments is not evident.
Since when you can write directly over a cd, without executing ATA commands?.
I've got ide-cd.ko and cdrom.ko compiled with all the debugging options turned up, but don't see anything in the kernel logs that give me any indication as to why it's failing. The sense failure is as cryptic as anything, and while I'm sure the 0x17 command is significant (0x10, or '1 << 4' seeming to be IMMED), the 0x07 portion seems like a magic number - I guess it came out of one of the rainbow books, but none of the source documents it. I'm learning kernel code as I go here, so somebody smack me if I'm being dumb. Any other ideas?
RB
-- Gustavo Guillermo Pérez Compunauta uLinux www.compunauta.com -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
Since when you can write directly over a cd, without executing ATA commands?.
That would be a question for the more knowledgeable (since I'm just following instructions here), but since this is going through ide-cd which 'inherits' from ide.h and cdrom.h, why on earth would a userspace program need to issue raw ATA commands? This is the very thing drivers are there for - abstract away the hardware implementation differences and present a consistent API. RB -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
On 2007.06.16 15:28, RB wrote: [snip]
I get the following with virgin Imation CD-R media:
[root@tst ~] cdrwtool -d /dev/hdc -m 359846 using device /dev/hdc formatting 359846 blocks
[snip]
RB -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
RB, There is no point in doing that to a CD-R, once you format it, its written and it can only be written once. You have to format as you go, with the data you want to write at the same time as the format. You should be using dev/pktcdvd too, not /dev/hdc. Regards, Roy Bamford -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
Roy Bamford wrote:
RB,
There is no point in doing that to a CD-R, once you format it, its written and it can only be written once.
I do not think this is correct. I believe that it is valid to format a CD-R media track for fixed length packet mode, which will record the packet headers, lead-ins, lead-outs, etc, but leave the actual data portions blank for later recording. I will have to dig up the MMC specs and double check.
You have to format as you go, with the data you want to write at the same time as the format.
You can't do this with packet mode. You format the track first, then you go back and write data to it. There is no way to record data while formatting.
You should be using dev/pktcdvd too, not /dev/hdc.
No, you don't want to use pktcdvd because it only supports cd-rw media. There is also no need as long as you have dd write only full length packets to the drive. -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
RB wrote:
I wish I had good news... Reading through the source of cdrwtool (udftools-1.0.0b3) and using strace, I'm able to follow the execution of '-m' up to the point that it actually tries to send the packet containing the format to the CD (cdrwtool.c:424), which fails immediately with a return value of EIO. I thought it was the fact that cdrwtool.h tries to reference HZ in it's timeouts, but hardcoding those to sane values did nothing for me. In three different drives from three different manufacturers, I get the following with virgin Imation CD-R media:
[root@tst ~] cdrwtool -d /dev/hdc -m 359846 using device /dev/hdc formatting 359846 blocks 1751KB internal buffer setting write speed to 12x wait_cmd: Input/output error Command failed: 04 17 00 00 00 00 00 00 00 00 00 00 - sense 05.30.06 format disc: Illegal seek
After refreshing my memory from the MMC standard, it looks like you can not use the FORMAT UNIT command on CD-R media, which is what cdrwtool is trying to do there. If I am reading the standard correctly however, you can issue a RESERVE TRACK command with the proper mode settings and then proceed to write to that track a packet at a time. I am not sure if it does it correctly or not, but cdrwtool has a -r switch to tell it to reserve a track. This may correctly set up the mode page and reserve the track, then allowing you to write to it 32kb at a time. If you eject the cd or reboot though, I think the mode page will need to be set up again before you can resume writing, and I don't see a way to do that currently. -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
On 6/18/07, Phillip Susi
can issue a RESERVE TRACK command with the proper mode settings and then proceed to write to that track a packet at a time. I am not sure if it does it correctly or not, but cdrwtool has a -r switch to tell it to reserve a track. This may correctly set up the mode page and reserve the track, then allowing you to write to it 32kb at a time.
Unfortunately not - it fails in much the same way. I'm going to have to try to sit down and fully digest the udf/pktcdvd/ide-cd/cdrom stack so I can understand what's going on here. I'm committed to figuring out how to get this working, but the desire for WORM-like capability seems pretty unique. RB -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
RB wrote:
Unfortunately not - it fails in much the same way.
What is the exact command packet?
I'm going to have to try to sit down and fully digest the udf/pktcdvd/ide-cd/cdrom stack so I can understand what's going on here. I'm committed to figuring out how to get this working, but the desire for WORM-like capability seems pretty unique.
You don't need to understand udf or pktcdvd, and probably not cdrom either; just need to get the cdrwtool commands right to get the media into the proper mode that it will accept the WRITE commands from the kernel. udf only comes into play when you put a udf filesystem on the disc, which as I have said before, you will not be able to do, and pktcdvd only comes into play with cd-rw media formatted in packet mode, not cd-r media. -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
On 6/18/07, Phillip Susi
What is the exact command packet? Command failed: 53 00 00 00 00 00 00 00 01 00 00 00 - sense 05.24.00
You don't need to understand udf or pktcdvd, and probably not cdrom
Understood, and I'll not let that get in my way, but I'd rather know *why* something works, especially if I've had such trouble with it. That's my mechanism for dealing with anything that takes me more than a trivial amount of time to understand - I draw it out and trace the code. RB -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
RB wrote:
On 6/18/07, Phillip Susi
wrote: What is the exact command packet? Command failed: 53 00 00 00 00 00 00 00 01 00 00 00 - sense 05.24.00
Hrm... it looks like cdrwtool is setting the reservation size to 1 sector, which is too small to be a valid track. -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
participants (5)
-
Gustavo Guillermo Pérez
-
John R. Sowden
-
Phillip Susi
-
RB
-
Roy Bamford