[Bug 353885] New: reiserfs: cannot write on disk with ~5GB free
https://bugzilla.novell.com/show_bug.cgi?id=353885 Summary: reiserfs: cannot write on disk with ~5GB free Product: openSUSE 10.3 Version: Final Platform: i586 OS/Version: Other Status: NEW Severity: Normal Priority: P5 - None Component: Kernel AssignedTo: kernel-maintainers@forge.provo.novell.com ReportedBy: bluedzins@wp.pl QAContact: qa@suse.de Found By: --- The problem is that I have on one partition nearly 5GB free space. But I cannot use it -- for example when I download 500MB from the net, I get an error stating writing error. Then I can check this manually -- for example: touch x.x joe x.x I press enter (just to get one character) save -> joe crashes If I free some space by deleting a lot of small files (or one big, but this is not interesting) and even if the sum of those files does not make the difference (for example 1000 files, 10MB total) then I can save 500MB file. It looks like there would be a problem with inodes, but reiserfs should dynamically allocate them, right? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User roland.kletzing@materna.de added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c1 roland kletzing <roland.kletzing@materna.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |roland.kletzing@materna.de --- Comment #1 from roland kletzing <roland.kletzing@materna.de> 2008-01-15 14:34:47 MST --- yes, reiserfs has dynamic inode allocation. weird - do you see any error/warning in dmesg or /var/log/messages ? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User bluedzins@wp.pl added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c2 --- Comment #2 from Maciej Pilichowski <bluedzins@wp.pl> 2008-01-16 03:02:19 MST --- Roland, I don't see anything related, but I could of course miss some 1-liner stating "disk error". This happened a long ago to me for the first time, but it was easy to "fix" the problem -- I simply deleted all ccache files, the #1 "abuser" creating tons of small files. This time it didn't help for long. So I figure out, that in time I have more and more problems with small files, where is the #2 abuser? KMail. Today I converted most of its folders from maildir type to mbox. I hope this helps for little longer. Anyway, the problem is still there -- and it is hard to anticipate when some app will crash due to being unable to write to disk. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User roland.kletzing@materna.de added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c3 --- Comment #3 from roland kletzing <roland.kletzing@materna.de> 2008-01-16 15:56:08 MST --- how big is that partition and how many files does it contain? i don`t know any better filesystem for having millions of small files on a partition - iirc, i had <20GB reiser filesystems with more than 2 million files on that - no problems -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User bluedzins@wp.pl added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c4 --- Comment #4 from Maciej Pilichowski <bluedzins@wp.pl> 2008-01-17 02:08:15 MST --- Partition total size is 67GB, files (now, after cleaning). 63452 files (I have a lot small files, and several relatively big files, around 1GB each). About ccache statistics (I keep it now on root partition), in 2 days, compiling solely one program only I get 475MB and 1648 files. Anyway I use reiserfs for a long time and it is not that this problem happens to me everyday, but I did happen -- it was possible to create new file but not to write anything to it (with several GB free at the same time). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User jeffm@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c5 Jeff Mahoney <jeffm@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jeffm@novell.com AssignedTo|kernel-maintainers@forge.provo.novell.com |jeffm@novell.com Status|NEW |ASSIGNED --- Comment #5 from Jeff Mahoney <jeffm@novell.com> 2008-01-22 14:00:47 MST --- The next time it happens, can you post the results of "strace dd if=/dev/zero of=file bs=512 count=1" so we can see what the error is. Reiserfs only has a "theoretical" inode limit, which is 2^32-1. They're dynamically allocated as needed, so I doubt you're running out of inodes. It's possible that a process has a large, recently deleted, file open for which the space won't become available until after the process has closed it. For now, there's really not enough information to go on. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 Jeff Mahoney <jeffm@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO Info Provider| |bluedzins@wp.pl -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User bluedzins@wp.pl added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c6 --- Comment #6 from Maciej Pilichowski <bluedzins@wp.pl> 2008-01-22 14:47:40 MST --- Jeff, thank you for comment, I'll sure do it. Btw. I am 100% sure now nothing goes to /var/log/messages, couple days ago this error happened again (~4GB free on disk, in theory) and this log file was intact -- last registered message was 5 hours ago (then). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User bluedzins@wp.pl added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c7 Maciej Pilichowski <bluedzins@wp.pl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED Info Provider|bluedzins@wp.pl | --- Comment #7 from Maciej Pilichowski <bluedzins@wp.pl> 2008-02-14 05:04:07 MST --- Despite what I said before about tons of small file, today the error hit me again, but today I removed small files to make some room for downloading few big files. With 1GB still free on disk I get error about writing to file. strace log is attached below. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User bluedzins@wp.pl added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c8 --- Comment #8 from Maciej Pilichowski <bluedzins@wp.pl> 2008-02-14 05:04:45 MST --- Created an attachment (id=194878) --> (https://bugzilla.novell.com/attachment.cgi?id=194878) strace.log -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 Jeff Mahoney <jeffm@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #194878|text/x-log |text/plain mime type| | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User jeffm@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c9 Jeff Mahoney <jeffm@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO Info Provider| |bluedzins@wp.pl --- Comment #9 from Jeff Mahoney <jeffm@novell.com> 2008-02-14 08:42:01 MST --- Ok, reiserfsprogs contains a tool that allows you to dump the metadata of the file system for outside analysis. It strips all data except for file names from the file system. Immediately after running into this problem, can you umount the file system and run: debugreiserfs -p <dev> | bzip2 -c > reiserfs-metadata.bz2 .. and attach the result to the bug? It shouldn't be too large. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User bluedzins@wp.pl added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c10 --- Comment #10 from Maciej Pilichowski <bluedzins@wp.pl> 2008-02-14 10:44:37 MST --- Yes, "luckily" I am still suffering from this bug. Btw. in case I would forget -- it seems like there is no relation to big files, small files, but how many files (or how many data) were written without rebooting machine. When I posted above report it was 12:58 I had 1GB free space, now it is 18:40, I didn't restarted computer till then, I removed several big files (each about 500-600MB), so now I have 10GB free space, and still I cannot write anything to the disk. It is home directory (home partition to be exact) so I doubt any system logs have any impact. I'll try to reproduce the bug, then I reboot and check if I can do it again. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User bluedzins@wp.pl added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c11 --- Comment #11 from Maciej Pilichowski <bluedzins@wp.pl> 2008-02-14 10:59:38 MST --- Another note, I am filling the disk by copying files: cp some_file x/ but despite I get for each file message that there is no space on device (but there are a lot in fact) each file is copied as 8912 portion. So I am able to "copy" another file and get another 8912 bytes of it (but no more!). On the other hand, when I rename files in x and try to copy them back to current dir: cp x/some_file_renamed ./ I only get a name of it in current directory and 0 bytes of it. Strange things :-) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User bluedzins@wp.pl added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c12 Maciej Pilichowski <bluedzins@wp.pl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED Info Provider|bluedzins@wp.pl | --- Comment #12 from Maciej Pilichowski <bluedzins@wp.pl> 2008-02-14 12:10:17 MST --- Just as I thought, after restarting I was able to copy two files (each 500MB) which was impossible before restarting. So something is cumulating during normal work. Maybe it is similar to memory leak -- inode leaking? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User jeffm@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c13 --- Comment #13 from Jeff Mahoney <jeffm@novell.com> 2008-02-14 12:16:52 MST --- It's more likely that there's a problem with the bitmap scanning caching code. Can you see if the same thing happens without rebooting, but with just umounting and mounting the file system again? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User jeffm@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c14 Jeff Mahoney <jeffm@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO Info Provider| |bluedzins@wp.pl --- Comment #14 from Jeff Mahoney <jeffm@novell.com> 2008-02-14 12:17:23 MST --- BTW, don't unset NEEDINFO. I'm still looking for the metadata dump from you. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User bluedzins@wp.pl added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c15 --- Comment #15 from Maciej Pilichowski <bluedzins@wp.pl> 2008-02-14 12:19:31 MST --- Created an attachment (id=194989) --> (https://bugzilla.novell.com/attachment.cgi?id=194989) reiserfs-metadata.bz2 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User bluedzins@wp.pl added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c16 --- Comment #16 from Maciej Pilichowski <bluedzins@wp.pl> 2008-02-14 12:34:39 MST --- Jeff, ad.14, the metadata is a big file, so you probably didn't see it was coming (and it took me a while to send it). About umount/mount, I will check it, but I have to wait for the error. I don't know how to simulate it -- I tried to copy files to 0B free space, but I didn't get the same result. I am leaving needinfo, since I guess, you set it for that answer. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User aj@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c17 Andreas Jaeger <aj@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED Info Provider|bluedzins@wp.pl | --- Comment #17 from Andreas Jaeger <aj@novell.com> 2008-03-11 11:46:39 MST --- I guess the info has been provided. in comment #15. Jeff, do you agree? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User jeffm@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c18 --- Comment #18 from Jeff Mahoney <jeffm@novell.com> 2008-03-13 11:01:57 MST --- Yes. I haven't been able to reproduce this either, though. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User joachimbanzhaf@compuserve.de added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c19 Joachim Banzhaf <joachimbanzhaf@compuserve.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO Info Provider| |bluedzins@wp.pl --- Comment #19 from Joachim Banzhaf <joachimbanzhaf@compuserve.de> 2008-04-04 14:38:16 MST --- I can confirm comment 13. I have had the described behaviour for the following system, so this is not a bug of only one system or arch: Linux job4 2.6.22.9-0.4-default #1 SMP 2007/10/05 21:32:04 UTC x86_64 x86_64 x86_64 GNU/Linux I already changed the harddisk, recreated the filesystem and copied the old files over, to rule out hd problems and corrupted fs metadata. But the problem showed up again now. This is what I see: Filesystem Size Used Avail Use% Mounted /dev/mapper/vidvg-video 581G 495G 86G 86% /var/spool/video root@job4 /var/spool/video/test # touch test touch: cannot touch `test': No space left on device I just unmounted the volume, did a reiserfsck and mounted again. # reiserfsck -y /dev/vidvg/video reiserfsck 3.6.19 (2003 www.namesys.com) .. Will read-only check consistency of the filesystem on /dev/vidvg/video Will put log info to 'stdout' ########### reiserfsck --check started at Fri Apr 4 21:25:46 2008 ########### Replaying journal: Done. Reiserfs journal '/dev/vidvg/video' in blocks [18..8211]: 0 transactions replayed Checking internal tree.. finished Comparing bitmaps..finished Checking Semantic tree: finished No corruptions found There are on the filesystem: Leaves 131173 Internal nodes 849 Directories 522 Other files 15468 Data block pointers 129590610 (42 of them are zero) Safe links 0 ########### reiserfsck finished at Fri Apr 4 21:36:57 2008 ########### Dmesg and /var/log/messages show nothing related except for the usual mounting messages: root@job4 ~ # dmesg | grep -v SFW2 | tail -50 REISERFS (device dm-2): found reiserfs format "3.6" with standard journal REISERFS (device dm-2): using ordered data mode reiserfs: using flush barriers REISERFS (device dm-2): journal params: device dm-2, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30 REISERFS (device dm-2): checking transaction log (dm-2) reiserfs: disabling flush barriers on dm-2 REISERFS (device dm-2): Using r5 hash to sort names The problem is gone for now. I could easily create 50 1GB files. The system is a vdr server which usually runs for quite some time without rebooting: root@job4 /var/spool # uptime 10:30pm up 14 days 7:19, 6 users, load average: 3.02, 3.12, 3.00 Hope that helps to identify the problem. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 Joachim Banzhaf <joachimbanzhaf@compuserve.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED Info Provider|bluedzins@wp.pl | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User bradleydanecook@gmail.com added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c20 Brad Cook <bradleydanecook@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bradleydanecook@gmail.com --- Comment #20 from Brad Cook <bradleydanecook@gmail.com> 2008-12-22 01:02:25 MST --- Hi Jeff, Same issue running reiserfs and openSUSE 10.3 on 2 systems (different hardware platforms) Is there any debugging info that I can provide you with that will assist you? Linux 2.6.22.5-31-bigsmp #1 SMP 2007/09/21 22:29:00 UTC i686 i686 i386 GNU/Linux Ive run through reiserfsck.. no issues there. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User roland.kletzing@materna.de added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c1 roland kletzing <roland.kletzing@materna.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |roland.kletzing@materna.de --- Comment #1 from roland kletzing <roland.kletzing@materna.de> 2008-01-15 14:34:47 MST --- yes, reiserfs has dynamic inode allocation. weird - do you see any error/warning in dmesg or /var/log/messages ? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User bluedzins@wp.pl added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c2 --- Comment #2 from Maciej Pilichowski <bluedzins@wp.pl> 2008-01-16 03:02:19 MST --- Roland, I don't see anything related, but I could of course miss some 1-liner stating "disk error". This happened a long ago to me for the first time, but it was easy to "fix" the problem -- I simply deleted all ccache files, the #1 "abuser" creating tons of small files. This time it didn't help for long. So I figure out, that in time I have more and more problems with small files, where is the #2 abuser? KMail. Today I converted most of its folders from maildir type to mbox. I hope this helps for little longer. Anyway, the problem is still there -- and it is hard to anticipate when some app will crash due to being unable to write to disk. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User roland.kletzing@materna.de added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c3 --- Comment #3 from roland kletzing <roland.kletzing@materna.de> 2008-01-16 15:56:08 MST --- how big is that partition and how many files does it contain? i don`t know any better filesystem for having millions of small files on a partition - iirc, i had <20GB reiser filesystems with more than 2 million files on that - no problems -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User bluedzins@wp.pl added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c4 --- Comment #4 from Maciej Pilichowski <bluedzins@wp.pl> 2008-01-17 02:08:15 MST --- Partition total size is 67GB, files (now, after cleaning). 63452 files (I have a lot small files, and several relatively big files, around 1GB each). About ccache statistics (I keep it now on root partition), in 2 days, compiling solely one program only I get 475MB and 1648 files. Anyway I use reiserfs for a long time and it is not that this problem happens to me everyday, but I did happen -- it was possible to create new file but not to write anything to it (with several GB free at the same time). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 User jeffm@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=353885#c5 Jeff Mahoney <jeffm@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jeffm@novell.com AssignedTo|kernel-maintainers@forge.provo.novell.com |jeffm@novell.com Status|NEW |ASSIGNED --- Comment #5 from Jeff Mahoney <jeffm@novell.com> 2008-01-22 14:00:47 MST --- The next time it happens, can you post the results of "strace dd if=/dev/zero of=file bs=512 count=1" so we can see what the error is. Reiserfs only has a "theoretical" inode limit, which is 2^32-1. They're dynamically allocated as needed, so I doubt you're running out of inodes. It's possible that a process has a large, recently deleted, file open for which the space won't become available until after the process has closed it. For now, there's really not enough information to go on. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=353885 Jeff Mahoney <jeffm@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO Info Provider| |bluedzins@wp.pl -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com