[opensuse] cannot rm file or directory from filesystem's / (permission denied)
Googling for cannot files/directories from / seems fruitless, everything being directed either deeper or to things I tried. This summarizes most of what I have tried: # cat /proc/mdstat Personalities : [raid1] ... md1 : active raid1 sdb9[3] sda9[2] 10650988 blocks super 1.0 [2/2] [UU] bitmap: 0/163 pages [0KB], 32KB chunk # tune2fs -l /dev/md1 tune2fs 1.42.12 (29-Aug-2014) Filesystem volume name: 09suse131 Last mounted on: /mnt Filesystem UUID: 057eab05-ebb3-4bfe-8a08-4dc60cc044d3 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: (none) Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 667648 Block count: 5325494 Reserved block count: 266274 Free blocks: 2939203 Free inodes: 528883 First block: 0 Block size: 2048 Fragment size: 2048 Reserved GDT blocks: 512 Blocks per group: 16384 Fragments per group: 16384 Inodes per group: 2048 Inode blocks per group: 128 Flex block group size: 16 Filesystem created: Sat Apr 12 01:43:08 2014 Last mount time: Sun Sep 20 16:20:30 2015 Last write time: Sun Sep 20 16:27:31 2015 Mount count: 2 Maximum mount count: -1 Last checked: Sun Sep 20 16:09:43 2015 Check interval: 0 (<none>) Lifetime writes: 17 GB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 128 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: 2c8dc3c9-9c5e-4e8d-805c-6d5f5b85f62a Journal backup: inode blocks # mount | grep /dev/md1 /dev/md1 on /mnt type ext4 (rw,noatime,data=ordered) # ls -l /mnt/out -rw-rw-rw- 1 root root 120 Apr 16 2014 /mnt/out # lsattr /mnt/out -------------e-- /mnt/out # rm /mnt/out rm: remove regular file /mnt/out? rm: cannot remove /mnt/out: Permission denied # rm -f /mnt/out rm: cannot remove /mnt/out: Permission denied Same/similar happens with two non-standard (non-FHS) 0775 directories and three 644 files, and trying to delete any of them using MC. It also happens when it's the running 13.1 / filesystem. Forced fsck didn't help. IIRC, this filesystem was originally EXT3, later updated to EXT4 following some cmdline instructions found Googling. Short of backup, delete stubborn entries from backup, format the filesystem, and restore, can anyone suggest any relatively user friendly way to remove these stubborn files? -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
John M Andersen composed on 2015-09-20 14:49 (UTC-0700):
Felix Miata wrote:
# ls -l /mnt/out -rw-rw-rw- 1 root root 120 Apr 16 2014 /mnt/out
What are the permissions of /mnt?
With /dev/md1 mounted to it: drwxr-xr-x 27 root root 2048 Apr 22 2014 Without /dev/md1 mounted to it: drwxr-xr-x 2 root root 2048 Sep 25 2014 -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Sun, Sep 20, 2015 at 2:57 PM, Felix Miata
With /dev/md1 mounted to it: drwxr-xr-x 27 root root 2048 Apr 22 2014
Without /dev/md1 mounted to it: drwxr-xr-x 2 root root 2048 Sep 25 2014
Can you provide the output of lsattr for /mnt? Brandon Vincent -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Brandon Vincent composed on 2015-09-20 15:21 (UTC-0700):
Felix Miata wrote:
With /dev/md1 mounted to it: drwxr-xr-x 27 root root 2048 Apr 22 2014
Without /dev/md1 mounted to it: drwxr-xr-x 2 root root 2048 Sep 25 2014
Can you provide the output of lsattr for /mnt?
Null with nothing mounted to it. Mounted, a semi-sorted subset, directories first, plain text files last: ---------------- /mnt/lost+found -------------e-- /mnt/mnt -------------e-- /mnt/nfs -------------e-- /mnt/opt -------------e-- /mnt/smbmnt -------------e-- /mnt/tmp ----------I--e-- /mnt/bin ----------I--e-- /mnt/etc ----------I--e-- /mnt/lib ----------I--e-- /mnt/root ----------I--e-- /mnt/sbin -------------e-- /mnt/out -------------e-- /mnt/shutdown-log-big31-131.txt -------------e-- /mnt/shutdown-log.txt # getfacl /mnt getfacl: Removing leading '/' from absolute path names # file: mnt # owner: root # group: root user::rwx group::r-x other::r-x -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Brandon Vincent composed on 2015-09-20 15:21 (UTC-0700):
Can you provide the output of lsattr for /mnt?
That almost gave the answer. Some more Googling told me I needed the -d flag for lsattr to make it complete.[1] I must have included an errant space in a very bad place in a chattr +i command once upon a time. :-p chattr -i /mnt normalized it. Unwanted files & directories now history. :-) [1] https://rachelbythebay.com/w/2011/10/17/attr/ -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Sun, Sep 20, 2015 at 7:06 PM, Felix Miata
That almost gave the answer. Some more Googling told me I needed the -d flag for lsattr to make it complete.[1] I must have included an errant space in a very bad place in a chattr +i command once upon a time. :-p chattr -i /mnt normalized it. Unwanted files & directories now history. :-)
Good catch! I forgot that lsattr doesn't show the file attributes on directories without that flag. Brandon Vincent -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Felix Miata wrote:
John M Andersen composed on 2015-09-20 14:49 (UTC-0700):
Felix Miata wrote:
# ls -l /mnt/out -rw-rw-rw- 1 root root 120 Apr 16 2014 /mnt/out
What are the permissions of /mnt?
With /dev/md1 mounted to it: drwxr-xr-x 27 root root 2048 Apr 22 2014
Without /dev/md1 mounted to it: drwxr-xr-x 2 root root 2048 Sep 25 2014
---- have you tried a 'strace' and maybe ltrace on the rm command to see if it is failing on a system or a library call? Is there some way in the md driver might be mounting the devices read only, like if the following was stuck?)... md_mod.start_ro=1 /sys/module/md_mod/parameters/start_ro This tells md to start all arrays in read-only mode. This is a soft read-only that will automatically switch to read-write on the first write request. However until that write request, nothing is written to any device by md, and in particular, no resync or recovery operation is started. Or am I understanding it is ONLY a few files that are affected? Can you create/delete other files in root? Can you read from those files? How about write to them i.e. dd if=/dev/null of=/mnt/troublesome_file or just echo >/mnt/troublesome file? Any extended ACL's or any other non-linux default security system installed? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Linda Walsh composed on 2015-09-20 15:35 (UTC-0700):
Felix Miata wrote:
John M Andersen composed on 2015-09-20 14:49 (UTC-0700):
Felix Miata wrote:
# ls -l /mnt/out -rw-rw-rw- 1 root root 120 Apr 16 2014 /mnt/out
What are the permissions of /mnt?
With /dev/md1 mounted to it: drwxr-xr-x 27 root root 2048 Apr 22 2014
Without /dev/md1 mounted to it: drwxr-xr-x 2 root root 2048 Sep 25 2014
---- have you tried a 'strace' and maybe ltrace on the rm command to see if it is failing on a system or a library call?
Not things I can make much sense of: http://fm.no-ip.com/Tmp/SUSE/132/strace-rmr.txt
Is there some way in the md driver might be mounting the devices read only, like if the following was stuck?)...
As files are able to be written to, I have no idea how such could be.
md_mod.start_ro=1
Is this something to try adding to kernel cmdline?
/sys/module/md_mod/parameters/start_ro This tells md to start all arrays in read-only mode. This is a soft read-only that will automatically switch to read-write on the first write request. However until that write request, nothing is written to any device by md, and in particular, no resync or recovery operation is started.
Or am I understanding it is ONLY a few files that are affected?
Can you create/delete other files in root?
Cannot remove or rename those that exist. Cannot create new. (e.g. touch newfile fails) Cannot move any there now to a subdir or to a different filesystem.
Can you read from those files?
Can read and write those that do exist....
How about write to them i.e. dd if=/dev/null of=/mnt/troublesome_file or just echo >/mnt/troublesome file?
mcedit works on the text files there. Subdirs and files can be added to and removed from the non-removable dirs.
Any extended ACL's or any other non-linux default security system installed?
Nothing knowingly done. If I've ever used ACLs, it wasn't intentional. What can I check? Possibly I have found what looks like a clue. This is a partial list of its content gathered while booted from it: -rw-r--r-- 1 root root 88139 Apr 15 2014 .readahead drwxr-xr-x 2 root root 2048 Sep 27 2013 mnt drwxrwxr-x 3 root root 2048 Sep 20 18:49 nfs -rw-rw-rw- 1 root root 120 Apr 16 2014 out -rw-r--r-- 1 root root 340668 Apr 16 2014 shutdown-log-big31-131.txt -rw-r--r-- 1 root root 66462 Sep 20 16:05 shutdown-log.txt drwxrwxr-x 2 root root 2048 Sep 19 14:06 smbmnt I have no recollection what could be responsible for the creation of shutdown-log.txt. From the date of the older one, it must have had to do with some kind of diagnostic request in a forum or Bugzilla. Oh, here: http://lists.opensuse.org/opensuse/2014-04/msg00436.html That reminds me, 13.1 installations randomly take an unusually long time to reboot, typically following applying updates, but not always. When I reboot it (13.1), its content is entirely replaced - I stripped all but the last few lines from it before rebooting, and they were gone afterward. The others either I'm responsible for, or I know what is or was. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 El 2015-09-20 a las 20:16 -0400, Felix Miata escribió:
Cannot remove or rename those that exist. Cannot create new. (e.g. touch newfile fails) Cannot move any there now to a subdir or to a different filesystem.
fsck it. - -- Cheers Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlX/TqsACgkQja8UbcUWM1zeCAD/RWM495UnxTv/+OhzKSslDKtU Kmt3wkAZsn08Uux06xYA/1ysYnV6EefTjMyuIwEUO65doxJ90oZzJchXD+xntTKu =Qe1M -----END PGP SIGNATURE-----
On Sun, Sep 20, 2015 at 5:26 PM, Carlos E. R.
fsck it.
If that doesn't work and you are positive it is not an issue with the block device being read-only, one possible solution would to rebuild the superblock. This should be considered a last resort option! Make sure that you backup the partition and/or are willing to loose the data if rebuilding the superblock does not work as intended. mkfs.ext4 -S /dev/md1 Afterwards, before mounting the filesystem, force a fsck. Brandon Vincent -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. composed on 2015-09-21 02:26 (UTC+0200):
Felix Miata composed:
Cannot remove or rename those that exist. Cannot create new. (e.g. touch newfile fails) Cannot move any there now to a subdir or to a different filesystem.
fsck it.
??? Felix Miata composed on 2015-09-20 17:24 (UTC-0400):
Last checked: Sun Sep 20 16:09:43 2015 -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation)
Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 El 2015-09-20 a las 21:17 -0400, Felix Miata escribió:
Carlos E. R. composed on 2015-09-21 02:26 (UTC+0200):
Felix Miata composed:
Cannot remove or rename those that exist. Cannot create new. (e.g. touch newfile fails) Cannot move any there now to a subdir or to a different filesystem.
fsck it.
??? Felix Miata composed on 2015-09-20 17:24 (UTC-0400):
Last checked: Sun Sep 20 16:09:43 2015
Well, force it, manually. A full manual fsck. - -- Cheers Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlX/XikACgkQja8UbcUWM1zCRgD7BcnRyOAuiWhbMZd/7wJAYqs2 7UGb/zLJLzcPNg1QsfgA/3AV5Rw31fx3gFuuCbRIiVldfM+dQULA7Eq8p/QosRAm =XkhU -----END PGP SIGNATURE-----
Carlos E. R. composed on 2015-09-21 03:32 (UTC-0700):
Felix Miata composed:
??? Felix Miata composed on 2015-09-20 17:24 (UTC-0400):
Last checked: Sun Sep 20 16:09:43 2015
Well, force it, manually. A full manual fsck.
# e2fsck -f /dev/md1 is how it got that way. Is there some more forceful way? -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 El 2015-09-20 a las 21:42 -0400, Felix Miata escribió:
Well, force it, manually. A full manual fsck.
# e2fsck -f /dev/md1
is how it got that way. Is there some more forceful way?
Oh. I'm afraid not :-( - -- Cheers Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlX/Y6QACgkQja8UbcUWM1xs7gD/WQ9RBnU5SrOfgObQiYRsNzNB EYOCyz3GGcAfVrJFs94A/jMRfFQxgOXS9iDcrQ6jiZSz71xNthGZ4m77IzZn0En2 =1g7t -----END PGP SIGNATURE-----
participants (5)
-
Brandon Vincent -
Carlos E. R. -
Felix Miata -
John M Andersen -
Linda Walsh