If the list is coming back to life, then I've got a weird but hopefully simple problem for you:
I've got a Synology DiskStation DS209 NAS that's giving me erroneous free space readings, or erroneous du output. On installation, it dedicated 2.4GB to its root partition /dev/md0 and according to du -cx / is using less than 587192K of that space. Unfortunately, however, df says it's full.
bash-3.2# df -k Filesystem 1K-blocks Used Available Use% Mounted on rootfs 2451064 2425764 0 100% / /dev/root 2451064 2425764 0 100% /
bash-3.2# mount /dev/md0 on / type ext3 (rw,data=ordered)
As a result, any number of apps from the stock firmware are prevented from running and those that do hobble along spam the logs to the point of uselessness w/complaints. The catch? I can't find a safe way to fsck it. Conventional shutdown commands to force a fsck aren't an option/are just plain missing on the box. Using tune2fs to set the check interval ridiculously low to force a check upon reboot similarly didn't work. Unfortunately, the ARM-based architecture also doesn't support booting to a removable USB device, knocking out the only other safe way I can think of to fsck it. Pop new firmware on to start from scratch? I'd love to at this point, but the firmware installation process requires 1024 megs of free disk space.
Thoughts? Any ideas on how to safely get it cleaned up and accurately reporting its free space while the root filesystem is live and booted?
-Sean
On Wed, Jun 2, 2010 at 10:47 PM, Sean Crago [email protected] wrote:
If the list is coming back to life, then I've got a weird but hopefully simple problem for you:
I've got a Synology DiskStation DS209 NAS that's giving me erroneous free space readings, or erroneous du output. On installation, it dedicated 2.4GB to its root partition /dev/md0 and according to du -cx / is using less than 587192K of that space. Unfortunately, however, df says it's full.
bash-3.2# df -k Filesystem 1K-blocks Used Available Use% Mounted on rootfs 2451064 2425764 0 100% / /dev/root 2451064 2425764 0 100% /
bash-3.2# mount /dev/md0 on / type ext3 (rw,data=ordered)
As a result, any number of apps from the stock firmware are prevented from running and those that do hobble along spam the logs to the point of uselessness w/complaints. The catch? I can't find a safe way to fsck it. Conventional shutdown commands to force a fsck aren't an option/are just plain missing on the box. Using tune2fs to set the check interval ridiculously low to force a check upon reboot similarly didn't work. Unfortunately, the ARM-based architecture also doesn't support booting to a removable USB device, knocking out the only other safe way I can think of to fsck it. Pop new firmware on to start from scratch? I'd love to at this point, but the firmware installation process requires 1024 megs of free disk space.
Thoughts? Any ideas on how to safely get it cleaned up and accurately reporting its free space while the root filesystem is live and booted?
-Sean
Oh, and I really doubt this changes the solution, but for full disclosure's sake:
Last change I made to the system was to install the ipkg for mlocate, which stores its small 500k db on a different partition. Also, power situation is abysmal here, and I'm not convinced it's shutting down properly when it detects that it's on UPS power.
Kelsay: Very nearly all space in use is being used by the OS proper and, strangely, far less than required for an install. I could hose the system with an rm -rf / and still not have enough space to install new firmware, due to this phantom disk usage problem.
Andrew: I'm assuming md0 is on one or more of my two disks, but still investigating whether this is safe to do. Researching mdadm-style software RAID is top of my list of things to do in 9-10 hours after work, but not comfortable popping it out just yet/not sure how to tell if this /dev/md0 device is only on one of my disks and not on both, or a third device/onboard flash.
Jack: Sorry about that - I did have the "fsck -n" output, but didn't want to make my post that much longer. Dumped it here, as well as "df -i" and "du -sxk /" output. The fsck does give multiple free space errors, and df -i only shows 13% of inodes in use.
http://docs.google.com/View?id=dgrxhrtx_82fmxvtsfg
Is there a way to safely fix the problem on a live filesystem, due to Synology's custom distro ignoring tune2fs's check interval settings on boot?
I would personally go to runlevel 1, unmount the disk and run fsck on it and fix the problems. Then once you have a clean disk you should know if more investigation is required. Right now you have a borked filesystem and you shouldn't be running anything pn it until it's fixed. But that's just my opinion.
Also, curious to how you set up the file system. What size blocks is it using? If your filesystem is going to have lots of small files, you are better off using a small lock size. Sometimes the default settings on creating filesystems are ok, but sometimes you want to tweak the settings for better use of the disk. A smaller disk with a lot of small files will benefit from optimizing the inode block-size. Although it looks like you have plenty of free inodes. Conversely if you have mostly big files, then using a bigger size block will improve system performance. Of course, this is irrelevant now, without wiping out the data on the drive.
Jack
--- On Wed, 6/2/10, Sean Crago [email protected] wrote:
From: Sean Crago [email protected] Subject: Re: Synology & Fsck To: "KCLUG" [email protected] Date: Wednesday, June 2, 2010, 6:05 PM
Kelsay: Very nearly all space in use is being used by the OS proper and, strangely, far less than required for an install. I could hose the system with an rm -rf / and still not have enough space to install new firmware, due to this phantom disk usage problem.
Andrew: I'm assuming md0 is on one or more of my two disks, but still investigating whether this is safe to do. Researching mdadm-style software RAID is top of my list of things to do in 9-10 hours after work, but not comfortable popping it out just yet/not sure how to tell if this /dev/md0 device is only on one of my disks and not on both, or a third device/onboard flash.
Jack: Sorry about that - I did have the "fsck -n" output, but didn't want to make my post that much longer. Dumped it here, as well as "df -i" and "du -sxk /" output. The fsck does give multiple free space errors, and df -i only shows 13% of inodes in use.
http://docs.google.com/View?id=dgrxhrtx_82fmxvtsfg
Is there a way to safely fix the problem on a live filesystem, due to Synology's custom distro ignoring tune2fs's check interval settings on boot?
Not the cleanest solution, but mdadm info showed /dev/md0 was mirrored in a 2.4G partition on both hard drives. Didn't want to risk hosing it altogether by making changes to just one over a USB SATA adapter, or trying to rebuild the array over two of 'em, so went the annoying but safer route of just whacking reset pin on the back and reinstalling the firmware. Handled it quite well, only re-mke3fsing /dev/md0 and requiring a handful of settings + my 3rd-party ipkg tree be reinstalled. The downloader remembered its task list and the data partitions were untouched.
Not bad for a $100 NAS. Never found a clear cause (no lsof available, etc, etc) but fairly confident the cause was just frequent city power failures.
Thanks for the advice, Sean
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 6/2/2010 8:05 PM, Sean Crago wrote:
Andrew: I'm assuming md0 is on one or more of my two disks, but still investigating whether this is safe to do. Researching mdadm-style software RAID is top of my list of things to do in 9-10 hours after work, but not comfortable popping it out just yet/not sure how to tell if this /dev/md0 device is only on one of my disks and not on both, or a third device/onboard flash.
What's in /proc/mdstat ?
This will tell you what sort of RAID device /dev/md0 is (likely a RAID1 mirror), and which devices it consists of.
Is there a way to safely fix the problem on a live filesystem, due to Synology's custom distro ignoring tune2fs's check interval settings on boot?
You mentioned the ARM system won't boot from a USB device, but can it see one once it's running? If so (or if you've got enough ram or space elsewhere), you can try to copy the existing OS files to a new location, do a pivot_root, and then unmount/fsck the old (broken) root filesystem:
http://linux.die.net/man/8/pivot_root
Also, if you have any way to play with the kernel boot command line (ie: alter the boot-loader settings), you might try some tricks like setting init=/bin/sh. You should then be able to fsck the root filesystem when it's mounted read-only and there's very little running (ie: similar to what most init scripts do check the root fs).
First, however, I'd look to see if there's a /forcefsck flag or similar you can pass to the init scripts (ie: "touch /forcefsck" on a debian based system, then reboot). Look through the init scripts for anything that runs fsck, and see if there's a way to force it into operation.
- -- Charles Steinkuehler [email protected]
-
Charles Steinkuehler -
Jack -
Sean Crago