Tag Archives: btrfs

btrfs inode errors with file extent discount / file extent holes

This was a new one for me. An external USB 3.0 btrfs RAID 0 volume had some random I/O errors of undetermined cause during heavy writes — no kernel oops or obvious bug, and no clear hardware failure either. It came after seeing a bunch of these in my kernel log:

xhci_hcd 0000:00:14.0: ERROR unknown event type 37

This apparently has something to do with filling a ring buffer according to this LKML post. I’m not sure if it’s related but it was coincident with the following errors.

I’ve been running btrfs on a number of volumes for years (with many previous catastrophes) and this is the first time I’ve seen this particular error:

btrfs check --repair /mountpoint
...
root 5 inode 7553112 errors 100, file extent discount
Found file extent holes:
start: 0, len: 16384
root 5 inode 7553113 errors 100, file extent discount
Found file extent holes:
start: 0, len: 12288
root 5 inode 7553115 errors 100, file extent discount
Found file extent holes:
start: 0, len: 28672
...
root 5 inode 7555752 errors 100, file extent discount
Found file extent holes:
start: 0, len: 45056
ERROR: errors found in fs roots
found 9165863124992 bytes used, error(s) found
total csum bytes: 8933283920
total tree bytes: 15941156864
total fs tree bytes: 5797117952
total extent tree bytes: 473710592
btree space waste bytes: 1645027279
file data blocks allocated: 9149921968128
referenced 9216958279680

The solution I came across is a bit time consuming. It relies on freeing the affected inodes, which in turn means removing the original files from the filesystem. You can copy them elsewhere or just delete them if you have a backup or decide they aren’t valuable. In my case I kept the files around long enough to see if they matched a backup — and they did, indicating that the files themselves weren’t affected by this problem.

First I took the output from btrfs check and picked out the inode numbers:

grep 'file extent discount' btrfs_check_output | cut -f4 -d' ' > bad_inodes

Explanation: I put the output from btrfs check into a file called btrfs_check_output. The grep command searches this file for the lines with inode numbers, and cut picks out just the inode numbers from the rest of the line: -f4 means take the fourth field -d' ' delimited by spaces. The output goes into a file called bad_inodes.

This ended up with a list of 705 bad inodes! Sheesh… now comes the long part. In order to find the files associated with these inodes, I put this Bash command line together:

while read inode; do find /mountpoint -xdev -inum $inode -print0 -quit; done < bad_inodes > bad_inode_files

Explanation: This reads the bad_inodes file as standard input and loops over each line as variable $inode and then executes a find. The output is sent to a file named bad_inode_files. The arguments to find are -inum which searches for a file by inode number, and -xdev which stops the file search from descending into other filesystems, since we don’t want to mistakenly find files with the same inode number on a different filesystem. The quit option stops the search after the first result, since we don’t need to keep searching the entire filesystem after we have located the file associated with a particular inode. -print0 is optional and is for handling filenames containing whitespace by delimiting each item of output with a null character instead of the default newline. You can substitute -print0 with plain old print if you aren’t going to process this output with another utility, or if you are sure there aren’t filenames with whitespace in your output.

This took a while to complete since it has to search for each inode starting at the root of the filesystem, and in this case it’s a 10TB array and 705 inodes. When it completed, it produced a list of affected files in a text file named bad_inode_files.

I could have just moved or deleted the files, but I wanted to inspect them first. The affected filesystem is a backup, so I compared the files to the originals. All of the files matched according to rsync which means that they’re probably not damaged. I then went ahead and deleted all of the affected files since I had them on a backup. If I didn’t have a backup I could have moved them to another location and then back.

Now after removing the files with the affected inodes, btrfs check returns no errors.