H4N: Unusual "invalid file" issue

Discuss the Zoom H6, H5, H4, H4n, H2, H2n, and H1. Please don't "post and run". Participate in the discussion. Thanks.
Post Reply
funkapus
new to this board
new to this board
Posts: 3
Joined: Tue Jan 02, 2018 12:35 am

H4N: Unusual "invalid file" issue

Post by funkapus » Thu Jan 04, 2018 12:47 am

Hi folks. I'm using an H4N with a Kingston 16GB card that's in the compatibility list and that I've used successfully to make stereo WAV recordings for over five years. Never a problem with the device or the card until now. I'm having an "invalid file" issue; but I've searched the archives here, and mine seems to be different from issues that others have posted about.

I have a ~25 minute recording that shows up in the list of the files in the folder; but any attempt to select it or get information about it returns an "invalid file" error. While it's listed in the contents of the folder, it doesn't appear in the main display if I step through the files or play them sequentially.

Connecting the device, or the card in an SD card reader, to a computer via USB and mounting the storage, I can see the file there, and its file size is listed as around 260 MB, which is what I'd expect for a 25 minute 2-channel, 16bit, 44.1kHz sample rate recording. But if I try to copy the file off the SD card and onto the computer, all I get is 32768 bytes before an I/O error is generated. Other applications such as VLC or audacity are also unable to read the file.

I've looked at repairing the filesystem using the FAT32 fsck utility with Linux. This successfully interprets the file system and identifies the problem file; but it wants to truncate the file to 0 bytes, which obviously isn't what I want.

I've been doing a lot of reading for the last several days and I have an idea what may have happened. As I understand it, Zoom uses the FAT32 file system for its cards. In their implementation of FAT32, space is allocated to files in 32kB clusters. The directory entry for the file contains its file size (~260MB) and the location of the first cluster in the file. The file allocation table is used to find the locations and order of the rest of the clusters in the file: it's a large linked list, where you go to a location in the file allocation table indexed by an offset corresponding to the first cluster in the file, and the contents of that location tells you the location of the next cluster in the file; you go to a location in the file allocation table indexed by an offset corresponding to that second cluster, at the contents of that location tells you the location of the third cluster in the file; and so on until you hit a value of FFFFFFFF, which means no more clusters in this file. And I *think* that the very first entry in the file allocation table for this file has somehow become corrupted with a FFFFFFFF value. So trying to copy the file only produces one 32kB cluster, because the file allocation table points to no additional clusters beyond the first; and it comes back as invalid on the H4N because one cluster only conflicts with the 260MB file size stored in the directory.

I have no doubt that I can continue to use the card if I reformat it. But it'd break my heart to lose this file; the contents cannot be replaced. But I don't know how to go about attempting to recover it. This is not the same as attempting to recover a deleted file. If my hypothesis above is correct, who knows where the first entry in the file allocation table *should* have pointed to. One could imagine reverse-engineering *all* the existing linked lists in the file allocation table, going backwards from every FFFFFFFF; but many of those will correspond to deleted files, and their linked lists will no longer be intact. And of course, maybe my hypothesis is wrong.

Any advice on how to proceed here would be very much appreciated.

Thanks.
0 x

Wulfraed
The Force
The Force
Posts: 3094
Joined: Wed Jan 23, 2008 8:18 pm
Location: Kentwood, MI

Re: H4N: Unusual "invalid file" issue

Post by Wulfraed » Thu Jan 04, 2018 10:37 am

Other than starting with some utility that can do a block copy -- ie; ignore file system and just copy the entire card as a card image (you mention Linux -- "dd" might be the utility I think of).

With a card image that you can write to a similar sized card, you might be able to afford to play around (test the copy first to ensure you can write the equivalent) with editing the file system.

I'd like to mention that "all-1s" is also the state of an erased FLASH memory allocation unit. Flash memory formats/erases to all-1s, and data writes can only modify bits to be a 0. To rewrite an area requires obtaining an unused allocation unit, erasing it to all-1, then copying unchanged data from the currently occupied allocation unit, up to where ever new/modified data is to be written. The "currently occupied" allocation unit is then marked as available and the new one is mapped into its place. Allocation units are determined by the Flash chip itself, and tend to be measured in MB -- they are independent from file system "sectors/blocks".

I don't know enough on how to mount an "image" as the file system, but if you envisage mucking about within the FAT, you'll likely need to do that in order to bypass the Flash memory allocation unit activity.

Do not do anything that tries to write to the card until you've obtained a viable backup image.
0 x
--
Baron Wulfraed
IISS Elusive Unicorn (detached)

Superscope PSD-300; BOSS BR-600, Zoom HD16cd, Zoom R16, BOSS BR-800, Zoom H2n
Now to (re)learn to play an instrument

Lanikai S-C, SMC-E; GoldTone Banjo-Uke; Flatiron 1C, A5; Big Muddy M1-W; Ovation MM68AX, CSE-44; Orpheus Valley Fiesta FS; Taylor NS-72ce, T5-S1; Musima (4st, 20 fret, tenor-tuned) banjo; bongos, dumbeks, bodhrans, hand drum, tambourine; recorder: soprano, alto, tenor; Cedar Flute (5 sizes); Pennywhistle (3 keys); Casio keyboards

funkapus
new to this board
new to this board
Posts: 3
Joined: Tue Jan 02, 2018 12:35 am

Re: H4N: Unusual "invalid file" issue

Post by funkapus » Thu Jan 04, 2018 3:46 pm

Wulfraed wrote:
Thu Jan 04, 2018 10:37 am
Other than starting with some utility that can do a block copy -- ie; ignore file system and just copy the entire card as a card image (you mention Linux -- "dd" might be the utility I think of).
Yeah, I made a raw copy of the card contents using dd (block size 512 bytes), and most of my poking around has actually been in that copy rather than the card itself. Most of the things one would run on the card also run on that image, with the file system check/repair fsck (or more specifically fsck.fat) being a significant exception. But I hadn't thought to put the image on another card. I do have two spares -- one is currently in use in the H4N so I can keep using it, but I can put the image on the other. That's a good idea going forward, thanks.
I'd like to mention that "all-1s" is also the state of an erased FLASH memory allocation unit. Flash memory formats/erases to all-1s, and data writes can only modify bits to be a 0. To rewrite an area requires obtaining an unused allocation unit, erasing it to all-1, then copying unchanged data from the currently occupied allocation unit, up to where ever new/modified data is to be written. The "currently occupied" allocation unit is then marked as available and the new one is mapped into its place. Allocation units are determined by the Flash chip itself, and tend to be measured in MB -- they are independent from file system "sectors/blocks".
Thanks, didn't know about this.
I don't know enough on how to mount an "image" as the file system, but if you envisage mucking about within the FAT, you'll likely need to do that in order to bypass the Flash memory allocation unit activity.
Yeah, I was hoping that this would be as straightforward as fixing the first bad entry in the linked list using a hex editor working on the image file I created via dd, and then dd'ing it back to a card. But the hard part is knowing what value to replace the FFFFFFFF with. And then there's the hope that that's the only problem with the linked list. It may not be a solvable problem, but I'd really like to solve it if it is possible.
0 x

Wulfraed
The Force
The Force
Posts: 3094
Joined: Wed Jan 23, 2008 8:18 pm
Location: Kentwood, MI

Re: H4N: Unusual "invalid file" issue

Post by Wulfraed » Thu Jan 04, 2018 8:15 pm

funkapus wrote:
Thu Jan 04, 2018 3:46 pm
I'd like to mention that "all-1s" is also the state of an erased FLASH memory allocation unit. Flash memory formats/erases to all-1s, and data writes can only modify bits to be a 0. To rewrite an area requires obtaining an unused allocation unit, erasing it to all-1, then copying unchanged data from the currently occupied allocation unit, up to where ever new/modified data is to be written. The "currently occupied" allocation unit is then marked as available and the new one is mapped into its place. Allocation units are determined by the Flash chip itself, and tend to be measured in MB -- they are independent from file system "sectors/blocks".
Thanks, didn't know about this.
I thought to mention that as the value you state FAT uses for end-of-chain would also be the result of an aborted "allocation unit copy" (though I'd think the FAT data would be the last to be updated, so either the card sees the "pre-FAT update" -- no file, or the update would be written with usage correct). The aborted copy could be from some inadvertent ejection of the card (physically removing it before the OS had finished flushing updates to it, etc.)
Yeah, I was hoping that this would be as straightforward as fixing the first bad entry in the linked list using a hex editor working on the image file I created via dd, and then dd'ing it back to a card. But the hard part is knowing what value to replace the FFFFFFFF with. And then there's the hope that that's the only problem with the linked list. It may not be a solvable problem, but I'd really like to solve it if it is possible.
Might have to do the tedious of first wiping all the good data blocks by following their chains, then trying to collect what is left, and hope they are relatively linear in distribution.

Heh... Many years ago, the original Amiga file system made such recovery trivial. But it also meant for a lot of overhead in each sector, leaving only about 480 bytes for data.

Each block had links to: previous data block, next data block, file header block. If the file header was corrupt, one could rebuild it using the previous/next data block links. File header blocks had a data section that the file name, a list of data blocks, pointers to previous/next file header blocks*, and pointer to parent directory block. Directory blocks had the directory name, and a hash table which pointed to file/directory header blocks (they basically looked the same but for a flag value).

The fast file system did away with the data block previous/next/parent links making all 512 bytes available for data, but making recovery more difficult.


* Other than my college data structures class, this is the only place I've ever seen a hashed-head multiple-linked list.
0 x
--
Baron Wulfraed
IISS Elusive Unicorn (detached)

Superscope PSD-300; BOSS BR-600, Zoom HD16cd, Zoom R16, BOSS BR-800, Zoom H2n
Now to (re)learn to play an instrument

Lanikai S-C, SMC-E; GoldTone Banjo-Uke; Flatiron 1C, A5; Big Muddy M1-W; Ovation MM68AX, CSE-44; Orpheus Valley Fiesta FS; Taylor NS-72ce, T5-S1; Musima (4st, 20 fret, tenor-tuned) banjo; bongos, dumbeks, bodhrans, hand drum, tambourine; recorder: soprano, alto, tenor; Cedar Flute (5 sizes); Pennywhistle (3 keys); Casio keyboards

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests