Hello IQfy, I'm thinking of attempting to write a filesystem for fun, so I'm currently thinking of what features it'll have. I don't believe that filesystems should traverse multiple block devices, so anything along the lines of RAID is crossed out. These are the features currently planned to be implemented, or at least attempted. I got a faint idea of using some kind of radix trie like HAMMER2 seems to do (from a quick look at https://apollo.backplane.com/DFlyMisc/hammer2.txt), but we'll see in due time. Currently this is like a wish list of mine for a new filesystem.
1. Fast sequential read/write (unlike other CoW filesystems that fragment very easily).
2. Fast and easy snapshots, hopefully it could scale to hundreds of thousands at least
3. Minimal seeking (for easiness on HDD)
4. Unlimited reflinks
5. Compression with zstd/lz4
6. Minimal performance and space overhead, should be pretty fast on both HDD and SSD.
7. Defragmentation, offline and maybe online
8. Low memory footprint
9. Cheap offline AND online deduplication
10. Streamable transfers (for piping backups like ZFS)
11. Easy to use CLI interface (like ZFS)
12. Data and Metadata Checksums.
13. Cheap and real cloning (unlike fake ZFS cloning that permanently enslaves the clone to a snapshot)
14. Subvolume/Dataset/Snapshot dependent settings? Not sure I actually want this, but seems useful when using ZFS
15. Mirroring/Reed Solomon codes on disk, can be used to recover from errors.
If there are any other neat features you'd like in a filesystem, please tell. It's useful to get a list of what one might want before trying to figure out how it'd all work.
what formfactor is that on the far right? adorable
apparently a 1-inch drive, introduced by IBM in 1999, see https://en.wikipedia.org/wiki/List_of_disk_drive_form_factors
Well I can't promise that anon
how else would i know that your filesystem isn't likely to eat my data?
the main feature i want is many corporations using it in production
https://en.wikipedia.org/wiki/Triple_modular_redundancy
I'm not entirely certain how this would be superior to Reed Solomon or just copying the data? I'm assuming you mean for it to be used for error correction?
Didn't think of that, I mostly associate with just hierarchical data organization. Seems like it would be interesting to have at the file system level, say in addition to snapshots and the like, you also have tags. Perhaps multiple tags could be forced in like 'tag_virtual_folder/tag1/tag2/' with the tagged files appearing? I'd assume this would be better done on top of the filesystem though, like e.g. TMSU.
>I'm not entirely certain how this would be superior to Reed Solomon or just copying the data? I'm assuming you mean for it to be used for error correction?
oh I'm stupid, you probably mean to do the I/O option three times and take the quorum, apologies.
>I'm not entirely certain how this would be superior to Reed Solomon
TMR is basically real time with virtually no overhead.
would prefer 3 drives, all basically mirrored. so could reasonably survive a double drive failure. rebuilding an array would basically just be copying data.
Anyone is free to argue getting 1/3 your storage space may not be worth it. but there is definitely a reliability improvement. I believe read speeds could be tripled as well. and rebuilds would be significantly faster and safer. a single drive failure would still leave you with 2 copies of data, which can be compared during a single drive failure. and i dont think this has been done for a "RAID" setup before, so it would be nice if it was brought into existence.
so you're basically just asking for a 3-way mirror if I'm reading correctly? The idea of a quorum for a paranoid filesystem seems interesting, even if I misread your original post.
Anyhow, like I said in OP, I don't really believe filesystems should traverse drives (that should be done by a block device layer below or an application over the network above). In this case, I'd imagine the best setup would be something like a three way RAID1 setup using mdadm
>, I don't really believe filesystems should traverse drives
it's a disaster waiting to happen if there's no redundancy.
>so you're basically just asking for a 3-way mirror if I'm reading correctly?
Well, a TMR setup can both detect errors and correct them. a 2-way mirror, if it sees a discrepancy between the 2 copies, will have to rely on other methods to correct the error.
>I don't really believe filesystems should traverse drives
why not? I feel like I agree, but I believe ZFS implements RAID at the file system level, and it seems pretty respected from what ive seen.
>why not?
mostly because it makes the whole thing a lot more complex and duplicates existing battle-tested code. I feel the file system should be simple (rather than the complicated beasts that are bcachefs, btrfs, and zfs). Perhaps when developing it I could see a simple way to mix the layers without causing needless complexity, but currently it's not really a priority.
The main concern I have against my own position is that if an error is found in the filesystem (say a bad checksum) and the redundancy errors out too, then its game over. Whereas for ZFS and btrfs, they can see exactly which drive gave the bad data and work around it. I don't believe a naive filesystem on top of a block device can know which drive, of the multiple underlying that virtual block device if mdadm, had the bad data. In short, I'll think about it, but the primary priority is to make a good filesystem first and then potentially make it cross-drive.
> Compression with zstd/lz4
not bad but would that be optional? would be a lot easier to let homosexuals roll their own solution and incorporate some kind of encryption because seeing this in the doc file for hammer2:
> "Encryption. Whole-disk encryption is supported by another layer, but I intend to give H2 an encryption feature at the logical layer which works approximately as follows:....
> CURRENT STATUS: Not yet operational.
>perhaps multiple tags could be forced in like 'tag_virtual_folder/tag1/tag2/' with the tagged files appearing?
tagging is a mistake. besides what i think about tagging, i think that is an interesting idea but i wonder how bloated that would end up being.
>not bad but would that be optional?
I originally planned for it to be optional, so yes. Also on the encryption tangent, I wasn't thinking of implementing encryption as LUKS is pretty good on its own. I could change my mind though, I mean I haven't even started (besides if I implement it right, it should be extensible enough to add it later).
>how bloated that would end up being.
probably very, hence my comment about it being better done a layer above. Still an interesting thought experiment, since I don't necessarily have to have snapshots be the only thing referencing blocks.
I'd agree, but it's a massive layering violation for a filesystem to do drive redundancy, and would drastically increase the complexity of the filesystem. There already exist solutions that can do it a layer below, like mdadm and LVM.
Ideally a filesystem should just turn a bunch of blocks of bytes into a usable filesystem.
tag system please
and have it project files outside of the file browser without any metadata by default that's annoying when it's included
Filesystems were solved in the 50s.
https://en.m.wikipedia.org/wiki/IBM_305_RAMAC
Ability to leave comments on any file using unicode emojis
I want HAMMER2 on linux