Summary
When using rustix::fs::Dir using the linux_raw backend, it's possible for the iterator to "get stuck" when an IO error is encountered. Combined with a memory over-allocation issue in rustix::fs::Dir::read_more, this can cause quick and unbounded memory explosion (gigabytes in a few seconds if used on a hot path) and eventually lead to an OOM crash of the application.
Details
Discovery
The symptoms were initially discovered in https://github.com/imsnif/bandwhich/issues/284. That post has lots of details of our investigation. See this post and the Discord thread for details.
Diagnosis
This issue is caused by the combination of two independent bugs:
- Stuck iterator
- The
rustix::fs::Dir iterator can fail to halt after encountering an IO error, causing the caller to be stuck in an infinite loop.
- Memory over-allocation
Dir::read_more incorrectly grows the read buffer unconditionally each time it is called, regardless of necessity.
Since <Dir as Iterator>::next calls Dir::read, which in turn calls Dir::read_more, this means an IO error encountered during reading a directory can lead to rapid and unbounded growth of memory use.
PoC
fn main() -> Result<(), Box<dyn std::error::Error>> {
// create a directory, get a FD to it, then unlink the directory but keep the FD
std::fs::create_dir("tmp_dir")?;
let dir_fd = rustix::fs::openat(
rustix::fs::CWD,
rustix::cstr!("tmp_dir"),
rustix::fs::OFlags::RDONLY | rustix::fs::OFlags::CLOEXEC,
rustix::fs::Mode::empty(),
)?;
std::fs::remove_dir("tmp_dir")?;
// iterator gets stuck in infinite loop and memory explodes
rustix::fs::Dir::read_from(dir_fd)?
// the iterator keeps returning `Some(Err(_))`, but never halts by returning `None`
// therefore if the implementation ignores the error (or otherwise continues
// after seeing the error instead of breaking), the loop will not halt
.filter_map(|dirent_maybe_error| dirent_maybe_error.ok())
.for_each(|dirent| {
// your happy path
println!("{dirent:?}");
});
Ok(())
}