I’m on a mission to find the fastest way to copy and delete files on a modern machine. I thought a quick Google search would reveal the answer, but I only found this Stack Overflow post with people’s opinion on the matter (but no proof), a discussion on what the kernel can do to improve performance, and various articles claiming you should tar directories to copy them faster (there’s no chance of that being faster than a properly written parallel cp implementation).

The only way to get an objective answer is by writing benchmarks, so here they are. 😁 The results will be highly dependent on your specific system, so I would recommend simply running those benchmarks yourself if you’re curious. However, I’ll share my findings from running these on my ext4 XPS 17 and an APFS Macbook.

Findings #

Honestly, the results are a little disappointing. With very small files (1 KiB), the fastest option is to use copy_file_range¹, but that’s expected since you’re going from four syscalls (open/read /write/close) to one. However, running the same benchmark with medium-sized files (1 MiB) results in a win for the mmap variant² (with direct I/O too³). Finally, with large files (32 MiB), the method of copy doesn’t matter since the disk can’t keep up⁴.

All is not lost though, as I did learn a few things:

• Write only mmaps are consistently terrible, don’t use them. If you want to use mmap, set up a read-only mmap for the from file and then copy the bytes using write with the memory address of your mmap as the buffer for the to file.
• copyfile performs extraordinarily well on macOS, presumably because APFS supports copy acceleration (meaning the files aren’t actually copied, instead only needing to update copy-on-write metadata). I’m guessing if my machine used Btrfs instead of ext4, I would see similar gains.
• Curiously enough, using fadvise with POSIX_FADV_SEQUENTIAL seemed to have no effect or make performance slightly worse. I’m not quite sure why that is — if I had to guess, I would say that being I/O bound means it doesn’t matter how far the kernel tries to read ahead because it’ll never fill its readahead buffer.
• Using fallocate or truncate to set the file length upfront (and thereby minimize metadata updates) improves performance on Linux, but seems to have no effect or slightly worsen performance on macOS.

My main takeaway is that you should just use copy_file_range (or copyfile on macOS) where possible since it’s good enough, will handle stuff like fadvise and fallocate for you, uses copy acceleration when the file system supports it, and will come with free improvements as the kernel is updated.

Sample benchmarks #

Copying 1 KiB files with the Kernel cache enabled #

↩

Copying 1 MiB files with the Kernel cache enabled #

↩

Copying 1 MiB files with the Kernel cache disabled for reads #

↩

Copying 32 MiB files with the Kernel cache enabled #

Ignore the extra probability of copy_file_range completing in 20ms, that’s just happens to be the first benchmark that ran before the system reached saturation.

↩