How to Build a Searchable File Index at Scale
How many files do you have on your computer? How much total space do they take up? Can you, armed with only the file name, find a specific file that you worked on 3 years ago? If your disk is getting full, which files would you delete or move first, in order to free up space?
Chances are that you know where to go to answer these questions, especially if you have file management software. These tools usually build a metadata index, a data structure that maps file names to metadata like access time, modified time, file size etc., and allows for aggregation and filter queries on this metadata. For example, a metadata index could quickly answer the question “what is the total size of all .mp3 …
Why we built our own NFS client in Golang
The first step in building a metadata index, as you might imagine, is actually getting the file metadata so that we can index it. This is easy — all we have to do is mount the NFS server:
sudo mount nfs.host.com:/nfs/mount/path localmount
and then write code to walk the
localmount directory, collecting metadata.
Ah, if only it had been that easy, we would have saved many late nights and I could end this article here. The problem with reading from a mounted volume is that you’re limited by what the kernel’s NFS client can do. And the Linux kernel’s NFS client implementation is not exactly built for high performance. We can figure out what the NFS client is doing by inspecting the network traffic. …