How to Build a Searchable File Index at Scale

How many files do you have on your computer? How much total space do they take up? Can you, armed with only the file name, find a specific file that you worked on 3 years ago? If your disk is getting full, which files would you delete or move first, in order to free up space?

Chances are that you know where to go to answer these questions, especially if you have file management software. These tools usually build a metadata index, a data structure that maps file names to metadata like access time, modified time, file size etc., and allows for aggregation and filter queries on this metadata. For example, a metadata index could quickly answer the question “what is the total size of all .mp3 files in this directory?”. …

About

Sudarshan Muralidhar

Software engineer at Igneous. Cofounder of Upbeat Music App. I do cloud things.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store