There is a limit to the number of directories that can be held, no matter what the size of the data in the cache is. You can set the limit by changing the value of ls-max-cache-dirs. If the directories you have typically only have a few files, you can set this number higher, whereas if your directories typically have many files, you should consider decreasing this number.
Examples: