Yes, if a director and SVP are discussing the technical details of an incident, I would be concerned -- they should be concerned with putting the right people and processes in place to make those decisions.
When "Something had gone wrong that shouldn't have" bubbles up that far, you have an organizational issue, not a technical issue.
If the person who is charged with resolving the issue is capable, there's no point in a senior manager knowing the details outside of professional curiosity.
That requires you to decide what a "word" is, which is not trivial (if you think that ignoring punctuation gets you to a clean "letters surrounded by spaces" you will get lots of issues with various Asian languages)
Also some languages have a lot of prefixes and suffixes on their verbs or even nouns, which dilutes your list of 1000 words by just adding the same common words over and over again with different suffixes designating grammatical tense, grammatical gender, etc.
The gzip version sounds more general and more obviously correct
Instead of deciding on words, maybe you can break them up into smaller, subword parts – let's call them "quantums". And these quantums can be the units the quantumizer works on to operate on inputs and outputs. We can then use them to build Expansive Dictionary Models, or EDMs. I suppose we'd need a software library to mak working on this easier, think something speedy, fast, hot, like fire: we can call it PHPFlame...
Byte Pair Encoding [1] will be different for different languages. Application of the per-language BPEs to the input text will produce encodings with different lengths.
It naturally takes care of common prefixes and suffixes.
It is easy and fast to apply using radix tree or with finite automata. Even without radix tree, it is possible to have processing speed in the range of hundredths of thousands of bytes per second.
FSST is based on a fixed size (255 items) dictionary of high frequency variable length strings/substrings (learned from the corpus) encoded as one byte.
Either that call would have to do the same (i.e., walking the files and counting), or you'd need some additional metadata in the directory entry to store how many files there are, requiring additional storage accesses for adding and removing files. Adding to that that both FAT32 and NTFS are quite old and had to run on older hardware. Cycles and disk accesses are not free.
On top of that, how often is it necessary to efficiently know the number of files in a directory while at the same time not caring about the files enough to list or display them? This algorithm is a special case where you could use the count of using a bit simpler code that ultimately would have the same file system API calls (since you cannot tell the FS to give you file #37 from that directory, so you'd have to use FindNextFile 37 times anyway, just like the sampling algorithm).
Not that it counters any point you're making, but ZFS displays the number of contained entries of a directory in the directory's size field; mind that . and .. are included, so you usually need to subtract 2 to get the count you actually want. I do find it useful sometimes to know the count without getting the listing; the former is a very inexpensive operation (since ZFS is keeping track of metadata like you suggested), the latter is expensive, potentially extremely with hundreds of thousands or more of entries.
This is more-or-less unique to ZFS. Other file systems even on Linux and FreeBSD generally don't provide this behavior.
> or you'd need some additional metadata in the directory entry to store how many files there are, requiring additional storage accesses for adding and removing files.
Most unix filesystems use inodes. inodes have the same format whether they represent a file or directory. So directory have a (usually unused) size member. NTFS doesn't use inodes, but the records in the MFT work the same way.
When adding or removing a file from a directory, you have to update the modification time of the folder, so you have to rewrite the entire inode anyway. Updating the size/file count at the same time would be free.
In my opinion the likely reason why file count isn't tracked is a lot more pedestrian: It wasn't tracked initially and we can never add it to existing file systems because the metadata would get out of sync if the FS was mounted on a kernel with no count support.
I base this assumption on the fact that many modern file systems do indeed keep track of the count.
reply