The problem there is that natural language processing is a highly specialized field, not something one can expect a generic SRE to know unless it's explicitly mentioned in the posting. I'm willing to bet the posting said nothing about NLP.

I could ask a detailed question on cryptography, robotics or reverse-engineering, and it would be just as (ir)relevant to the SRE. Computer science is a much broader field than your niche area. Arbitrary goal-posts are arbitrary.

Ah yes, I whack out four or five natural language input tokenizers before lunch most days.

Then this problem is solved by a gazillion of libraries already, no?

This problem was solved by medieval monks by introducing white spaces.

Many languages still don't put whitespace between some or all words (Chinese, Japanese). Many languages tend to use long compound words which are better represented as sequences of tokens rather than one token (German, Russian). Many inputs are dirty (coming from OCR, for example) so whitespace can be unreliable. And even in clean English text, inflected words may be better tokenized as sequences of tokens rather than a single token.