This is very interesting, thank you for sharing! I do have one tangential comment based on this:
> I continued to use [the interview question] because I got good signal from the candidates.
Working at Google, this is something I hear a lot: that some question, or other interview thing, is good for getting "signal" from candidates. But how do you actually know?
Like, I feel to definitely know a question is good you would actually need to track candidates after they start working and see if their performance on the question correlates with their job performance. (Unfortunately, it seems impossible to track candidates who don't get an offer.)
Most people who say this think the question gives them a good ability to distinguish whether a candidate will be good if hired into a given role, however honestly mostly it gives them a good sense of whether they like the person and/or the person is like them.
As an example, the author of TFA clearly is selecting for people who have the patience to sit around and be told how smart the author is. That is the signal in this “great interview question”: candidate will suck it up for hours while I tell them stuff.
Thanks for the comment. By saying "I get good signal" I mean that it's not a binary yes/no, but gives a range to score the candidate. You are right about actual correlation to job performance. I know the HR at Google performed correlation analysis between interview scores and job performance and made internal observations, which as you note is biased because only includes hired candidates.
At google you used to see scores and feedback of everyone on the panel and whether hire/no hire decision was made by the Hiring Committee (which you otherwise didn't have any visibility into except sometimes they would leaver you feedback). You could also see your histogram of scores as well as other peoples' on the panel colored by recency and hiring decision which was very useful to calibrate your internal sense. Something so simple can go a long way and I haven't seen it in any of those crappy interviewing saas products out there. Did they scrap that system?
> I continued to use [the interview question] because I got good signal from the candidates.
Working at Google, this is something I hear a lot: that some question, or other interview thing, is good for getting "signal" from candidates. But how do you actually know?
Like, I feel to definitely know a question is good you would actually need to track candidates after they start working and see if their performance on the question correlates with their job performance. (Unfortunately, it seems impossible to track candidates who don't get an offer.)