This is incredibly fast progress on large contexts and I would like to see if they are actually attending equally as well to all of the information or there is some sparse approximation leading to intelligence/reasoning degradation.
So I tried this prompt in their chatbot arena multiple times. Each time getting the wrong answer:
"Given that Beth is Sue's sister and Arnold is Sue's father and Beth Junior is Beth's Daughter and Jacob is Arnold's Great Grandfather, who is Jacob to Beth Junior?"
This is incredibly fast progress on large contexts and I would like to see if they are actually attending equally as well to all of the information or there is some sparse approximation leading to intelligence/reasoning degradation.