I think you need to view whatever process generated the answers as part of your model. In some cases, and in all textbook examples, we have a ground truth that is correct. But in real-world applications, such as segmentation problem in medial imaging, we have a gold standard which represents our best estimate, but is not necessarily correct.
Validation is not a magic bullet, we need to be critical of any part of the model that is given as truth, otherwise we might end up fitting a solution to the wrong problem.
More generally I think that textbooks should emphasize the need for the scientific method and stress that any model (or theory) is only as good as its ability to explain the entire problem domain.
Validation is not a magic bullet, we need to be critical of any part of the model that is given as truth, otherwise we might end up fitting a solution to the wrong problem.
More generally I think that textbooks should emphasize the need for the scientific method and stress that any model (or theory) is only as good as its ability to explain the entire problem domain.