The issue of multiple comparisons is a really big topic. There have been many opinions and many disagreements. This is due to many things; among others, it is partly because the issue is really important, and partly because there really is no ultimate rule or criterion. Take a prototypical case: You conduct an experiment with
manpreet
Best Answer
2 years ago
When trying to evaluate what constitutes a family of hypotheses within an experiment/project/analysis, I've found "similar in purpose" and "similar in content" given as guidelines for delimiting families, but these leave quite a lot open to interpretation (to say the least).
It seems clear that if in the course of an analysis, I do several tests of group means and a separate batch of tests of homogeneity of proportions, that I wouldn't bundle everything up together into a single family of hypotheses.
However, if I have several batches of somewhat related tests of group means, what criterion brings them together into a family (or splits them into separate families)? Should all members of a family have the same response variable? If I had different response variables but the same set of cases involved, would those all get bundled into a family of hypotheses?