But is there a way to judge these things formally? And should there be?Be aware of Goodhart's Law:
When a measure becomes a target, it ceases to be a good measure.One way to measure how good a journal or conference is impact factor which is based on number-of-citations. Does this work well? Some issues:
- Even with an honest effort these things are hard to do well. People may cite the conference version, the journal version, the arXiv version, or just give a website.
- Even with an honest effort just a few articles can skew the results.
- Its hard to compare cross-fields. I suspect that pure math journals have lower citations rates than biology journals.
- If an author cites himself, should that count?
- The above points were all about HONEST efforts. Could a journal do things to boost its impact factor and then brag about how high their impact factor is? Would they? Alas yes, see this article and this article.
I COULD rant that we should all be well rounded enough to read outside of our field, but I know how hard that is.
I COULD rant about the dishonesty pointed out in the above links, but that's only part of the problem.
I COULD say What do you think? so I will.