Ideally you should care about the best papers. You only have time and energy to go to a fraction of the talks anyway so you can just skip the weaker papers. You'll learn the most from the best papers in the conference.
But how do you distinguish the best from the worst? The program committees purposely don't releases any ranking information of the accepted papers beyond a few award winners. So if the worst papers are pretty good this gives a lower bound on any talk you attend.
But the real answer lies in the fact that we don't care about conferences because of the papers that we want to see but for what it does for our papers if they appear there. If the worst papers are very good and your paper gets accepted this implies some quality level about your paper. And in the end we want conferences that make our papers look good. And so we focus more on the worst than the best. Yet another paradox of the CS conference system.