One way of characterizing the difference between traditional empiricism and traditional rationalism is that traditional rationalists have been dazzled by the impressive certainty of our a priori knowledge. Logic and mathematics are so remarkable that many rationalists have literally accorded them the status of magic, attributing them to some mystical contact with the divine. It is likely not a mere rhetorical device when Parmenides presents his logical arguments as having been given to him by a goddess. Plato accords his forms a kind of divine status, and says we know them from previous exposure to them when our souls were in a higher realm of existence; Descartes says what may amount to the same thing, that mathematical knowledge and a few other items were implanted in our souls by God.
Those in the rationalist tradition have also almost always classified ethics as a priori. Of course, lots of specific reasons for that could be given, but there are also very general motivations from the rationalist tradition which push that way. First, of course, rationalists have generally been imperialists when it comes to the application of reason; since a priori knowledge is the really good knowledge, the rationalists have sought to reduce everything, or at least as much as possible, to the a priori, to have the best possible knowledge of everything. Further, ethics specifically is about what's valuable, important, and good, and when it comes to knowledge the a priori is, according to the rationalist, the most valuable, important, and good, so while this does not itself amount to a rational argument, there seems to be some affinity between ethics and the a priori. This further connection is no doubt enhanced by the tradition of connecting moral good to the divine; since the rationalists also connected the a priori to the divine, this would further encourage bundling the two together.
Of course, the mainstream of the empiricist tradition has long maintained that the reason logic and mathematics have their apparent infallibility is that they are not actually giving us information about the world; since they don't tell us how things stand with the world, the world cannot refute them. But the empiricists insist that real truth is about the world, so these a priori matters the rationalists regard with such enthusiasm are at best some kind of honorary truth. A priori claims embody useful tools, ways of thinking about the world, but don't report facts. The rationalist project of relying only on the a priori is, from the empiricist perspective, a project of ignoring the real world, of casting aside the only truth worth looking for.
Empiricists have thus traditionally sought to reduce the scope of the a priori, rejecting for example the a priori approaches to science championed by some of the rationalists. It is perhaps for this reason that some empiricists have tried to argue against a priori ethics as well, saying that we need to be more naturalistic in our approach to ethical matters.
However, there seems to be another possible reaction, which I'm surprised hasn't been more common. Many empiricists have also been meta-ethical subjectivists (Hume being probably the most famous example). Such empiricists should find it quite congenial to categorize ethical claims as honorary truths, useful tools which don't reports facts about the world. So why is it so rare for empiricists to treat ethics as a priori, just like logic and mathematics? A good reason does not occur to me. I can think of some bad reasons; perhaps even empiricists are partly under the spell of the apparent certainty of logic and mathematics. Thus, perhaps they ignore the historical controversies in logic and mathematics, and think that the controversies in ethics show that ethics must be something entirely different from our stable logic and mathematics.
Actually, on one interpretation Kant might be an example of the sort of philosophy I think should be more common. Of course, Kant claims to chart a third way, neither empiricist nor rationalist, but it has been very common to be skeptical of this. Many interpreters take him to have simply been either a sneaky rationalist or, less commonly, a sneaky empiricist. If he was a sneaky empiricist, he was an empiricist of the rare kind I've been puzzling about. I wonder if the fact that people generally don't connect a priori ethics to empiricism has contributed to the empiricist interpretation of Kant being the less common reading.