This post explores the question of how justified this division is.
The starting point for my thinking is something I learned from the wonderful essay In Oldenburg's Long Shadow about the serial pricing crisis.
To understand the serial pricing crisis you must first understand the science citation index. This is nothing more or less than an index of how many times a given paper has been cited by other published papers. When you look at it, you find that papers that appear in some journals are consistently cited more often than others. This is a direct measurement of how influential the journal is, and leads to the impact factor. When you look at journals across math and the hard sciences you find that there is a hierarchy of journals. At the bottom you have low impact journals that publish only for a small niche. But key papers from that niche are published in more prominent journals that are looked at by people in a wider range of subjects. And this goes all of the way to the highest impact journal of all, Nature, which is where people try to publish the absolute best work across all of math and the hard sciences. It doesn't matter whether you're a physicist or a biologist, the absolute best research goes to Nature.
So from the science citation index we can measure the impact factor of a journal, which in turn tells us its value to researchers. The value to researchers told publishers what universities were willing to pay, and so publishers have been steadily increasing the price of the most important journals. This costs universities more than they are happy with, and so is called a crisis. Librarians call journals "serials" (because you get a series of copies of a journal), hence this crisis is called the serial pricing crisis.
Now let's look at this in reverse. Across all of math and the hard sciences it is possible to make a somewhat reasonable comparison of how important any given paper is, and how good any given journal is. Furthermore there are small groups of editors whose judgment is regularly trusted to compare the best papers from different areas of science and select which are worthy to be in their journals. In the extreme example, the editors of Nature are trusted to draw comparison across all of the hard sciences. And for the most part, scientists agree with these decisions.
When you think about it, it is truly remarkable. It implies that there is a relatively well shared concept of relative value across all of the sciences. Of course people in the hard sciences seldom remark on it, it is just how things are.
To see how remarkable it is, compare with the humanities and social sciences. They have no such hierarchy. Instead of one grand hierarchy you get independent clumps of researchers who talk to each other but not the other groups. And they find this so natural that I have seen social scientists express disbelief that, say, a physicist in fluid mechanics can hear a key result in particle physics and will know that it is important. But it is true. If you ask one physicist, "What are the 10 most important results in physics in the last 30 years" and take that list to another, the other physicist will agree that those are all important. If you ask a psychologist for a similar list and take it to another, the other is likely to not even recognize many of the items.
What is going on here is a confirmation of one of Thomas Kuhn's key claims in The Structure of Scientific Revolutions. Which is that in a mature science (his term) researchers have come to share a paradigm about what would be progress. When a paradigm has become so compelling that virtually all researchers in the area accept it, then people who are not in that field can see the agreement that progress is happening. When no paradigm can compel general acceptance, then from a distance all that is visible is confusion.
Kuhn is careful to point out that within the field there will be groups of researchers who are doing good work and making progress. This is certainly is true. For instance I've brought up psychology. Yet if you read books like Parenting From the Inside Out you will find that solid research is being done, that comes up with valuable information. (I highly recommend this book to anyone, parent or not, who is willing to work through it carefully.) But the problem is that the case for this line of research is not compelling enough to convince other psychologists that this is the right way to try to understand the mind. So from a distance there isn't a clear impression of solid progress being made.
Therefore the hard/soft science division boils down to shared paradigms. In the hard sciences certain lines of research have become so compelling that everyone agrees that they are the right way to go. Because of this agreement, people in nearby fields get a clear picture of what progress looks like in that field. With clear pictures of what progress looks like in multiple fields, the ground is set for making comparisons between fields, which has evolved into a reasonably well shared value system across the entirety of the hard sciences.
The soft sciences share none of this structure. As a result there is no shared agreement within the soft science about what is important, let alone a shared agreement on the relative importance of different areas of science.
To close I would like to illustrate how much shared agreement there is within the hard science about what progress looks like. I'll do this by giving my personal top 10 lists of scientific advances in each century since science began to take off in the 1600s. I haven't tried to put them in any particular order. (They often are somewhat chronological.) While people may quibble with some of my specific choices, people who are well versed in the hard sciences will generally agree on the importance of these items.
- Objects of different mass fall at the same rate (Galileo, physics)
- Telescope used for astronomy (Galileo, astronomy)
- Kepler's laws for planetary orbits (Kepler, astronomy)
- Circulatory system accurately described (Harvey, biology)
- Microbes discovered (Leeuwenhoek, biology)
- Hooke's law of elasticity (Hooke, physics)
- Newton's laws of motion (Newton, physics)
- Newton's law of gravity (Newton, physics)
- Speed of light first measured (Ole Römer, astronomy/physics)
- Calculus (Newton/Leibniz, mathematics)
- Lightning explained as static electricity (Ben Franklin, physics)
- Fluid mechanics began to be analyzed (Bernoulli, physics)
- Linnaean taxonomy system created (Linnaeus, biology)
- Halley's comet's orbit predicted (Halley, astronomy)
- Coulomb's law for attraction of electric charges (Coulomb, physics)
- Oxygen discovered (Priestly/Scheele, chemistry) leading to the rejection of pholostigon (Lavoisier)
- Uranus discovered (William Herschel, astronomy)
- Conservation of mass demonstrated (Lavoisier, chemistry)
- Stability of solar system confirmed (Laplace, astronomy)
- Gravitational constant measured (Cavendish, physics)
- Fourier series discovered, used to analyze heat transport (Joseph Fourier, mathematics/physics)
- Ice ages discovered, theory of The Flood rejected (Louis Agassiz, geology)
- Central Limit Theorem aka The Bell Curve (de Moivre/Laplace/Galton/Lyapunov etc, statistics) different versions were proven at different times, and Galton was making good use of it years before it was finally proven in generality by Lyapunov
- Thermodynamics (many people starting with Carnot, physics)
- Conservation of Energy (Joule/Mayer, physics)
- Descent with Modification aka Evolution (Darwin, biology)
- Germ theory (Pasteur, biology)
- Atomic theory (Avagadro/Loschmidt etc, chemistry)
- Maxwell's equations of electromagnetism (James Maxwell, physics)
- Periodic table (Mendeleev, chemistry)
- Relativity (Einstein, physics)
- Radioactive Dating (Ernest Rutherford/Bertrand Boltwood, physics)
- Quantum Mechanics (Heisenberg/Schrödinger, physics)
- Gödel's Incompleteness Theorem (Kurt Gödel, mathematics)
- Hypothesis Testing (Ronald Fisher/Jerzy Neyman/Karl Pearson/Egon Pearson, statistics) - Egon was Karl's son
- The Structure of DNA (Watson/Crick/Franklin, biology)
- Continental Drift (proposed Wegener and confirmed by lots of people at once, geology)
- The Big Bang (Georges Lemaître/Edwin Hubble, astronomy) general acceptance followed the discovery of the CMBR by Arno Penzias and Robert Wilson
- Synthesis of the Elements in Stars aka B2FH (Geoffrey Burbidge/Margaret Burbidge/William Fowler/Fred Hoyle, astronomy/physics)
- Standard Model (Sheldon Glashow/Steven Weinberg/Abdus Salam, physics) tens of billions of dollars have been spent verifying this theory!
- 2000s - There is likely more disagreement over these
- Human Genome Project
- Neutrino oscillation
- Rapidly improving knowledge of planets around other stars
- Poincare conjecture solved
- Age of universe measured to within 1% accuracy (it is 13.7 billion years old)
- FOXP2 critical for language
- Preserved soft tissue from dinosaur?
- Stem cells from skin cells
- New family of high temperature superconductors
- Molecular evolution is irreversible