The most basic way to put this is that not all electromagnetic (EM) waves interact the same with the rest of the world.
All radio frequency (RF) waves are EM waves, but so are a lot of other things, light, microwaves, etc. Light for example is an EM wave and easier to understand conceptually. RF is the same except lower in frequency. The most obvious parallel to light is the 'direct' wave. Think of your radio like a flashlight and the observer only sees the beam directly (ignoring reflected light for the moment). RF works this way, too, on a point-to-point direct path.
The interesting part, you remember from physics that blue and red are on opposite sides of the light spectrum? The light spectrum has a range of wavelengths. You can see from prisms how colors get separated physically due to various amounts of bending (called refraction). This is the light's wavelength reaction through a particular medium. RF does this too, react within the medium in which it travels. Light is just a relatively narrow range which is why the spectrum seems continuous to our eyes.
The VHF part of the spectrum is something all hams know. These frequencies do not often (they can, just rarely) refract in the earth's atmosphere, so they pass through into outer space. This would be roughly similar to some parts of the EM spectrum going right through the prism and not bending. The medium of glass or plastic might conceptually be invisible so-to-speak to UV, for example. This is true of VHF and basically always true of UHF to the ionosphere, where it is essentially invisible and goes right through. That's why VHF/UHF are substantially line of sight (or direct wave) bands and only work if you can basically 'see' the other station. VHF is affected by reflection, which is why you get multipath interference with it. But RF reflection happens from objects and not an ionospheric charge. You do get some reflection from the atmosphere from dust for example, just not much of it and it's basically not useful for communication.
What happens as you go down in frequency (up in wavelength) is that EM waves begin to 'see' the ionosphere and begin to interact with it rather than passing through it. They seem to reflect off of it (it's actually refraction, just like light in the prism). This begins to happen quite rarely on 6m and more periodically on 10m. It happens reliably when you get down to 20m, etc. It does work down to the longer wavelengths, though other things (like atmospheric absorption) begin to change how well. At some point the wavelength becomes so much longer than the height of the atmosphere over the surface of the earth that it become 'invisible' again and no longer refracts. For example, 160m is only marginally refracting in the ionosphere.
All of this mostly depends on the sun. During the day we have 4 layers (D, E, and two in the F) in the ionosphere and at night some of them collapse, the F1 and F2 become one layer and the D layer disappears. Effectiveness of refraction varies with how many charged particles are available and what layers of the ionosphere are present. What you count on is the majority of energy during refraction going back to earth. This is where a beyond-the-horizon contact is made. Waves travel better during the day with the sun charging ions, although this also creates more noise. At night the atmosphere is quieter but there are fewer ions available for refraction and the layers having collapsed provide a narrower range of potential frequencies. Basically in this you can think of the sun and ionosphere as varying composition of glass in a light prism.
Sometimes it then reflects (actually reflects in this case) back off the ground and refracts in the ionosphere again. When it does the ionosphere-ground boogie several times, called hops, a wave can travel a long distance. The most effective wavelengths for the ionospheric phenomenon move up and down over the day and around the year, but generally the best DX chances lies within 40m to 15m. It's here where QRP has the best chance to go long distances because your signal does not lose much energy along the way. It might only happen efficiently at a very narrow range of wavelengths at any instant in time or it can happen over a large range.
This barely scratches the surface of RF propagation (reflection, polarization, incident angle, wave penetration depth, ionic charge density for example) but maybe will get you going to investigate it some. There are all kinds of good books about RF propagation, whole textbooks about it. I personally like Seybold's book, but the one the ARRL sells by Poole isn't bad either.
And BTW, yes, you have to put on a long antenna if you want to work mobile HF effectively. Just the nature of the beast. There are some laws that you just cannot break no matter how hard you push the throttle pedal.