The origin of whales (order Cetacea) is one of the best-documented examples of macroevolutionary change in vertebrates. As the earliest whales became obligately marine, all of their organ systems adapted to the new environment. The fossil record indicates that this evolutionary transition took less than 15 million years, and that different organ systems followed different evolutionary trajectories. Here we document the evolutionary changes that took place in the sound transmission mechanism of the outer and middle ear in early whales. Sound transmission mechanisms change early on in whale evolution and pass through a stage (in pakicetids) in which hearing in both air and water is unsophisticated. This intermediate stage is soon abandoned and is replaced (in remingtonocetids and protocetids) by a sound transmission mechanism similar to that in modern toothed whales. The mechanism of these fossil whales lacks sophistication, and still retains some of the key elements that land mammals use to hear airborne sound.