You are correct about the fact that electrons “shift” between holes must be equal to the speed at which they “move in the opposite direction. It isn’t, and no one says so.
But, you need to realize that it is not THESE ELECTRONS that we are referring to when we say that electrons are faster that holes. These electrons are bonded because they are located in the Valence band.
We refer to the electrons within the conduction band when we use the term electron current.
This is how you should think about it
There are two rooms with 50 chairs each.
The one room can accommodate approximately 45 people and the other about 5 people.
There are five empty chairs in the room where 45 people are present. Imagine these people moving from one chair or another. They must move slowly because the room is full. The speed with which they move is the exact same as the speed that empty chairs seem to move. It is easier to track empty chairs than people, so that’s what we do.
However, you can see that there are many chairs in the room. This allows people to move freely and gives them higher speeds. These are the guys we’re referring to when we say electrons travel at higher speeds.
We hope you find the analogy useful!