That's not something to worry about too much IMHO. it's just engine braking. (braking, not breaking)
Once the engine gets spinning fast enough, it can no longer help you accelerate er even keep going the same speed. So since your engine can't keep up, and the wheels are still attached to the engine (since the clutch is engages), the engine slows you down.
Truck drivers do that down big hills. Shift into first gear and don't waste your brakes.
I wouldn't worry too much about it.