Not sure if it's a bug but it seems like camera property idleRotationWaitTime is not working.
Look at this playground, whatever the time you put, it always wait 2-3 seconds before starting moving again :
https://www.babylonjs-playground.com/#6FBD14#75