Zooming in on canvas algorithm only works with a factor of 2

[RedShift]

I'm trying to create an HTML canvas where you can zoom in, but whilst zooming, the point that's being zoomed upon remains in the same spot on the viewport. The algorithm I wrote works if I set the factor to 2:

    var mousePos = mousePosInCanvas();
    var factor = 2;

    var xPan = (viewport.xPan / factor) - ((mousePos.x / factor));
    var yPan = (viewport.yPan / factor) - ((mousePos.y / factor));

    viewport.xPan = xPan;
    viewport.yPan = yPan;
    viewport.xScale = viewport.xScale * factor;
    viewport.yScale = viewport.yScale * factor;

But if I set

var factor = 1.1

the algorithm breaks. How can I make this work with a variable factor?

Code with factor 2: https://jsfiddle.net/x7s9zLys/

Code with factor 1.1: https://jsfiddle.net/x7s9zLys/1/

I've coded the scrollwheel to zoom in.