I am unable to edit it so I am replying with the updated code , I managed to do it by setting viewport minimum, but If i pan back its causing problem , please check this codepen – https://codepen.io/adjmpw/pen/ZMMGpY

If i click pan tool and try to move back its acting weird , zooming out viewports..