In some situations, this is a restriction which limits the user functionality and does not make sense. E.g. if you are using a range bar chart to display events on a time range and there are e.g. only 2 events within 24 hours (full time range), but the initial viewport is only the last 10 hours: In that situation it should be possible to use the panning functionality to move back to the first event which may actually be outside the viewport (e.g. at the beginning of the 24h range). But due to the restriction you described, panning and zooming is not possible.
.
