It is easier to manufacture an external feature of size than an internal feature of size. One can use short rigid tooling vs longer cantilevered tooling - less deflection, more access and more predictability.
The ANSI tolerance tables you're referring to reflect functionality as well as manufacturability..
Ultimately end item products and components need to function as intended and if the design can be facilitated to embrace ease of manufacture - everybody wins.
More here: Design for Manufacturability and Assembly (DFM / DFMA)