Out of Bounds (OoB for short) refers to the space uninhabitable by Mario under expected gameplay conditions. It is technically the default state of collision interaction for Mario, since OoB occurs in the absence of a floor triangle beneath Mario.
The Super Mario 64 engine attempts to prevent Mario from moving out of bounds by canceling movement into it. This was explained by certain strategy guides as Mario reaching the border of the painting world. Mario will push or bonk "against" OoB, regardless of direction, if his intended next position lies within it. Unlike walls, OoB preserves Mario's angle rather than reflecting it as he bonks, and Mario cannot punch, kick, or Wall Jump off of the OoB region. Colliding with OoB also produces a different sound effect than colliding with walls.
As a failsafe, Mario immediately loses a life if he manages to move out of bounds. Methods to do this include recovering from a ledge grab into OoB, forcing the the only floor below Mario (i.e. a floor triangle of an object) to become intangible, being thrown by Chuckya, reading a sign clone, warping via teleporter clone, and using a door from below the intended height.