Some others have said some good things, but I’ll add a few more comments:
Objects will continue moving in the same direction at constant velocity unless acted upon by an outside force. In the circular motion case, as you correctly state, the velocity vector is always perpendicular to the acceleration vector, which does point inwards towards the center of the circle. The object IS accelerating inwards – the water in the bucket in your example is accelerating inwards. It’s just that it is also already moving tangential to the circle and the result of the inwards acceleration is the object turning to follow the circular trajectory. It’s distance from the center remains constant instead of increasing.
This is all the ideas of centripetal acceleration from a stationary frame of reference outside of the water-bucket system. There is no extra centripetal force. In your water-bucket example, there is a force the bucket applies against the water to give the water its centripetal acceleration. Some people call this a centripetal force, but I don’t like that phrasing, because it sounds like its an extra force. A water-filled bucket sitting on the ground also exerts a force upwards on the bucket. We typically call this a normal force, so in the spinning example, the normal force of the bucket against the water is causing the water to move in a circle. Likewise, the string tied to the bucket - the string is exerting an inwards tension force on the bucket that causes the bucket to travel in a circle, and thus have centripetal acceleration. If you cut the string, the bucket stops turning and instead goes in the straight path in the direction of its velocity vector.
Now, think of the fictitious centrifugal force as from the frame of reference of someone moving in the circle. I like to use the example of you sitting in a school bus with those slick plastic-leather seats as the bus goes around a turn. You “feel” like you’re being pushed outwards. In actuality, there is insufficient frictional force to keep you turning in the same circle as the bus, so you are traveling in the straight line that your velocity vector is pointing. Thus you “feel” like you are being pushed outwards. The term centrifugal force arises because that “feel” can be modeled mathematically from the perspective of the rotating object, and we call it a centrifugal force. This term is mathematically equivalent to the centripetal acceleration term from the previous stationary frame of reference.
Planescape setting has it for you already! Bytopia.