My kids are not shy, and math-wise they're ahead of where I was at the same age. I went on to get a math degree from MIT. Your story is the first time I've heard of a Montessori school being so unstructured. Do you know if it was accredited?
I don't know, sorry.. I am sure that it was the Montessori system though. They even had pictures of Dr. Montessori hanging on the walls. I haven't really looked back into that time in my life, 'til now. I'm not upset, I turned out fine, I just personally feel that in a bigger school I could have become social more quickly by interacting with a wider variety of other kids.
As with everything, different implementors may be better/worse at the same thing. I've certainly seen that with my girl (who was in Montessori as a youngster).
Also, no one approach is perfect for everybody. As frightening as it may sounds, some people need more "structure" than others.