I am tall and thin

Today I went to this office to deliver a document and in the corridor I asked the first person I saw,"Excuse me do you know which room is Ms. TMZ's office"
She replied "I am her!"
"O'sorry, sorry ....good to see you"I said ,she added"in years I knew you by voice over the phone,I always thought you were some very tall and very thin guy, like 6 plus feet"then she smiled.
"I am sorry,I am not that person"I said with a smile too (you know business etiquette).

Anyway,the earlier unknown is known now, but I have this question :
Can we guess people's physique,race,colour by talking over the phone?
I have been more wrong then right many times,do you have any such experiences too!
From now onwards,I will not assume anything.
I will just ask: "let me guess, you must be very pretty"