Indeed. You also keep your German or Austrian citizenship if you did not apply for the other citizenship (e.g. if someone was born from German parents in Australia).

In general different countries have (very) different sets of rules, see e.g. http://en.wikipedia.org/wiki/Multiple_citizenship