The problem with email authentication systems isn't the systems themselves usually, you can design a good email authentication system that has a good secret with strong crypto and no leaks and all that.  It's not a trivial task but it isn't terribly difficult either.

The problem with email authentication systems is that most common email providers' own security less resemble plates of steel armour and more resembles a piece of swiss cheese.  If even Google Mail gets compromised frequently, what are companies with less resources going to do?

The email account becomes the key.  Maybe its a very strong Abloy Medeco key that is impossible to forge without specialized tools and a non-considerable amount of time.  That doesn't matter if the key is under the welcome mat.