Alias reply-to address is wrong

I have setup an Office 365 account with an alias.
Whenever I sent an email using the alias (I have check Message > Reply to address is the alias one)
the email is delivered but the reply address is always the primary Office 365 account username.