You can then either “right click on the contact” to delete / remove it on the dropdown menu or “select the contact” and delete / remove it on the toolbar along the top.
If it’s a suggested contact that’s automatically appearing in your To: line that you want to delete see this thread for help.