In order to delete a contact, you must access its profile from your active contacts section by clicking on its name.
Then you can delete it clicking the Delete button on its profile view.
Once deleted it will disappear from your Active list, and you will also disappear from its Active list.