The 22-year-old singer - who called time on his nine-month relationship with the 'Twilight' actress in March - admitted he is still feeling the affects of the end of the romance, which left him feeling devastated.
He explained: "I've been heartbroken. People like to think it's only girls that wind up heartbroken, but I've also had it happen to me."
Despite having also been in relationships with the likes of Demi Lovato, Camilla Belle and Taylor Swift - who claims he dumped her via a 27-second phone call - Joe insists he is "not always" the one who calls time on his romances.
He added to Twist magazine: "I have been the one to end relationships. Sometimes, but not always."
Ashley has previously admitted she and Joe were close friends before their romance developed, which she thinks is the "most important thing" in a relationship.
Asked if she felt love at first sight with Joe, Ashley said: "A Friendship developed first, and that's the most important thing for me. If you are going to be with someone all the time, they need to be your best friend."