The musician landed a deal with Cowell's Syco management company in 2012 after winning his Tv talent show The X Factor.
He has received a barrage of criticism after becoming embroiled in a homophobia row and blasting his own promotional team, and now Arthur has reportedly been axed from the label.
It comes after he recorded a track called Follow The Leader, which critics allege features lyrics about terrorism.
A source tells Britain's The Sun on Sunday, "It seems James is self destructing. He has done himself no favours with his outbursts. It's as if he's almost willing Syco to drop him."
The singer took to Twitter.com on Saturday night (05Apr14) to insist the song is about an obsessed fan similar to Eminem's hit Stan, writing, "The song 'follow the leader' from the mixtape is not promoting terrorism, it's a story, a reciting of a Stan Esc (sic) letter from a demented fan. Just for the record."