The Tv personality let the big news slip on the red carpet at One Direction's tour movie premiere in London on Tuesday (20Aug13), confessing he'd like to take the singer's place.
O'Leary said, "He'll (Barlow) be leaving at the end of this year... (but) I'd love to judge. I'd be so strict. I'd be nice and strict."
Barlow has yet to confirm the news of his departure, but he recently stated that it was "too early" to comment on whether he would return to The X Factor for the show's 11th season.
He'll be back as a judge when the 10th season debuts in Britain on 31 August (13), when he'll be joined by Nicole Scherzinger, Louis Walsh and Sharon Osbourne.