The 36-year-old beauty co-hosted the pop star search show with Olly Murs last year and also presented spin-off programme 'The Xtra Factor' with the pop star.
When Justin made one of his appearances on the series Caroline and Olly found Justin's discarded pants in head judge Simon Cowell's dressing room, which the 'Sorry' hitmaker had used to take a bath.
She said: ''We found [Justin's] pants once in his dressing room at 'X Factor'! He was in Simon Cowell's dressing room, the one with the bath in it, and me and Olly were doing 'Xtra Factor' so we shared a dressing room down the other end. We all went in there afterwards and Justin had left his pants!''
Caroline has joked she thinks that 'You Don't Know Love' singer Olly, 32, took Justin's designer briefs home with him.
She added to the Daily Star newspaper: ''I think Olly's still got them actually. They were Calvin Klein - I didn't look, some of us did but I didn't...''
The only pants that Caroline will be getting her hands on in the future will be her boyfriend's Blue Logan after her relationship with the British artist became public last week.
Speaking about the romance, a source said: ''Caroline is so happy with Blue. They've been enjoying each other's company and she's very proud of his work as an illustrator, not to mention how handsome she finds him. Blue has completely won over her friends. They all approve.''
The 'Love Island' host previously admitted she was out of practice when it comes to dating, and said she'd ''forgotten'' what being in love with someone feels like.
She said: ''I haven't been in love for so long that I've forgotten what it feels like to want to tell everyone 'look how happy I am!' ''