Joel Madden wants to have a another baby with Nicole Richie.

The Good Charlotte rocker, who already has daughter, Harlow, six, and son, Sparrow, five, with his wife, is eager for them to expand their family, but she isn't convinced it's a good time.

A source told National Enquirer magazine: ''Nicole isn't so sure she wants any more kids right now, but she's playing along.

''Her fear is that if she gets pregnant, Joel will wants a fourth one right away, so the baby has another sibling close in age.

''The fact that Joel wants not one, but two, more children has her beside herself.''

The 'I Just Wanna Live' singer, 35, has reportedly been desperate to have more children for some time although last year, Nicole, 32, admitted she isn't certain if she wants to add to her brood.

She said: ''I'm just chillin'. Let me just chill! I really like that they're doing their own thing. They're at school until 3. I don't have to, like, look after them -- or even think about them! It's like every other day.''