When can I put my doe rabbit and 2 wk old bunnies back with their neutered father?

My doe has had 4 healthy bunnies in my dining room, they are now eating pellets gently and looking well, we have worked out they are 2 weeks old, mum is happy too. I know general recommendation is not to put them back with the dad, but he has been neutered - will this make a difference. I intend to check sex at 6 weeks with vet and then sell/neuter as appropriate.