Either way it should be fine. However, I would recommend you all going together for the VISA interview - sometimes VO might not ask anything to the H4 applicants :)
In case you are not aware, you don't have to take your child for VISA interview if he'she is less than 7 years of age.