In general, you can travel, but if you plan to return on H4 visa and re-enter US, you will have to ensure that your I-94 is updated after the extension and COS is properly reflected. Because, if you travel when your extension or COS is filed, the COS status is abandoned.
If you enter on F1, there is nothing to worry, your status is F1 and your H4 does not matter.
F1 Visa stamping has no relationship to your H4. Make sure you carry all documents related to F1 and remember your spouse's H1B documents may not help here as this is a totally different visa type.
Either case, you will need to speak to your H1B, H4 filing immigration attorney and explain your case and seek their advice. They will give you advice, if there are anything that you need to consider and be sure.