If you look at the html source a h:commandButton creates, it is creating a form submit button (type="submit") which will always submit the form/page.
If you want a button that doesn't submit the page, you probably don't want a h:commandButton. You could create your own custom component that creates html like:
<input type="button" onclick="put javascript here to close the other window" />
Or alternatively, just use plain html for this button and surround it in <verbatim> </verbatim> tags.
If by action you are talking about a method in the backing bean, then it will require a page submit.
Another way to get a command button to not submit:
<h:commandButton value="test" onclick="closeWindow();return false;" />
If you want the command button to perform an action, ie something like:
<h:commandButton value="test" action="#{someBackingBean.someMethod}" />
then it will *require* a page submit as the backing bean method code is on the web server, not the user's web browser. There's no way around it.