what version of macOS are you using?
I can see the same behaviour as well, using 10.12.3
the bad news is it appears to be some weird bug in Java, (perhaps on 10.12.3, hence my question)
we use a simple FileChooser from java, where we tell it what files are allowed (a so called 'FileFilter' at you see this at the bottom) , then we tell java to process the dialog , we don't control how the dialog behaves... then once the user has selected it, we have a check to see if the extension matches the file filter, and inform the user if they are different.
( so the inconsistency is within this 'standard file handling' thats done by java)
I played with it a bit, and indeed its now really inconsistent, e.g. you cant change the file filter to patch (from subpatch) but you can change it to help, once you do that you can then change it to patch... then you can change the extension and it works....
I remember testing this, a long time ago, when I introduced the file filters, and it worked perfectly, so something has happened since (new java, new os etc.. its been so long)
I guess the only way forward is custom file dialogs, or wait until java/osx fixes it