that totally depends on what you are trying to achieve...
as I said inlets are not polyphonic, so that's why (currently) midi is placed inside.
for everything else, of course, anything that you don't want to be shared across voices, has to be in the patch ... so its 'common' to want to have oscillators, filters, envelopes etc per voice
however, for somethings (most obviously effects) you will want these to be 'global' , often because you want to save resources, so having the per voice is expensive (especially at higher voice counts)
examples include, wavetables, reverbs
also bare in mind, that sub-patches can filter midi channels etc...
so your 'main patch' may actually have many (potentially polyphonic) sub patches, for different midi channels, or splits/layering.
so yes, I deliberately pointed you to a simple patch to highlight how to do polyphony, but things get much more interesting, when you get creative