There are several constraints missing:
- Allowed output level / emissions at various frequencies (Fsw, harmonics)
- Maximum output impedance at signal frequency; output regulation
- Acceptable component cost, size, space, losses, etc.
To minimize Zout, we must minimize series impedance between inverter and load. This dictates Fc as high as possible (close to, but still below, Fsw).
That is, notice that a filter isn't simply "pass or cut", it's by degree, and for frequencies near Fc (cutoff frequency), signals are not just modified in terms of amplitude, but the source impedance is affected by the filter as well. We want Fc far enough above the signal frequency that its impedance remains low, and therefore regulation (change in Vout for change in Iload) remains acceptable.
Alternately, this dictates low Zo of the filter (so that L will be small and C large), but this may be frowned upon, or prohibited outright, by transient behavior: peak startup current from the inverter, or short-circuit discharge current to the output. (Though I'm not sure anyone would really care about short-circuit current for a mains inverter. It can be important for general power supply applications.) Large C also consumes reactive power at the signal frequency, which generally costs efficiency in a mains inverter, and in the extreme case, wastes inverter capacity (apparent power, VAs).
Or as a final alternative, we employ a more complicated control circuit that adjusts for output voltage (closed loop control, whether tracking the waveform as a proper feedback amplifier, or servoing the amplitude to keep it stable).
To get desired emissions, we must set Fc at least somewhat lower than Fsw, and filter order high enough, to get sufficient attenuation. For example, if allowed output is 1mV at 10MHz, the inverter delivers 200V, and it's at the 10MHz/50kHz = 500th harmonic, which for a square wave is 1/500th the fundamental amplitude, or ballpark 400mV, so we require at least (400mV)/(1mV) = 400x, or 52dB, of attenuation between inverter and output at 10MHz. And so on for other frequencies of interest. Probably, getting Fsw itself down to some 100s or 10s of mV is the primary goal, and harmonics fall trivially after that, but some frequencies/bands may end up peaked due to component parasitics and layout, and this can necessitate revisions, including adding a high-frequency filter.
To minimize component size, we also prefer a high Fc. Since L and C are both inversely proportional to Fc, while component volume is more or less proportional to value.
Filter order is an open question. We might further optimize for minimum energy storage -- which further minimizes component volume -- and choose accordingly. For example, if 40dB attenuation is required at Fsw, and asymptotic attenuation goes as 20dB/dec/order, then we need Fc one decade below Fsw for a 2nd order filter (simple LC), 1/2 decade (about a third) for 4th, etc. As order goes up, Fc changes less (approaches Fsw), while the number of components (of approximately equal sizes) is proportional to order. It turns out, this tradeoff balances at about \$n = \ln A\$, for attenuation ratio A and filter order n. That is, if we need 40dB or 100x attenuation, a filter order around 4.6 will minimize component size.
Finally, filter type is dictated by two factors: 1. The inverter must have an inductive load, otherwise switching loss is increased at least some (resistive load, all harmonics dissipated as heat), or wildly so (capacitive load, switching incurs huge peak switching currents). So we must have an inductor-input type lowpass filter. 2. Capacitors are cheaper and more compact than inductors, so we prefer increasing filter order one more step, to maximize capacitor count. Combined with the L-input constraint, we shall choose filters of even order. So we would choose \$n = 4\$ for the above case, and adjust Fc a little bit lower than ideal to account for the order not being the exact fraction calculated.
One final detail, not evident from the above, but separately: a filter only performs to design specs when terminated into at least one resistance. The inverter is an extremely low resistance, bordering on a short circuit. The load/output may be a completely open circuit (unplugged), or it may be inductive or capacitive loads (motors, SMPSs...). (It could even be a short circuit, which we should design for from a system protection standpoint, but wouldn't be expected to pass emissions under.) How do we resolve these requirements? Two ways: 1. choose a constant-output-resistance filter, or diplexing filter; 2. add a snubber network. ((2) can also be seen as a crude subset of (1).) I won't go into detail about CR or diplexing filters, but the snubber is simple, and suitable for this case. Simply connect an R+C in shunt at the output, of value R = Zo (of the filter), and C > 2.5 C_total (that is, much greater than the total capacitance in the filter). This doesn't work so well into a capacitive load (where the load adds to the total), and if we desired to account for capacitive or low-impedance loads as well, a series R||L network must be added. (Usually, this won't be considered an important use-case, and, again, inductors are relatively expensive and bulky components, so we'd prefer to mitigate it with R+Cs where we can.)
Also note that, if this is a push-pull or H-bridge application, consider each output channel as stand-alone with respect to circuit ground: you need one filter each. Providing DM and CM filtering independently, isn't very practical, and simply filtering each in the normal mode is simplest. I would allow that, a lower order 1st-pass normal mode filter might be used, and then DM and CM can be treated separately -- the optimization being that, DM emissions may be more tolerable than CM emissions, so a pair of NM filters followed by a CMC or two might offer adequate filtering. (But it probably won't be adequate for a grid-tied inverter, where emissions from each port must meet limits, not just their combined CM emissions.)
a priori
information can be used to design a better filter than if you lacked this information. It's extra knowledge and can be used to design a more optimal filter. Just FYI. \$\endgroup\$