The schematic looks nice, but it sure is over-engineered!
Also there is no short circuit protection.
Last time I made a similar circuit was a while ago, that's how it went:
I used a hysteretic buck current source. Since I used low voltage that was a current sense amp, but you could use a faster Hall effect sensor. The output of that feeds a hysteresis comparator that controls the FET driver. So when output current is too high, it turns off the top FET. When output current is too low, it turns on the top FET. This regulates current to an average that is proportional to the comparator threshold, and switching frequency depends on hysteresis and inductor value.
Being a current source, it is inherently short circuit protected and will run into a short indefinitely.
I then used the ENABLE pin on the FET driver to control it. That was implemented in software, but it could also be done with a bunch of comparators. When output voltage reaches the desired value, it turns off. When the battery needs charging, it turns on. Same for input voltage to keep it at the maximum power point: it turns off when input voltage drops below the maximum power point, and turns back on when it rises. You could also control the hysteresis comparator threshold to modulate the current instead of using on/off control.
It's a pretty simple circuit because there is no compensation and the control loop is dead simple.
Then I had a problem which you will perhaps also encounter: the FET driver I used (ADP3120) would not start if the bootstrap capacitor was not charged. In your case, the output of the DC-DC is connected to the battery, so it is above the +15V from your aux supply. Thus, the bootstrap cap will not charge through the bootstrap diode. To charge it, the driver must turn the bottom FET ON. But my driver could not turn the bottom FET on if the bootstrap cap was not charged! This was solved by pulsing the MOSFET that, in my circuit, did the same thing as your output protection relay, which disconnected the battery, allowed the output voltage to drop to zero, and charged the bootstrap cap. Then, the driver would start, and with a 1µF bootstrap cap, it would stay awake even during the off-periods when it was not running for a millisecond or two because the target output voltage had been reached.
Some FET drivers will turn on the bottom FET when the bootstrap cap is not charged, but some will not, they'll just stay off and do nothing. You need to check datasheets for that, or you'll get a surprise.
I put a MOSFET at the output also to prevent current from flowing back from the battery into the source. You'll have to switch the relay to do the same, or the leakage current of the solar panels will be a drain on the battery, plus the battery will power your circuit through the 15V supply even at night.
Note the control algo is much simpler to implement with a microcontroller. The hysteresis comparator did the fast stuff, and the rest is pretty slow, so it can be done with software and an ADC. I used the micro's analog comparator for overvoltage protection, so if the battery is removed while the micro is in debug mode and software is not running, it still stops the PWM driving the ENABLE pin of the FET driver, which means the 6.3V cap on the output does not get shot into orbit with 40 volts.
You also need plenty of surface mount MLCC caps on the input and output, because the large caps you used have substantial inductance. Also check they can take the AC ripple current you'll use. For the input, AC ripple current is equal to the full output current. For the output, it's your inductor ripple current. The FET driver also requires solid decoupling with MLCCs.
You could use a FET with a lower gate charge, too.
More stuff while looking at the schematic:
I had not realized the power MOSFETs were not on the board. That will increase the gate drive inductance, which will require larger gate resistors to prevent oscillations, slowing down switching and resulting in lower efficiency. It will also increase inductance everywhere and generate more EMI. Also you can't use SMD MLCCs to decouple the power rails, which means trouble. Combined with wiring inductance, I'd expect large voltage spikes which will blow the FETs.
As drawn on the schematic, with no decoupling, I would be very surprised if it works.
A better solution would be to use a 4 layer board, which costs 20€ these days. You can put the FETs on the board, and use power and ground planes for the input and ouput.
Best stackup would be a pour for VIN, ground plane on next layer. This gives minimum inductance to the input MLCCs since the space between the layer1/layer2 or L3/L4 is only 0.2mm. You can put the pours on top or bottom, but the ground plane should be on the next layer. Then you can put the SW node on the other side of the board so there is 1.4mm between it and the ground plane, for lower capacitance.
You can also solder fat copper wire on the board, so you don't have to pay extra for thick copper. That way you get low inductance from the planes, and high current handling from the thick wire.
Also there is no decoupling on the FET driver (10µF electrolytics with several ohms ESR do not count as decoupling). So when it tries to switch the FETs, its VCC will spike down, and it may bias the FETs somewhere in linear mode, which will make them dissipate a lot of power.
Since the huge input and output caps have high inductance and low ESR, they will resonate with the MLCCs at a frequency close to your switching frequency. Since one 5600µF input cap will never be able to take 20 amps ripple, it would be simpler to use many lower value caps, which lowers the inductance, the ESR, and increases ripple current handling. You will still probably need another cap in between, for example a low ESR electrolytic with lower value and low inductance, or a 10µF MLCC with a small resistor in series. The impedance of the capacitor combination should be simulated accounting for PCB inductance to make sure it will not cause a resonance peak.
FET choice
Duty cycle will be around 50%. You don't mention the current, so I'll take the 20 Amps value in the schematic. At 10 mOhms RdsON, that's 4W conduction losses total, so 2W per FET. This means it is not necessary to use a high inductance TO-247 package. A low inductance SMD FET will allow faster switching, so it will have much lower switching losses.
I'd replace your Qg=135-200nC, 5.3 mOhm, TO-247 FET, which costs $8, with two of TPH1500CNH in parallel, 22nC, 13 mOhm, SMD, at $1.88 each. It's 2x cheaper, has 3-4x lower Qg, about same RdsON, and about 10x faster. So that's about 1W conduction loss per FET, plus about 4W switching losses at 200kHz if they switch in 20ns.
IR2104, which has a wimpy output current of 210mA, should be replaced by something much beefier and much faster. Even more so if you keep your FETs with huge gate charge.
IRF200 needs 50nC of gate charge to go through the switching zone, with 210mA current that's a whooping 250ns, which means about 25W switching loss per FET. So you need a big heat sink.
The top SMD FET is easily cooled since its drain is in a power plane. The bottom FET has its drain on the SW node which should have low capacitance, so it can't be a big heat sink. An elegant solution would be to just put some "thermal jumpers" between the SW node and the ground plane. These "thermal jumpers" are small ceramic chips with two solderable pins that conduct heat very well. For example, 4°C/W for 0612 size. So you need two or three. They are sold at inflated price under the name "thermal jumpers" but they really are SMD resistors without the resistor film on it, so you can put in high value (megohms) resistors instead, which costs nothing.
Once the heat has been transferred to a plane that doesn't have 100V 100kHz square wave on it, then you can slap a heat sink on your board, and it won't act as a wideband radio jammer.
Or you could use a TO-220 FET as a compromise. It has higher inductance than the SMD, but it's easier to cool. Since switching loss is the killer here, it's all about Qg and package inductance.
