

Segment the network as much as feasible, forbid the communication between the segments via FW rules, and set an alert when those rules are triggered.
For example: your dmz should never initiate any type of communication with your lan segment, your lan segment should not try to access services outside ports 80/443, your dns should log all resolutions performed and it would be nice to have at least a black list.
None of them should have dns over tls, and for specific hosts and networks segments, new domains with very looong active but idle connections should trigger an alert.
My personal opinion is that for a homelab is not realistic to perform a dpi to check that there is not an active attack ongoing, neither from the raw processing power, either from the human effort side, your best chance is to alert when something unusual is happening and then adjust your rules of the are false positives

My setup is “simple” and all these monitoring functions are performed in my opnsense box with the telegram plugin.
Most of the alerts are pretty basic and are done into the FW level or the outbound basic logging. So opnsense with the basic tooling is just enough.
I have in my todo to connect the logging system from opnsense to a proper Prometheus/grafana system to really have proper log of several days without having an impact on the FW but I never find the time to do it (lazyness problem)