Messages - eugenmayer

Maybe some stats, i run an external uptime tracker so i have at least some timings of when the boxes are going down for now - at least fully for the AWS box ( see screenshot )

It seems like the "every 2 weeks is not perfectly right, seems like it was ok for e.g. nearly 1 month now, then crashed.

The pattern for the KVM boxes is 100% predictable though, every week, on saturday.

---

Also something interesting, out of those 5 KVM boxes, only 2 run HAproxy - those 2 which are crashing. Also i migrated away from HAproxy on the other 3 and it seems like this might be the reason they stopped crashing.

The AWS box has HAproxy too - also crashing.

---

Could that be HAproxy related or maybe something with the ACME plugin which runs a companion there? Not sure, do not want to misguide, but it seems like an interesting pattern here.

- when do the ACME task run usually? ( the one in cron are rather daily ) - are there any HAproxy related tasks?

Problem2 of those 5 AWS keep stalling on Saturday every single week ( for 5 and more weeks no). Right now its always the same boxes, it used to be randomly for those 5.

The AWS box seems to stall every week, also Saturday.

What i mean by "stall":it seems some traffic is still passing through the OPNsense box it looks like NAT is still working as also stateful connections. It seems like the boxes behind OPNsense though cannot access WAN anylonger (outbound issue?)

Also i cannot connect using SSH or terminal, in both cases i can enter the user, but then instead of asking for the password - it just "hangs" there.

What i deductedFor several weeks now, after i detected that the auto-upgrade did not work and they are stuck at 18.7.4, i upgraded them to 18.7.7 ( then .8 ). Now always the same get stuck. I suspected that it is the upgrade so i deactivated the upgrade cron tasks - but this week no update was available, still those 2 stalled and the AWS box.

I also suspected the KVM boxes to "stall" on proxmox backups, i disabled them but that did not help either. Also since the AWS box is not backup using that at all, i expect that was not the right assumption anyway.

Also, 18.1 and 18.7 boxes are affected by this - host on totally different hypervisors (AWS/kvm proxmox).

While the KVM boxes have about a every similar duty, the AWS box is rather different, still affected.

HelpCould anyway help me getting to the bottom of this - this becomes a real blocker for me in a sense that i might also consider to migrate away if i cannot solve this at all at some point.

If i can get any logs or can let the boxes log additional things while stale out, let me know. Maybe some rrd graph could be interesting or whatever, let me know. Thanks!

I am using a public TLD for which i use the private-domain flag in unbound and also a domain override.

So lets assume it company.com - i use the namespace <namspace>.company.com as a internal domain, so internal.company.com. (Domain override in unbound).

The problem now is, that i am using a tool form ACME DNS-01 challenges which will do a dns lookup on the default DNS server ( OPNsense in this question ) searching for a NS record ( primary nameserver for company.com ) like

This setup should be based on a proxmox, being behind a opnsense VM hosted on the Proxmox itself which will protect proxmox, offer a firewall, a privat LAN and DHCP/DNS to the VMs and offer a IPsec connection into the LAN to access all VMs/Proxmox which are not NATed. The server is the typical Hetzner Server, so only on NIC but multiple IPs or/subnets on this NIC.

Proxmox Server with 1 NIC(eth0)3 Public 1IPs, IP2/3 are routed by MAC in the datacenter (to eth0)eth0 is PCI-Passthroughed to the OPNsense KVMA private network on vmbr30, 10.1.7.0/24An openvpn mobile client connect (172.16.0.0/24) to LAN

When i got this straighten out i would love to place a comprehensive guide on how to run OPNsense as a Appliance with a private network in on Proxmox, passing some services to the outer world using HAproxe+LE and also accessing the private lan using IPsec

little update on this, after fiddling around with shimo vpn i was not able to get split DNS to work even though they explicitly offer it - i asked the support because i think thats a software bug. Also shimo VPN does not properly detect the network list, thus always configures to send the whole traffic through VPN, no matter how you setup the mobile client connection - this can be fixed by manual route overrides

i tried vpn tracker 9 or 365 then and that worked out completely, DNS and gateway work right away. You do not choose a device here, but rather a customer ipsec connection.

If there is any interest, i can paste the general configuration for both clients - in the end, they are very straight forward and aligned at exact the same terms used in opnsense