When Monitoring Fails Silently

3 AM during on-call. Grafana dashboards show all green. Everything looks perfect.

Then support tickets flood in — websites won't load. My investigation:

• nginx returns 200 OK
• Database operational
• No network packet loss

• The culprit? DNS resolvers failed silently. Our monitoring only checked TCP port 53 availability, not actual resolution.

  My updated checklist now includes:

  1. dig @8.8.8.8 example.com (functional resolution test)
  3. TTL validation
  5. Cross-comparing responses from multiple resolvers

  6. Like the Allbridge hack, threats often strike where you least expect. We now monitor not just ports, but business logic integrity.