Site downtime today - My bad!

[frostfire 1,279]

Hi guys,

Was told a few minutes ago that the site was down... when I went to check, sure enough the DNS was not resolving.

The issue was twofold. First, my primary DNS server hiccuped. Second, the secondary and tertiary servers did not pick up the slack, though I still do not know why yet. I am looking in to it and will make sure it does not happen again. I suspect I will be able to find the answer fast enough as those other 2 servers are still not serving pattayatalk.com addresses for some reason. 

Frosty

[jacko 8,786]

How did someone get a message to you?

I noticed the site was down early this morning, and have been checking back throughout the day. (Actually thinking it was a biggie as the down-time got longer).

I realized I didn't have a way of getting a message to you. Perhaps you could give a few of us Mods your phone number or some way of contacting you  for the future?

[frostfire 1,279]

Facebook. Dave post messaged me at noon, but I was sleeping. I just woke like 15 minutes ago. 

I will send my number to you guys in the private area. 

[frostfire 1,279]

Ok,

Not that it matters, but I found and fixed the issues. 

This was simple laziness on my part. When I set up my enterprise network 10 years ago I did so with multiple redundancies to prevent issues like this. I got it set up, configured, and *poof I thought, done! Uh huh. But Microsoft.

A series of issues caused this downtime, I just did not know they were happening... because the system was working as I had intended it. lol. 

I have 3 name servers. All on different physical servers and in different racks inside the datacenter, and on different subnets, all "best practices" for DNS hosting. What I failed to do is run periodic tests to make sure all 3 servers were up and running. My primary server hosts 99% of DNS requests (about 500 per second), and unless you specifically ask for a specific server for some reason, #1 will answer. Apparently #'s 2 and 3 both had issues that arose from windows updates sometime in the past 2 years or so (since the last time I checked them physically). One was down due to a change to the firewall rules (turned off the rule that was supposed to allow the traffic), and the other was too sensitive in its automatic IP blocking rules. It seems that 10 requests per second not over the top for dns servers), which resulted in that server blocking every single IP that was requesting data. lol. The end of the chain, my primary server, had a hiccup for a yet unknown issue-  I simply hit "start" and it started working again. Ill research that later... but all 3 are now back up and running, and now on my monthly maintenance to-do list. On the plus side of things, its unusual for a windows server to not go thru a reboot at least once a month or so... and DNS servers 2 and 3 managed to stay up and stable for over 2 years! Woohoo? 555. 

Anyway, sorry about that lapse guys. Totally not cool of me. Ill do my best to prevent it in future... and have put my phone number out for the mods in case it does, so I get notified faster. Ill also get our facebook presence up and running ASAP so those that want to can join it and be able to contact me there if need be.

 

Frosty

[TheFiend 1,181]

I was unable to access the site on my desktop PC yesterday, but strangely did manage to access on my mobile when I tried it on that... trying again on my PC later on resulted in no access again....