The adventure started last week when it was decided by the board that our current co-location facility for our servers was inadequate due to their inability to communicate during problem times such as an internet outage from their upstream provider. So Thursday I set all the DNS records TTL to 5 mins and we are off to the races.
Along comes Sunday afternoon (4 PM EST), and I head up to the co-location facility armed with my new IP list and ACL router rules. Everything is going along swimmingly, We get all the servers offline and organize the 200 lbs of network wire, servers and power equipment onto the cart and into the transport SUV and head over to the new facility! Oh joy! It looks like we just might get this done in time for me to get home before my fleet save returns.
So we get to the new facility, and finally get through the security systems. Fortunately, we were able to get in without the rectal scan. Off go the doors to the new cabinet. As we align the rails for installation, *gasp* there is a power bar running the entire height of the cabinet that appears immovable that is directly in the way of any screwdriver that would be employed to ***** in the damn rails.
N.O.C. support call one.
20 minutes later a technician shows up with keys to the above 1/2 of the cabinet and snaps the power rails from their holders to allow us access to mount our rails. No biggie, only 20 minutes behind schedule. 1 hour passes and all of the hardware is finally mounted into the cabinet. I fire up the primary and secondary name servers and update all the records with our spanking new \27 address block, and do the same with the router. Wait a minute, no internet connectivity. What's going on? There is NO feed wire to the router and no ports can be found in the cabinet what-so-ever.
N.O.C. support call two.
20 minutes later a technician shows up with keys to the above 1/2 of the cabinet again, and unravels a feed port that was nicely locked away in the above cabinet.
Now we have internet access! 4 hours later, all the machines have been tested, inbound internet connectivity has been tested, DNS has propagated to some key ISP DNS servers that we test, and DNSReport.com says we have glue and revDNS. Fantastic. Time to go home! (10:30pm EST)
The trip from my home or office to the co-location facility is about 40 minutes. Shortly after 11PM, I decide to continue testing the applications and sites that run from our servers. I found a problem with the ACL rules on the router. 11:30PM, I start to update the ACL and forgot a command, so I have most wonderfully locked myself out of the router and have blocked all inbound internet access to the servers.
Back to the co-location facility for me.
Finally, I get to the facility and get the problem resolved and finish my testing on site so that I would not risk another horrible mistake. Not bad, 2:30am. I send a final email to the techs at the office so they know how to handle any support calls for the next day. Time to go home for some
. Later that morning, I get a call at 9:15am from a tech at the office who DIDN'T READ THE BLASTED EMAIL! I then try to go back to sleep for a bit, but no luck. So I called into the office to check with the team on how the testing has been going. It turns out that our secondary DNS was not syncing with the primary, so we had some people experiencing intermittent connection problems because that server was telling them to go to the wrong IP. Lucky for me all the TTL's were still 5 mins and the problem was resolved promptly.
I take a shower to wake up a bit more and head into the office for noon, just to make sure everything is all kosher. Come 3pm we get an email from a client who explains that one email they sent out was bouncing to various recipients that he knew the emails were correct for. After some testing I learn that the problem is the IP we assigned to the SMTP server was listed as part of a dynamic IP block and was therefore on the dynamic IP blacklist. Further investigation revealed that my entire \27 AND the \24 that it belonged to were on this list.
N.O.C. support call three.
I was not impressed as email is imperative to my clients, so we were issued a new \27 on a different class C. Tests on this subnet show that it is clear, and we should be fine with it. Yet another trip back to the co-location facility 4PM EST.
Repeating all of the changes from the night before, the transitions seems to go smooth so I get out of there around 7:30pm EST. When I got home I laid down just for a minute on top of my bed as I was dead tired. I awoke the next morning at 8:30AM still fully clothed from the night before. Bless my wife for not waking me, I needed the sleep.
This morning I checked the backup tape to see if everything ran correctly overnight. I see the wonderful message "Bad tape - please insert a new tape and click OK".
4PM today, I am on my way back to the blasted co-location facility. I am really getting tired of this.
...and that is why computers are teh suck.
I put that disclaimer in there just for you ya know. 
