Quick Tip: Troubleshooting /etc/hosts issues

Update: The original post contained some false information – according to this post by Justin Carmony OS 10.7’s IPv6 handling breaks /etc/hosts files with more than one entry per line (I’m assuming 10.8 is affected as well). In my testing it was after 10 entries on a line that the mDNS error started appearing in my system log.

I ran into an interesting error this evening: I was working on a new WordPress site for a freelance client when my local server (I use MAMP) would start hanging. Now my MacBook Pro is certainly past its prime (spring 2008, to be exact – I can still remove the battery) but there’s no reason a bare-bones WordPress theme running on my local machine should be slower than most of the external sites I’m visiting so I decided to do some digging.

My first step was to reboot; first MAMP, then the whole system. After a few minutes to boot back up (I really can’t wait for the 2013s to come out) the result was the same: click a link, watch the page hang for 15-30 seconds, then finally load all at once.

As this was one of the ~2% of computer issues that couldn’t be solved by turning it off and on again I decided to dig into my log files. Apache logs looked normal. PHP error logs didn’t have anything interesting. I even checked my mod_rewrite logs, thinking I could have messed up some redirects somewhere and found myself in some not-so-infinite loop. My site and server appeared to be clean.

I’ve found that anytime a process takes a long time then suddenly works all at once (especially after about 30 seconds of lag time) there’s a good chance some background process holding everything up. I opened up my system log and saw it filled with the following error message (client names have been changed to protect the innocent):

A quick Google search for “AnswerQuestionByFollowingCNAME” didn’t provide much help but the error message reeked of DNS shenanigans so I popped into my /etc/hosts file to investigate. The relevant portions of the file looked like this:

Nothing seemed out of the ordinary, so I started questioning everything. Can you have two lines with the same IP address? If so, why has it worked up until this point? Does it matter if you use tabs or spaces? Is there a limit to the number of entries on one line? I started adding comments, removing the last few .dev hosts (as they were the most recent additions to the file). Suddenly everything was zippy again, so I knew I was moving in the right direction.

I found a Debian mailing list post where it was determined (inconclusively) that no, there probably shouldn’t be more than one line per IP, so I moved my line of 17 .dev domains up to the first entry for 127.0.0.1. Suddenly MAMP was zippy and my logs were clean, happy days!

After spending an hour blogging about what I thought was a solution I found myself facing the same problem. I started moving a # character down my now-single line for 127.0.0.1 and found the sweet spot to be ten entries. As soon as an eleventh was added the error started appearing again. “It must be a limitation in /etc/hosts!”, I thought to myself. After a little more research I found Justin Carmony’s Mac OS X Lion, /etc/hosts Bugs, and DNS Resolution where he faces similar issues. One particular piece of information caught my eye:

With the new Lion Update comes better support for IPv6, however this seems to have broken the /etc/hosts file from allowing multiple domains per line. Before you could do something like this, which is works in several other operating systems:

127.0.0.1 local.joind.in local.example.com

1	127.0.0.1 local.joind.in local.example.com

However, this now seems to break and not work. You must keep each line to one ip and one domain. So your configuration should look like this:

127.0.0.1 local.joind.in 127.0.0.1 local.example.com

1 2	127.0.0.1 local.joind.in 127.0.0.1 local.example.com

Justin hit it right on the nose: OSX doesn’t like multiple hosts per line in /etc/hosts. I had never caught it before (I recently switched jobs and have gone from working on a work-issued MacBook Pro running Snow Leopard to my personal laptop running Mountain Lion) as I haven’t had more than ten hostnames to a line until recently.

As I was writing this blog post I noticed another potential cause for those system console errors – one of the clientname.dev hostnames was declared twice. I’m not sure if that was hurting or not but it was at the very least redundant.

What have we learned? Sometimes it takes a trip into the system log to get to the root of a problem. Sometimes you can solve problems by simply following the rules for formatting a system text file. Also that a Google search for “AnswerQuestionByFollowingCNAME” is less than helpful when you’re troubleshooting. Most of all, make sure you have a problem actually solved before you blog about it.

I hope this helps somebody!