Googles’ rolled out with a new tool – Mobile-Friendly Test. At first I was a little skeptical about it like I am with any new thing – I question things before I start trusting it. It’s not that I’m resistant to change, that would be a bad thing – change is good. Change is growth.
In my consulting work, I noticed there has been a high demand for using this tool – mainly marketing agencies looking to further audit and optimize their client websites. A visual audit just isn’t good enough, meaning the obvious isn’t so obvious anymore just by visually testing a site on a mobile device. There’s a difference between how we see it and how Google sees it and let’s face it – Google owns everything. Sometimes I feel like Google is the AI (that’s going to take over the world) everyones been talking about.
So I had this website roll in, it was failing the test. This wasn’t my first by the way – I’ve fixed a few websites to pass this test by tweaking some CSS and adding a few media queries. In this particular case though, it had nothing to do with the source code. So I had to really step back. Sometimes as a developer we get so consumed with source code we fail to see the big picture.
I was stumped for a while. I was wondering why Google was rendering an unstyled version of the website – meaning the CSS styling was stripped off. Could it be that the CSS file was being blocked by Google Bot?
Yes, it was being blocked. The source of the issue was contained within the robots.txt file – if you don’t know what this file does, there’s actually a Robots.txt website explaining it in further detail.
While inspecting the robots.txt file for the site, I saw that the themes file was excluded from being crawled.
The Disallow protocol tells Google what to exclude from crawling
User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-content/cache/ # Disallow: /wp-content/themes/
To fix the issue, I simply needed to comment or remove that line. In this case, I just commented it out – I am not sure what the client intends to do with their robots.txt file. Before I go removing anything I always like to make sure every party is aware of my reasons for removing or commenting some code out.
On the client side of things or the internet marketing side of things, there are reasons why they would want to exclude Google bot from crawling certain directories. I leave these to the internet marketing experts but for starters you usually don’t want Google to be crawling unnecessary wordpress directories in files – in this case “wp-admin”, “wp-includes”, “plugins”, “cache”. These are source files, libraries, and configuration files. What we are interested in having Google crawl are relevant pages, pages with relevant content.
The themes folder (the line causing the issue) is disallowed entirely in the example above. This means that everything in it is disallowed. There are some configuration files contained within the themes folder so I can see a case for this. To get around failing Google’s Mobile-Friendly Test, I would recommend disallowing on a per file and per directory basis. Yeah this is more work, but this is the only way to pass the test.
Which begs the question – if it’s all about passing the test then this is where things start getting really ambiguous. A site shouldn’t fail the test because the theme directory is disallowed – is what I’m saying. Anyway, I’m glad this fix required a one-liner so I’m not going to get anal on this issue, especially with Google.