• Google indexing problem

    A follow up to that strange Google problem. On a tip from a reader, I checked to see if the site is being indexed. Seems like Google is choking. No idea why or how to fix. Suggestions welcome.

    zzz  2013-01-18 15.17.32

    Share
    Comments closed
     
    • Looks like your /robots.txt file may be screwed down too tight. Yours has this:

      # This is your robots.txt file. Visit Options->Robots.txt to change this text.
      User-agent: *
      Disallow: /wordpress/cgi-bin
      Disallow: /wordpress/wp-admin
      Disallow: /wordpress/wp-includes
      Disallow: /wordpress/wp-content/plugins
      Disallow: /wordpress/wp-content/cache
      Disallow: /wordpress/wp-content/themes
      Disallow: /wordpress/trackback
      Disallow: /wordpress/feed
      Disallow: /wordpress/comments
      Disallow: /wordpress/category
      Disallow: /wordpress/tag
      Disallow: /wordpress/page
      Disallow: /*?*
      Disallow: /*?
      Sitemap: http://theincidentaleconomist.com/wordpress/sitemap.xml.gz

      It’s currently set to block all robot (unmanned) user agents from visiting certain URLs. (The two Disallow rules at the bottom look like they might inadvertently block everything, though.)

      • Yeah, I did that ages ago. I think this is a new problem. Others have said the robots.txt looks fine. The last two lines prevent indexing of page URLs with a “?”. I forget what those were/are, but they were messing up search returns. So, I don’t think this is the answer.

    • Have you looked at the crawl errors page? What does it say?