• Google indexing problem

      4 comments
      January 18, 2013 at 3:20 pm
      Austin Frakt

    A follow up to that strange Google problem. On a tip from a reader, I checked to see if the site is being indexed. Seems like Google is choking. No idea why or how to fix. Suggestions welcome.

    zzz 2013-01-18 15.17.32

    TwitterFacebookDiggDeliciousStumbleUponShare
     
      Uncategorized
      Google

    • by Alan Krueger on January 18th, 2013 at 15:46

      Looks like your /robots.txt file may be screwed down too tight. Yours has this:

      # This is your robots.txt file. Visit Options->Robots.txt to change this text.
      User-agent: *
      Disallow: /wordpress/cgi-bin
      Disallow: /wordpress/wp-admin
      Disallow: /wordpress/wp-includes
      Disallow: /wordpress/wp-content/plugins
      Disallow: /wordpress/wp-content/cache
      Disallow: /wordpress/wp-content/themes
      Disallow: /wordpress/trackback
      Disallow: /wordpress/feed
      Disallow: /wordpress/comments
      Disallow: /wordpress/category
      Disallow: /wordpress/tag
      Disallow: /wordpress/page
      Disallow: /*?*
      Disallow: /*?
      Sitemap: http://theincidentaleconomist.com/wordpress/sitemap.xml.gz

      It’s currently set to block all robot (unmanned) user agents from visiting certain URLs. (The two Disallow rules at the bottom look like they might inadvertently block everything, though.)

      Reply

      [top]

      • by Austin Frakt on January 18th, 2013 at 17:28

        Yeah, I did that ages ago. I think this is a new problem. Others have said the robots.txt looks fine. The last two lines prevent indexing of page URLs with a “?”. I forget what those were/are, but they were messing up search returns. So, I don’t think this is the answer.

        Reply

        [top]

    • by JJ on January 18th, 2013 at 15:47

      Have you looked at the crawl errors page? What does it say?

      Reply

      [top]

    Write a comment




    (Submission implies acceptance of comments policy.)