Why Pages Disallowed in robots.txt Still Appear in Google

Meaning of Robots.txt

Robots.txt is a useful file which places in your website’s root and controls how search engines index your pages. One of the most useful declarations is “Disallow” — it stops search engines accessing private or irrelevant sections or pages of your website, e.g.

Disallow: /temp/
Disallow: /mypage.html/

You can Even Block Search Engines Indexing Every Page on Your Domain, e.g.:

User-agent: *
Disallow: /

Blocked Pages can Still Appear in Google – HOW?

Take a little while to understand how and why it happens. Assume you have a page at http://www.abc.com/mypage.html containing confidential information about your company’s new “coupon codes” project. You may want to share that page with partners, but don’t want the information to be public knowledge just yet. Therefore, you block the page using a declaration in http://www.abc.com/robots.txt:

User-agent: *
Disallow: /mypage.html

A few weeks later, you’re searching for “coupon codes” in Google and you found http://www.abc.com/mypage.html at 1st Page of Google. How could this happen? It means, Google abides with your robots.txt instructions, isn’t?

However, this is not a violation of robots.txt rules. This happens because of very simple reason that Google found your link from elsewhere, means http://www.abc.com/mypage.html might be linked from any external website, so Google caught you from there. Meta information also comes from that particular external link, not from your page content.

There are Several Solutions that will Stop Your Pages Appearing in Google Search Results:

  • Set a “no index” Meta Tag: Google will never show your page or follow its links if you add this code to your HTML head section:

  • Use the URL removal tool: Google offer a URL removal tool within their Webmaster Tools.

  • Add authentication: Apache, IIS, and most other web servers offer basic authentication facilities. The visitor must enter a user ID and password before the page can be viewed. This may not stop Google showing the page URL in results, but it will stop unauthorized visitors reading the content.

67 comments:

Anonymous said...

Hey! Appears like you've a excellent group going on here. I am unable to submit wherever else due to the fact my password hasn't arrived yet, but I desired to introduce myself here in this.
Identify is Gerald; At present finishing up my Physics Important at OSU, and can most likely remain at OSU for my graduate studies. I'm a recent convert on the YEC model and I'm quite fired up to discover more about it -- that is why I'm here.
I in all probability won't publish for per week or so, as I'll be investing my time reading all past messages making an attempt to have a feel for what has previously been mentioned.
I look ahead to debate and discussion.

http://trentvdowns.zoomshare.com/2.shtml
http://freejoomlatemplates75.blog.friendster.com/2010/07/how-to-pick-your-next-reseller-hosting-company-in-3-easy-steps/
http://www.gather.com/viewArticle.action?articleId=281474978258789
http://www.gather.com/viewArticle.action?articleId=281474978023391
http://clarkwscott.tripod.com/resellerhosting82/index.blog/1977752/some-noteworthy-ideas-on-reseller-web-hosting/

Anonymous said...

Hey! Seems like you have a excellent group going on here. I am unable to put up anyplace else because my password hasn't arrived however, but I desired to introduce myself here in this.
Title is Despina; At the moment finishing up my Physics Key at OSU, and will almost certainly continue to be at OSU for my graduate studies. I'm a latest convert to the YEC model and I'm fairly energized to learn far more about it -- which is why I'm right here.
I in all probability will not post for a week or so, as I'll be investing my time studying all previous messages attempting to acquire a experience for what has by now been discussed.
I search ahead to debate and discussion.

http://www.gather.com/viewArticle.action?articleId=281474978181948
http://jessiegarnold.insanejournal.com/510.html
http://jarvislmercer.xanga.com/729886766/joomla-hosting---guide-for-joomla-websites/
http://scottwfisher.livejournal.com/750.html
http://bennietwooten.blog.com/2010/05/25/wordpress-webpage-hosting-an-easy-wordpress-installation-guide/

Wicked Online Marketing said...

i have came across this situation where i put the links i dont want onto robot.txt but it still appears in google. your post helped me understand better. thanks.

http://wicked-internet-marketing.blogspot.com

Anonymous said...

I used to be recommended this website by means of my cousin.

I am not certain whether or not this post is written by
means of him as no one else understand such unique approximately my trouble.
You're incredible! Thanks!

my blog:
My website ::

Anonymous said...

I’m not that much of a internet reader to be honest but your
sites really nice, keep it up! I'll go ahead and bookmark your site to come back in the future. Cheers
Also visit my web page kosten private krankenversicherung im alter

Anonymous said...

hi!,I love your writing very much! proportion we communicate more approximately your article on
AOL? I require a specialist on this area to resolve my problem.

Maybe that is you! Taking a look forward to peer you.
My blog : search engine optimisation process

Anonymous said...

I almost never leave a response, but i did some searching and wound up here "Why Pages Disallowed in robots.txt Still Appear in Google".
And I do have a few questions for you if you don't mind. Is it just me or does it seem like a few of the comments look like left by brain dead people? :-P And, if you are writing on other social sites, I'd like to
keep up with anything fresh you have to post. Could you make
a list of every one of your public pages like your Facebook page, twitter feed, or linkedin profile?


Also visit my web page; clickbank giveaways
Have a look at my weblog :

Anonymous said...

Your style is very unique compared to other people I have read stuff from.
Thanks for posting when you have the opportunity, Guess I'll just book mark this page.
Also visit my webpage - kredite ohne schufa auskunft sofort

Anonymous said...

Way cool! Some extremely valid points! I appreciate you
penning this article and the rest of the website is extremely good.
Feel free to visit my site ; Vacations For kids

Anonymous said...

Please let me know if you're looking for a article writer for your site. You have some really good posts and I think I would be a good asset. If you ever want to take some of the load off, I'd absolutely love to write some articles for your blog in exchange
for a link back to mine. Please shoot me an e-mail if interested.
Many thanks!
Also visit my blog ... website affiliates

Anonymous said...

Hey there! I've been reading your weblog for a long time now and finally got the courage to go ahead and give you a shout out from Dallas Tx! Just wanted to tell you keep up the great job!
Feel free to surf my web blog : how do you qualify for a home equity loan

Anonymous said...

Hey! This is my 1st comment here so I just wanted to
give a quick shout out and tell you I truly enjoy reading through
your posts. Can you suggest any other blogs/websites/forums that
cover the same topics? Many thanks!

Also visit my blog post; luxus kleidung online shop
Look into my blog : just click the up coming site

Anonymous said...

Wondeгful blog you haνe here but I was curiouѕ
about if уou κnеw of any ԁisсussion boards thаt cοveг the sаme topiсs talκеd about in this artісle?
I'd really like to be a part of online community where I can get advice from other knowledgeable people that share the same interest. If you have any recommendations, please let me know. Kudos!

Feel free to visit my weblog :: hcg diet side effects
My web page > hcg snohomish

Anonymous said...

Can you tell us more about thіs? І'd like to find out more details.

Review my site :: http://www.moodlesocial.com/blog/index.php?postid=36968

Anonymous said...

If some one desires to be updated with hottest technologies therefore he must be go to see this
website and be up to date all the time.

Here is my website new businesses

Anonymous said...

Today, I went to the beach front with my kids. I found a sea shell and gave it
to my 4 year old daughter and said "You can hear the ocean if you put this to your ear." She
put the shell to her ear and screamed. There was a hermit crab inside and it pinched her ear.
She never wants to go back! LoL I know this is entirely off
topic but I had to tell someone!

my webpage; web hosting bangkok

Anonymous said...

These are genuіnelу іmрreѕsive
ideas іn about blogging. Yοu have toucheԁ sоme nice faсtors here.
Anу ωаy keep up wrintіng.



my wеbpagе hcg diets

Anonymous said...

Thеse are genuinelу impгeѕsive
idеas in about bloggіng. Υou havе tοuched
sοmе nice fаctors here.
Anу wаy keep up wrіnting.

Ηеre is my webpаge; hcg diets
my website :: weight loss supplements

Anonymous said...

I am in fact grateful to the holder of this website who has shared this fantastic post at
at this time.

Look at my homepage; von private krankenversicherung in gesetzliche

Anonymous said...

Heya i am for the first time here. I came across this board and
I find It really useful & it helped me out a lot. I'm hoping to give one thing back and help others such as you aided me.

Also visit my page :: trinity debt management

Anonymous said...

You've made some really good points there. I looked on the net for additional information about the issue and found most people will go along with your views on this site.

Here is my blog post ... Affiliate income

Anonymous said...

Do you mind if I quote a few of your articles as long as I provide credit and
sources back to your blog? My blog is in the exact same niche as yours and my users would definitely benefit from a lot of the information you
provide here. Please let me know if this okay with you.

Thanks!

Look into my web-site - 4n25 transistor opto isolator

Anonymous said...

Hmm it appears like your site ate my first comment (it was extremely long) so I guess I'll just sum it up what I wrote and say, I'm thoroughly enjoying your blog.
I as well am an aspiring blog writer but I'm still new to the whole thing. Do you have any helpful hints for inexperienced blog writers? I'd certainly appreciate it.



Feel free to visit my website ... kingston flash memory format tool

Anonymous said...

When I originally commented I clicked the "Notify me when new comments are added" checkbox and
now each time a comment is added I get several emails with the same comment.
Is there any way you can remove people from that service? Thanks!


Stop by my site ballet shoes pictures to print

Anonymous said...

Greetings from Idaho! I'm bored to tears at work so I decided to check out your website on my iphone during lunch break. I love the info you present here and can't wait to take a look when I get home.
I'm amazed at how quick your blog loaded on my cell phone .. I'm not even using
WIFI, just 3G .. Anyways, awesome blog!

my web site go back

Anonymous said...

WOW just what I was searching for. Came here by searching for customer data
integration

Stop by my website ... designer outlet mode

Anonymous said...

The other day, while I was at work, my sister stole my iPad and tested to see if it can survive a 40 foot drop, just so she can be a youtube sensation.
My iPad is now destroyed and she has 83 views. I know this is completely off topic
but I had to share it with someone!

Feel free to surf to my website :: refrigerator freezer not freezing

Anonymous said...

I really wanted to develop a brief comment
to be able to say thanks to you for the precious suggestions you are placing at this site.
My time consuming internet research has at the end of
the day been paid with incredibly good know-how to exchange with my two friends.
I 'd tell you that many of us readers are rather endowed to be in a useful network with many perfect individuals with great solutions. I feel truly fortunate to have encountered your entire web page and look forward to so many more enjoyable times reading here. Thanks a lot again for all the details.

my page ... how to naturally lighten skin

Anonymous said...

I'm gone to say to my little brother, that he should also visit this web site on regular basis to get updated from newest news.

Look at my web site; purchase hcg injections

Anonymous said...

My spouse and I stumbled over here by a different page and thought I may
as well check things out. I like what I see so now
i'm following you. Look forward to looking at your web page repeatedly.

Also visit my homepage: Natural Colon Cleanse

Anonymous said...

It could be hard to find good info about jewelery. There's a plethora of info at present available which may confuse you. The best suggestions you in this article.

Here is my web blog: cheap tag watches

Anonymous said...

Τhe іncrеdible ωoгth that chіck fil a оnline cοuρons іs a lοt more
еѕsentiаl since fгom the еconоmу today.
To gеt probably the moѕt οut of
сouponing, іt's very best that you simply patiently find out all there is to understand about this outstanding method to shop. Read on and discover about using chick fil a online coupons.

My webpage ... chick fil a coupon printable

Anonymous said...

Μemοry loss does not need to be аs bad as eνеrуοne maκeѕ it
out to be. Υou will find lοts of methods уou'll be able to try to slow down this procedure. With patience, analysis, and hard perform, you'll uncoѵer helpful guidаnce
for memοry loss.

Vіѕit my blοg post :: buy procera

Anonymous said...

Sevеre couрoners cаn save
ѕufficient funds with theіг coupon hοbby.

Ηowevег, without having the right іnfοrmаtion anԁ understanding of thе gгeateѕt techniques to uѕе torrid coupon, ρlеntу of peoρle learn it
tο beсome rаther tiresomе.



Ηerе is my web blog :: extremecouponingonline.net

Anonymous said...

The process of car shopping can be lengthy.
You have so many options, not to mention things to consider as you compare each one.
You will be able to make the right decision if you know
enough about car shopping. Here are some great tips on buying a car.


Feel free to surf to my web-site :: cars

Anonymous said...

Pretty еlеmеnt of content. I just stumbled upon your website and in acсession cаpitаl tο say
that I get aсtuаlly еnjoyeԁ аccount youг weblog postѕ.
Αny ωay I will bе subscribing
in your feeds oг eνen I suсcеѕs
уou get entry to constantly rapidly.

Review my web site: unlock samsung galaxy 3

Anonymous said...

It may be hard to find great information about jewelery. There's a plethora of information presently available which can confuse you. The very best suggestions you within this post.

Feel free to visit my blog post ... tag heuer link watch

Anonymous said...

Are you seeking to save your funds on groceries аnd hοusehοlԁ items?
The ρrіme method to finԁ the best mеthod foг acquiгіng whаt you need аt a low price would be to use
friendly s coupons. Where are you ablе to look to find these eхсеllent frіenԁly
ѕ couponѕ?Τhis short article is
gоіng to help reveal thе secrets of friendlys printable coupon.


my web site - friendly s restaurant coupons

Anonymous said...

Are you loоκing to ѕaѵe уour
funds on groсerіeѕ and househοld thingѕ?
The pгime methoԁ to lоcate the bеst technique for аcquiring whаt you would like at
a lоw cost would be to use printable supeгсuts coupon.

Whеre can you loοk to finԁ these gгeat printable couponѕ for
supercuts?This аrticle iѕ going tο аѕsist reveal the secrеtѕ of supercuts
сouροn $5.

Feel free to visit my wеbsite: Extremecouponingonline.net

Anonymous said...

Are you currently attempting tо ѕtrеtch
your buck? You no doubt ωant to get аs much аѕ you can foг
each and eѵery hard-eaгned dοllаr.
Coupons can enable уou quite a little of money on thе necessаrу and nοnessential things yοu
wіll need. The following write-up beneath has bеneficial ѕuggestions anԁ mеthods associatеd with bd
mоngolіan grill coupons.

Here is my web blog Genghis Grill Coupons

Anonymous said...

Hi there! I just wish to give a huge thumbs up for the great information
you might have here on this post. I might be coming back to your blog
for extra soon.

Also visit my page - seoul garden ann arbor michigan

Anonymous said...

Good day! I simply want to give an enormous thumbs up for
the good info you will have right here on this post.
I can be coming back to your blog for more soon.

My blog post - authentic semolina pasta dough recipe

Anonymous said...

Hey! I simply wish to give an enormous thumbs up for the nice data you have right here on
this post. I shall be coming again to your weblog for more soon.


Feel free to visit my blog hershey's semi sweet chocolate chips cocoa percentage

Anonymous said...

Good day! I just want to give an enormous thumbs up for the good information you have got right here
on this post. I might be coming back to your weblog for
more soon.

Feel free to surf to my homepage :: seohyun snsd pictures

Anonymous said...

I guess my question to Martin would be about Taubes and
his operate. Do you disagree with his analysis, see yourself lining up with his research, or falling somewhere in in between?



Stop by my page - clean Eating magazine subscriptions

Anonymous said...

Hello there! This is my first comment here so I just wanted to
give a quick shout out and tell you I genuinely enjoy reading through your articles.

Can you suggest any other blogs/websites/forums that go over the
same subjects? Thanks for your time!

Stop by my homepage ... home improvement season 3 episode 23

Anonymous said...

Hey there! Would you mind if I share your blog with my facebook
group? There's a lot of folks that I think would really enjoy your content. Please let me know. Thank you

My site xfire.com

Anonymous said...

Whats up! I just want to give an enormous thumbs up for the nice information you’ve got here on this post.
I might be coming back to your weblog for extra soon.



Take a look at my homepage: 排名優化

Anonymous said...

Hi there! I just would like to give a huge thumbs up
for the nice data you might have right here on this post.
I will probably be coming back to your weblog for more soon.


Also visit my page; socratic seminar

Anonymous said...

Hello! I simply would like to give an enormous thumbs up
for the nice info you have got right here on this post.
I shall be coming again to your blog for extra soon.

Have a look at my blog ... google seo toolbar download

Anonymous said...

Hi there! I just want to give a huge thumbs up for the great data you may have here on this
post. I can be coming again to your weblog for extra soon.


Here is my page :: semco energy ventures inc

Anonymous said...

Hold hearing the news I get started to talk about free of charge
on the internet grant applications so we participate in around the high-quality website to get a single.


My page: top business ideas 2013

Anonymous said...

Whats up! I just wish to give an enormous thumbs up for the nice
data you may have here on this post. I might be coming back
to your weblog for more soon.

Look at my webpage :: amd sempron le 140 unlock core

Anonymous said...

Howdy! I simply wish to give an enormous thumbs up for the good data you have here
on this post. I will be coming back to your blog for more soon.


my web page; semantics and pragmatics of the lambda calculus

Anonymous said...

Hey! I simply want to give a huge thumbs up for the great information you have right here on this post.
I will be coming back to your blog for extra soon.

Look into my web blog :: korea hostel seoul

Anonymous said...

Hiya! I simply wish to give a huge thumbs up for the great information
you will have right here on this post. I might be coming again to your blog for extra soon.


My web blog :: seo tools free software

Anonymous said...

Hey! I just want to give a huge thumbs up for the great info you have got right here on this post.
I will likely be coming again to your blog for extra soon.


Visit my blog SEO

Anonymous said...

Hey! I just wish to give an enormous thumbs up for
the nice data you will have here on this post. I will likely be coming back to your blog for extra
soon.

Visit my webpage - 排名優化

Anonymous said...

Hiya! I simply want to give a huge thumbs up for the good data you’ve got right here on
this post. I can be coming back to your weblog for extra soon.


Look at my weblog; semantics in generative grammar heim

Anonymous said...

Good day! I just wish to give a huge thumbs up for the nice info you’ve gotten here on this post.
I will probably be coming again to your blog for more soon.


My web site semak blender canada

Anonymous said...

Good day! I just want to give a huge thumbs up for the good info you will have right here
on this post. I can be coming back to your blog for more soon.



Here is my web blog seoul subway song download

Anonymous said...

Good day! I simply wish to give a huge thumbs
up for the great information you have got here
on this post. I will probably be coming again to your weblog for
extra soon.

Here is my web blog :: seoul subway map english 2010

Anonymous said...

Here's my issue with this idea, it form of assumes each and every kid wants to discover. I hated major school, and only ever did perform if the instructor was in the room watching me and other college students get it done. I like training? now (just finishing college(UK) but all by way of major and secondary education I might only at any time do the bare minimal, and I never feel leaving a kid to just get on with their finding out on a computer would actually operate within the very same way as becoming in a classroom with a teacher.

Here is my blog post ... premature ejaculation causes and treatment

Anonymous said...

Hi there! I simply want to give an enormous thumbs
up for the nice data you’ve gotten here on this post.
I will probably be coming back to your blog for extra soon.


Here is my webpage - semak saman pdrm summons semakan saman semak saman trafik online

Anonymous said...

Hiya! I just want to give a huge thumbs up for the nice information you might have here on this post.
I might be coming back to your blog for more soon.



Also visit my homepage ... nestle semi sweet chocolate chips dairy free

Anonymous said...

Whats up! I just wish to give a huge thumbs up for the
nice info you have got right here on this post. I will be coming again to your weblog for
more soon.

Visit my weblog - seoul garden new york menu

Anonymous said...

Hey! I just want to give a huge thumbs up for the good
information you may have right here on this post.
I might be coming back to your blog for more soon.



My web blog seoul times newspaper eating babies