We have a hosted website on SquareSpace, which uses a custom CMS product.
They do not allow editing of the default Robots.txt file, but use a 301 redirect to the one we customize.

Unfortunately, it appears that ES is not picking up/using/honoring the robots.txt file. It appears that other engines do - for example we have a Google Custom search and the disallowed locations do not appear in Google’s results.

The url is:  http://www.henssler.com/robots.txt

Which is redirected to the modified file. However, none of the disallows in the list are being respected.
I did force a full crawl after making the change, but the results were the same.
FYI - the only difference in the default and the customized robots.txt is the last entry /tag/.

Any ideas on what we can check/why this CMS may be different?

Redirecting the robots.txt? Sounds like a bad bad practice. But if people have started to do so I guess we will have to support it.

Have goolge’d a bit around, and robots.txt redirection isn’t even supported by LWP::RobotUA, the standard Perl library.

I will look into this, and see what we can do here Topry.

Many thanks your feedback. Problems like this is so rear that it’s hard to plan for. I guess one have to be out crawling and failing in the real world.

Thanks Runar - many hosted services like SquareSpace utilize CMS to simplify the web creation/management process for end users. As such, there are a number of functions that are not allowed (like meta tags which is irritating).

Let me know if I can do anything to help. I am going to remove the redirect and test again to see if there is something on the host that is causing an issue.

Quick update:
I removed the 301 redirect for robots.txt. The robots.txt is now in the default location: http://www.henssler.com/robots.txt

I forced a full crawl, but it is still indexing/returning results for pages within the Disallowed folders.
This site is public if you want to test it, or if you have suggestions on what to check.

The host is http://www.squarespace.com

I did some more testing, and it appears that the Perl method/library you mentioned doesn’t work with SquareSpace.  I have also tested with IIS and Wordpress without any problems, but I cannot tell what SquareSpace may be doing differently.

I created a test/demo site on SquareSpace (something with fewer pages to test against), it uses their default Robots.txt file, which you can view here: http://lynnandtim.squarespace.com/robots.txt

We then uploaded some test pages and indexed it using your engine. Search for ‘valencia’ and it returns 102 hits, including results within the disallow folders.

The demo/sample site will be active for 14 days and you can create your own if you wish.

Regards,

Tim

I did some more testing, and it appears that the Perl method/library you mentioned doesn’t work with SquareSpace.  I have also tested with IIS and Wordpress without any problems, but I cannot tell what SquareSpace may be doing differently.

That sounds very strange. The Perl library Lwp is probably the most used crawling library in the world. When Searchdaimon did internet search back in the days we used it to crawl many billion of web pages, without any major problems.

If this is SquareSpace’s fault thay most bee doing something very strange…

I am currently looking into it. Will post her when I find out more.

A gym is an ideal place to take part in some activities. Most gyms not just have digital camera models, weight, as well as some training; some of these also experienced gyms with regard to activities you to definitely partake within. These are great way for you not just get healthy, but also provide more enjoyable getting physical. sportsactivity.net

One from the major elements militating from the development associated with sports within Nigeria these days is insufficient effective administration. A great deal of solutions are now being proffered through concerned as well as patriotic Nigerians every day to bail all of us out the actual quagmire. Among such options is this particular text called “Modern Developments in Sports activities Administration as well as Management”. drsports.net

The common-law-rules because defined through the IRS might help business proprietors determine regardless of whether their workers are thought employees or even contractors. It is vital for internet marketers to get this to determination simply because each category is susceptible to different taxes withholdings as well as laws. lawrules.net

No matter your regulation firm’s concentrate - prison, civil, business, family, company, etc. - the best challenge the majority of small in order to mid-size companies face is having less a proper and disciplined method of business improvement. Relying upon referrals doesn’t a strategy make! And waiting to obtain serious regarding marketing till that main case is actually wrapped upward is far too late. laworder.net

Before you begin a company, legally, you have to obtain any kind of business permit or enable required legally. Doing therefore will identify your company, protect public safety and health and assist you to keep an eye on your particular predicament for taxes purposes. lawethics.net

Just like the procedure for e-discovery is actually complex as well as complicated, so may be the task of fabricating the best option between moving the process in-house or even outsourcing. A decision really should not be made with no clear familiarity with the benefits and drawbacks of each approaches because the wrong proceed can place a firm’s reputation in danger. lawdiscovery.net

It happens often when a job candidate is not truly granted a good indefinite leave to stay. To counter-top this; you might like to consider a good appeal with regard to ILR refusal towards the UK Edge Agency based on the UK Immigration regulation. lawappeal.net

Retro is usually defined since the style that’s outdated or even outmoded but trendy as well as fashionable. These days, the vintage style continues to be prominent in several areas such as fashion. Retro style involves putting on clothes which are used previously. fashionretro.net

Haute Couture is really a French expression for higher fashion. Couture indicates dressmaking, sewing, or needlework as well as haute indicates elegant or even high, therefore the two mix fashioncouture.net

You might be stuck with regard to ideas on which to wear for those who have the honor to be invited for an 80s style party. You might want to do a few research upon 80s style trends before assembling your ensemble even though you fancy your self an 80s style expert. thinkfashion.net

Seasonal handbags really are a smart expense. They can offer a fast fashion boost for almost any woman on the budget. Rather than worrying regarding replacing a whole wardrobe with regard to summer, women can purchase seasonal bag design which will enhance virtually every piece in their wardrobe. quickfashion.net

Whatever age a particular individual is actually, he/she includes a unique taste for that clothes, add-ons, and clothing that your girl friend will wear as well as use. Since the start of time, people have their very own fashion affirmation. They adhere to trends associated with clothing. expertfashion.net

Shop Insurance policy and packages are made to cover stores and list premises and all of the risks that the shopkeeper might face within the daily running from the business. Most plans contain an array of liability as well as property handles as regular, with an opportunity to choose amounts of buildings, material and share cover. shoppingco.net

Very few Get paid To accomplish Offers sites available are because successful because Treasure Trooper as well as Cash Cage. They would be the most dependable, desired as well as pro shoppingtreasure.net

Brighton is actually renowned due to its two most in-demand shopping districts in the Lanes as well as North Laine exactly where lovers may surely discover the perfect gift for special somebody. The Lanes is really a virtual buying paradise, packed to filled with designer shops, specialty stores and distinctive independent shops. It’s a great place to provide your credit-based card a great workout. shoppingspecial.net

An on the internet shopping sale is definitely an interesting time for them to do your own shopping. Leading online stores will offer global manufacturers and nearly their entire selection of products for the shopping enjoyment. So, why wait in purchasing a good high quality product due to the fact the price doesn’t suit your financial allowance! shoppingsales.net

According towards the new numbers, shop cost inflation flower further within April. The actual BRC-Nielsen Store Price Catalog has learned that general shop cost inflation was at two. 0% within April, upward from 1. 2% within March. The increase means one more pressure on individuals who are already experiencing debt. shoppingprice.net

Should you expect to become doing a few serious Xmas shopping this year, now may be the time in order to plan the amount of you may spend as well as where it will cost your cash. Indeed, the typical American loved ones spends a lot more than $700 every year on vacation gifts, a lot more than that based on some observers. shoppingbudget.net

Nowadays the very best and exciting method to do buying is on the internet where it’s convenient as well as endless hours to go to one store to another, checking away product following product. You’ll be able to get all of the bargains you would like without exiting the comfort of your house. It is going on through the web. shoppingaffordable.net

Do you need to do online buying the old-fashioned method? Do you have search engines to search for the item you would like? Shopping developments have revolutionized as well as evolved even to another level using the phenomenon associated with social buying networks. bonusshopping.net

Do a person hear regarding stories of individuals who earn cash through blogging and also you yearn to complete this also but don’t know how? Blog marketing is becoming one of the greatest internet advertising trends in the present internet business community, and this short article will highlight how you are able to join the actual fun within 5 simple ways. blogshopping.net

In this short article you’ll discover the easiest method to become a good alpha man. If you need to attract much more women into your lifetime, achieve much more social standing then this short article is essential read for you personally. Becoming a good alpha male is really a skill that may be developed the same as dancing. alphashopping.net

Exclusivity, rarity, or uniqueness is really a very powerful psychological trigger for that right item or the best situation. The fundamental concept would be to make the chance feel that she / he is special-that you’re really enabling that prospect to purchase a specific product which few people can acquire regardless associated with price. (Sugarman, 1999). newsexclusive.net

One day following the discovery of the program the diary page could be released with regard to publication without the idea of what Used to do going in reverse on me personally. I might or might not be around to determine this journal page proceed public and when I’m not really around unique provisions happen to be made with this page to become released. newsdiary.net

With this global, busy environment, situations can alter as quickly as blinking a wrist watch. For this particular reason, news blogs have become increasingly well-liked tools with regard to succinctly as well as concisely cutting towards the chase as well as presenting this news in a manner that makes probably the most sense for that most individuals. newsblogger.net

We just about all love the pets. Some people are completely crazy more than our cats and the dogs. We love how they meow whenever they’re depriving or fun. We love all of the different barks the pooch makes based on what her / his situation or even mood tend to be. petmeow.net

You have a few choices whenever deciding where you can buy dog rats. You can purchase from dog shops, or directly from the pet rat seller. It is better to get them young, as this provides you with more time for them to train all of them, and in order to socialize together. buypet.net

Top trainers claim that every dog learn to spend tranquility in the pet cage. This allows you to have the satisfaction that he or she isn’t engaging in anything whenever you can’t view him. If your dog crate lies in the actual lounge space or an additional area where there are several activity, he may still participate the circulation without obtaining underfoot. bigpet.net

In this posting I’ve detailed some businesses that focus on various kinds of vacation deals. You’ll discover services as well as information for a myriad of great travel packages for solitary people. That you can do a easy search from Google or even any search results to obtain the website of the company. travelresort.net

Deciding on where you can go to have an October 1 / 2 term holiday could be a little difficult. Some locations are too much to travel just for a 7 days, others just do not have the correct weather in those days of 12 months. Holidays within Cape Verde are a perfect solution with regard to families attempting to take a rest during the school holidays. touristholiday.net