Excluding tag cloud words

Posted: 22 March 2012 11:27 PM

[ Ignore ]

Newbie

Total Posts: 29

Joined 2012-02-10

Thanks to Runar, we can use SD to crawl our internal WordPress sites and it works great.

We have one wrinkle I have yet to resolve and that is to prevent/exclude the indexing of the tag cloud words. A common standard widget on many WP sites is a tag cloud. Since each ‘tag’ in the cloud is a link to all pages with that tag, and every page on the site has a tag cloud - if you search for any word that is the tag cloud, you end up with results for all pages on your site….which isn’t very helpful.

While I can modify the robots.txt to exclude all pages with /tag/ in the url, what I haven’t been able to resolve is not indexing the actual tag words that appear in within the tag cloud widget on every page. Fortunately, the ranking tends to put the more relevant page at the top, I’m just trying to reduce the clutter (though it may not be practical/possible to do so).

We use SquareSpace to host our external website and they use a custom CMS that has similar issues.

Any suggestions?

topry

Posted: 22 March 2012 11:45 PM

[ Ignore ] [ # 1 ]

Newbie

Total Posts: 29

Joined 2012-02-10

Runar - fyi, the tag cloud in WP does have a unique classID:
<div class=“tagcloud”>

Unfortunately, I have not been successful in modifying the connector to exclude that - is that possible?

Runar Buvik

Posted: 23 March 2012 12:07 AM

[ Ignore ] [ # 2 ]

Administrator

Total Posts: 253

Joined 2010-09-13

Off the top of my head: Mabye we can implement a <noindex> function which only prevents indexing of the content between the tags, not a whole web page. For example:

<body>
Do index this text block.
<noindex>Don’t index this text block</noindex>
</body>

Then modify the WordPress crawler to remove everything between the <noindex> tags before adding it to the ES.

Are you able to edit you’r system to add <noindex> code before and after the tag cloud?

( Fyi: <noindex> isn’t a standard yet, but several enterprise search engine have started to support it. Se: http://en.wikipedia.org/wiki/Noindex#Russian_.3Cnoindex.3E_tag . The ES will probably also have backed support for it soon. )

Runar Buvik

Posted: 23 March 2012 12:12 AM

[ Ignore ] [ # 3 ]

Administrator

Total Posts: 253

Joined 2010-09-13

topry - 22 March 2012 11:45 PM
Runar - fyi, the tag cloud in WP does have a unique classID:
<div class=“tagcloud”>

Unfortunately, I have not been successful in modifying the connector to exclude that - is that possible?

Yes, think so. Do you have an example of how the html of the tag cloud looks like between the <div class=“tagcloud”> and the ending </div>.

topry

Posted: 23 March 2012 12:13 AM

[ Ignore ] [ # 4 ]

Newbie

Total Posts: 29

Joined 2012-02-10

Short answer: I don’t know

The tag cloud is a standard plug-in that is included/part of WordPress (3.x)- I will have to do some research to see if its possible to modify the HTML/CSS info that it inserts into each page. I will get back to you - thanks for the quick reply.

topry

Posted: 23 March 2012 12:24 AM

[ Ignore ] [ # 5 ]

Newbie

Total Posts: 29

Joined 2012-02-10

Following is a sample of the tag cloud html:

</li><li id="tag_cloud-3" class="widget widget_tag_cloud"><h3 class="widgettitle">Tags</h3><div class="tagcloud"><a href='http://kb.wp.mydomain.com/tag/arrange/' class='tag-link-73' title='1 topic' >arrange</a>
<a href='http://kb.wp.mydomain.com/tag/axys/' class='tag-link-62' title='2 topics' >axys</a>
<a href='http://kb.wp.mydomain.com/tag/bloomberg/' class='tag-link-61' title='2 topics' >bloomberg</a>
8pt;'>Templates</a>
<a href='http://kb.wp.mydomain.com/tag/terminal-server/' class='tag-link-40' title='1 topic' >terminal server</a></div> 

Runar Buvik

Posted: 23 March 2012 12:50 AM

[ Ignore ] [ # 6 ]

Administrator

Total Posts: 253

Joined 2010-09-13

To solve this kind of problem I normally just fires up The Regex Coach and sees if I cant do a “non greedy match” whit regex, like this ( the yellow text is the part that will be removed ):

Then add it as a regex in the Perl code.

Worked easy with your html example code. Hold on a couple of minutes her, writing a patch to the crawler now

Runar Buvik

Posted: 23 March 2012 01:56 AM

[ Ignore ] [ # 7 ]

Administrator

Total Posts: 253

Joined 2010-09-13

Found a sulution

Aroun line 342 wher we have the $crawler->add_document() you can do this:

+   # Remove tag cloud to minimize noise.
+   my $htmlcontent = $response->content;
+   $htmlcontent =~ s/<div class="tagcloud">.*?<\/div>//is;
+
    print "Added: $url_normalized\n";
    $crawler->add_document(
      url     => $url_normalized,
      title   => $title,
-     content => $response->content,
+     content => $htmlcontent,
      last_modified => str2time($response->header('Last-Modified')),
      type    => $ct,
      acl_allow => "Everyone", 

Wher + is ro add code, - is to remove code.

Or use the updated crawler code at http://www.searchdaimon.com/source/boitho/websearch/crawlers/Wordpress/main.pm ( just find Revision 1.3 and click on view text).

topry

Posted: 23 March 2012 03:05 PM

[ Ignore ] [ # 8 ]

Newbie

Total Posts: 29

Joined 2012-02-10

Thanks Runar! That is helping -
One quick follow-up, if I can. I have found some additional classes that I need to exclude that the WP theme is embedding in the page for pagination.

for example: <div id=“pagination”>

To modify your edits:
$htmlcontent =~ s/<div class=“tagcloud”>.*?<\/div>//is;

How would I append multiple sections? (I’m afraid Perl is not one of my languages - currently, but guess I need to bone up on it).

Runar Buvik

Posted: 23 March 2012 03:54 PM

[ Ignore ] [ # 9 ]

Administrator

Total Posts: 253

Joined 2010-09-13

You cold use grouping and or’ing like this to match both classes and id’s of type “tagcloud” or “pagination”.


$htmlcontent =~ s/<div (class|id)="(tagcloud|pagination)">.*?<\/div>//isg;

But I never do that, because it quickly becomes impossible to understand what it really does

Instead just go thru the data several times, one time to remove “tagcloud”, then a second time to remove “pagination”, like this:

# Remove some html to minimize noise.
my $htmlcontent = $response->content;
$htmlcontent =~ s/<div class="tagcloud">.*?<\/div>//isg; # Removing the tag cloud
$htmlcontent =~ s/<div id="pagination">.*?<\/div>//isg; # Removing the pagination 

I would highly recommend that you create the regex pattern in Regex Coach first, then add it to Perl. Just remember to escape all special characters like “/”. Se list of metacharacters at http://perldoc.perl.org/perlre.html#Regular-Expressions .

topry

Posted: 26 March 2012 11:54 PM

[ Ignore ] [ # 10 ]

Newbie

Total Posts: 29

Joined 2012-02-10

Thanks, this works very well and RegEx Coach is a nice tool. Since all WP themes are different it requires some tweaking for each site with a different theme, but definitely doable. Having the ability to customize the connector to filter the results is a great feature.

Generalservices

Posted: 27 July 2016 07:26 AM

[ Ignore ] [ # 11 ]

Sr. Member

Total Posts: 805

Joined 2016-03-16

There are usually literally 1000s of possible puppy house tips waiting so that you can make them your own personal. Whether it’s really a complex artist dog kennel, or a straightforward traditional puppy house, the probabilities are infinite. hosueideas.us

This is a listing of creative vacation blogs i always read and also follow. They may be written simply by independent vacation writers, the record include those who I take into account as heavy-weights inside travel blogging and site-building. mrtravel.us

Cultural has a bearing on are stronger once we consider the particular economic aspect of the trend industry. Business wants market where it could present refreshing ideas and also products atlanta divorce attorneys new evening. fashionhosue.us

Well, I’m here to share with you the secret to having healthy meals readily available anytime and each and every time is organizing and having sensible food staples. healthiness.us

There is a thing that no a single else is fore warning you about the law of Interest and it’s about time you observed it. It really is what My partner and i call the particular WOW aspect. It is while using the law regarding attraction coming from inside out there. insidelaw.us

Not simply does a lower interest fee mean lower premiums but it will also save an individual substantial money on the life of one’s mortgage. However choosing the best rates may be difficult if you can’t have a quality mortgage fee shopping assistance. shoppingadvice.us

Everyday can be a day on the beach! ” because the old declaring goes, even though everybody knows we can not really spend each day at the particular beach. Nonetheless, you can easily experience each day at the particular beach together with beach residence decor. housedecor.us

As a result of it’s abundant heritage, cozy friendly folks and unspoiled dynamics, Cambodia can be a fabulous vacation spot for high end tours. luxurytour.us

Any fashion symbol is an individual whose garments, hair, or total appeal continues to be imitated simply by everyone and contains survived years after. fashionicon.us

That is certainly what counts one of the most. Once an individual develop this kind of healthy practice and stay with it consistently, planning out there your healthful day the night before can be your ritual and the ones pounds can really commence falling off your system. healthyday.us

Generalservices

Posted: 27 July 2016 07:26 AM

[ Ignore ] [ # 12 ]

Sr. Member

Total Posts: 805

Joined 2016-03-16

Why you will need professional legislation advice in case there is accident injury can be a query of several around the world. This question just isn’t limited to be able to any location or region as incidents occur everywhere and the great need to know about legal professionals. lawadvice.us

The emergence with the Internet being a booming market place outlines the importance for on the web shopping sites. All customers resort for the Internet when we should pick upwards something fresh. shoppingblog.us

When it concerns college dwelling, the nonprofit student property development option is frequently overlooked. Nonetheless, there are many benefits to proceeding this course, starting with all the development’s key focus: the particular young school scholar. housedevelopment.us

There are tons of items that make the holiday just in which bit far better when choosing this sort of travel. Below are a few of the huge benefits that My partner and i find lead to a fantastic small party tour. grouptour.us

Personal Bags are already around for 1000s of years. Bags symbolize a blend of technological and also social changes and so they express ethnic advancements inside fashionable phrases. fashionkeeper.us

How good an insurance plan is is dependent upon how properly it suits the average person. But, if you are looking for a place to begin, it could be good to check out these a few top medical insurance providers initial. tophealth.us

According for the Law regarding Attraction, you’re being a giant magnet pulling into your daily life whatever you give attention to. You can easily apply this kind of basic principle with the Law regarding Attraction to bring out the most effective in other folks. lawcafe.us

Now per day in the times of overall economy spending money on unnecessary things can be a very unwise action to take. With the particular recent belt shrinking measures regarding governments around the globe shopping regarding non-essential things can be a very bad strategy to use. shoppingguru.us

The simplest way to improve your property security is always to install an outdoor camera safety camera method. It is straightforward to carry out and less costly than it may seem. Read on to acquire more information about one of the better there will be. improveyourhomes.us

Generalservices

Posted: 27 July 2016 07:27 AM

[ Ignore ] [ # 13 ]

Sr. Member

Total Posts: 805

Joined 2016-03-16

A fresh series in regards to the world regarding fashion as well as the top artist labels that define it. Each sequel will give attention to a key top trend brand, the mission, and its particular history. Inside our first sequel we examine Virgins, Saints, and Angels. fashionsaint.us

Marketing virtually any business not online with enterprise cards and offer fliers needs to be in your game plan. These are usually two great types of real planet non-internet advertising and marketing possibilities you could take a look at today. anybusiness.us

Law occupations are being among the most satisfying and also fulfilling occupations around. Winning an incident is just like striking gold of course, if you, is likely to way understand that justice continues to be served, and all as a result of your work, you sense absolutely fantastic. lawcareer.us

It will be never prematurily . to commence your Holiday shopping. The intelligent shoppers know you could already uncover what the very hot products will probably be this tumble. Use our own list to assist you avoid the particular rush. shoppingnow.us

Regardless of reasons why you’ll probably decide a no more luxury at home, the luxury do-it-yourself tips will provide you with the tips on where to start. luxuryhomeimprovement.us

By discovering multiple situations that will confront leaders inside their career way, this write-up brings to focus forcefully in which leadership style can not be static to get a leader and definately will require these wearing diverse leadership less difficult, suiting many different situations. stylekeeper.us

For people considering signing up for the jet-to-let owning a home set listed below are the a few main solutions in terms of raising real-estate finance, loans or perhaps mortgages to get property in another country. localfinance.us

Well, take the time to understand your rights and flip the particular antics back around the debt enthusiast. This information from Debt negotiation Legal Guides will show you how. legalguide.us

The identical applies inside home redecorating; such tiny changes will give your home a fresh and improved upon look. It may also enhance the particular living environment inside it. A simple do-it-yourself guide are capable of doing great items for far better living. homeimprovementguide.us

Travelers have the main benefit of learning and also gathering information about different areas. Not only which they see and also explore diverse places nevertheless they also understand those areas intentionally along with unintentionally. toptravelers.us

Alver1989

Posted: 03 January 2017 11:40 PM

[ Ignore ] [ # 14 ]

Newbie

Total Posts: 25

Joined 2017-01-03

Nice website
Watch Billy Lynns Long Halftime Walk Online
Watch Resident Evil The Final Chapter Online
Watch Arrival Online
Watch Manchester by the Sea Online
Watch The Great Wall Online
Watch Trolls Online
Watch Gold Online
Watch The Founder Online
Watch Resident Evil The Final Chapter Online
Watch The Edge of Seventeen Online
Watch A Dogs Purpose Online
Watch A Dogs Purpose Online
Watch Billy Lynns Long Halftime Walk Online
Watch The Accountant Online
Watch Logan Online
Watch The Girl on the Train Online
Watch Doctor Strange Online
Watch A United Kingdom Online
Watch Passengers Online
Watch Collateral Beauty Online
Watch Rogue One Online
Watch Manchester by the Sea Online
Watch Passengers Online
Watch Billy Lynns Long Halftime Walk Online
Watch Trolls Online
Watch A United Kingdom Online
Watch Assassins Creed Online
Watch A United Kingdom Online
Watch Fantastic Beasts and Where to Find Them Online
Watch The Lego Batman Movie Online
Watch Collateral Beauty Online
Watch La La Land Online
Watch The Space Between Us Online
Watch Jack Reacher 2 Never Go Back Online
Watch Sing Online
Watch Sing Online
Watch The Great Wall Online
Watch A Cure for Wellness Online
Watch Trolls Online
Watch Doctor Strange Online
Watch Manchester by the Sea Online
Watch The Girl on the Train Online
Watch Rogue One Online
Watch The Accountant Online
Watch A Monster Calls Online
Watch Assassins Creed Online
Watch The Space Between Us Online
Watch Tulip Fever Online
Watch John Wick Chapter 2 Online
Watch Assassins Creed Online
Watch A Cure for Wellness Online
Watch Fifty Shades Darker Online
Watch Split Online
Watch Jack Reacher 2 Never Go Back Online
Watch Underworld Blood Wars Online
Watch Jack Reacher 2 Never Go Back Online
Watch Gold Online
I am pretty much pleased with your good work.

‹‹ ATOM crawler? 301 Redirect on Robots.txt ››

Products

Demo

Documentation

About

Community