We are evaluating the VM version of your product to try and create a centralized search of external hosted (anon) sites, intranet secured sites and file shares. Thank you for creating the eval as a VM - very smart move on your part, makes it easy for SMBs to test and confirm your solution will work for them.

No problem with indexing the hosted external sites that do not have access restrictions. However, I have been unsuccessful with our WordPress intranet sites.

We have a multi-site WordPress 3.2 server (Linux) on our intranet that uses an AD integration plug-in.
I can index an ‘open’ site (one that does not require authentication), but cannot index those that require AD authentication, even though I am using a ‘resource authentication’ user account that has domain admin priv.

I’ve tried cloning your intranet connector, modifying the $user variable to look for ‘username’ which is what the AD login dialog uses, but to no avail.

Running the crawl does not result in an error, it shows 2 documents indexed - which are the wp.login.php and wp-signup.php pages (accessible to anon).

Any ideas/suggestions on how I can craw/index these AD integrated WP sites?

Other comments:

‘Stop Crawl’ does not seem to work. I started an index of an external site, created another collection, realized I could only do one of each type at a time, so I pressed ‘Stop Crawl’, but it continues to index the site.  Same issue with an SMB collection.

Incremental crawls: It appears your crawl data is not available for search until the crawl completes (and ostensibly the data is then saved in the database). Is this correct?

I also tried a Windows intranet site (IIS based) set to Windows Auth, using the intranet connector and a user account that has full rights. That one returns a 401 error.

Any suggestions on how to modify the connector work with this one?

Additional info for you-
With the Windows/IIS site: If I set it to accept Basic Auth, then using your AD user system does work. As such, this one isn’t an issue for us currently, though working with standard Windows Auth would be a nice feature.

Still no success in getting it to work with the WordPress sites that also use AD integration. I tried it both with AD and your ‘connector pushed’ user system, but same results. No errors, but nothing other than the logon pages are indexed. If I specify the url to a specific post vs the default, same results.

Vmware and “virtual appliances” as they call it is has definitely been working well for us. It’s quit easy to create something that works out of the box with no software installation, and dependencies on other software being installed on the target server. It’s cloud ready, and great for testing since you easily can setup several instillation on you Vmware servers/cloud/desktop to play with.

Windows/IIS

You are absolutely correct. The Intranet crawler do’s not currently support the ntml authentication used by ISS as default. However we have a patched version in testing that do so. 

If you want you can use this new version. It appear to work great, but I am not sure how thoroughly it has been tested yet. The source code can be downloaded from:
http://www.searchdaimon.com/source/boitho/websearch/crawlers/Intranet/main.pm?rev=1.13 , then just clone the old crawler and replace the code.

WordPress

Our AD user system wont help you here. The AD user system is used for authenticating users when log into the ES to search.

When logging in to your WordPress system, do your users enter there username and password in a html form ? Like this form: http://wordpress.org/wp-login.php ?

If so, the crawler will also have to do so to. Or more correctly, it will have to sent username and password as a http-post request to your WordPress to get authenticated, then start to crawl. The problem with form based autentication is that the method for logging inn is diffrent from system to system. You therefor will have to ad some code in the crawler to sent the username and password to WordPress.

We have a local WordPress installation her. I can try to figure out what you wold have to do to get it working.

Incremental crawls

Yes, the first time the crawler runs it first crawl all data, then creates the search index. If you want to just do a small test crawl on a large data source, there is a function for that called “Test crawl” in the advanced management tab for each collection.

Stop Crawl

I will look into this.

Thank you for the prompt response.
IIS: Since its currently working with Basic Auth, I can make that work until you push that mod into production. It is but one site that we have and its being retired soon, so it’s not our priority.

WordPress is our primary intranet source, so any help you can provide with that would be appreciated.
Yes, the default login form like what you listed is what we use. In addition we are using this plug-in for AD integration:
http://plugins.ten-321.com/category/active-directory-authentication-integration/
It works great, just need a way to crawl the content.  I have yet to find a way to get through the forms based auth dialog.

While the native WP search (and plug-ins) work great for standard WP content, yours is the first I have found that will (hopefully) do what we need - aggregate multiple sources in one search - AND allow restriction based upon AD account.

I look forward to your follow-up on WordPress

As I understand it, to create a intranet with Wordpress, you have to use some kind of plugin to restrict access to your internal pages. For example this plugin: http://wordpress.org/extend/plugins/intranet-restriction-for-posts-and-pages/ .

What plugin are you using for this ?

No restrictions, just using an Active Directory plug-in for authentication vs standard usernames/passwords.
The plug-in we are using is:

http://plugins.ten-321.com/category/active-directory-authentication-integration/

I just setup a test Wordpress site at http://wptest.searchdaimon.com/ . I din’t get any option to restrict access to only logged in users. Insted I can create users the has this roles: http://codex.wordpress.org/Roles_and_Capabilities#Summary_of_Roles . Thus roles only restrict if/how a user can post blog post and pages.

There is no option to restrict access. If you don’t use a plugin to restrict access, how is it then done?

The ad plugin only authenticates the users. I can find any information about it also enable you to restrict pages to only the users logged inn thrue the ad….

I guess I didn’t explain it well. We are only using the AD authentication plug-in for authentication / logging into the WP site, vs using the standard WP methodology of username/password.

Once the user logs in, they have full access to view everything on the site.

I could not get the default Intranet connector to get to the site. It indexes only the wp-login.php page.

Does that explain it better?

WordPress doesn’t support to restrict access to logged in users out of the box. I wanted to know what changes you had done to your installation so I could replicate it on my end Anyway, I have set up a test of my own test site at http://wptest.searchdaimon.com/ using instructions from http://wordpress.org/support/topic/members-only-sections?replies=31 , and created a WordPress connector that works there.

To use it, you must add a new connector from the connectors menu at the admin panel. Then on the Settings and parameters tab, do the following:

* Rename the new connector to myWordpress, and click update.
* Select accesslevel and authentication as input fields.
* Set it to be “Ready for use”.
* Add two parameters “url” and “loginurl”.

You should end up with somting looking like this:

Then edit source, and add this code: http://www.searchdaimon.com/div/wptest/wp.txt

Now save it, and go to the add manually section and create a new collection using your myWordpress connector. loginurl will be the url to the WordPress log in form, and url is where the crawler should start to crawl when finished logging in.

This at list works on the http://wptest.searchdaimon.com/ site, using loginurl = http://wptest.searchdaimon.com/wp-login.php and url = http://wptest.searchdaimon.com/ .

Ps: You should probably setup a robots.txt file to block access to some of the WordPress virtual directory’s for the sdbot. If not you will get a lot of duplicate data.

Thank you very much for working on this - unfortunately, it’s not working. Using this custom connector it doesn’t index any pages (even the wp-login.php page, which it does with the default intranet connector).

I’m assuming I would use my existing AD User System?
I tried both an AD account that has full access (super-admin rights) to the WP site as well as the default Admin account that was created upon installation, both same results. The default Admin account is not an active directory account.

I looked through your logs, the only one that seemed to have anything pertinent was:

Crawling Management Log:
Feb 15 19:24:27 searchdaimon crawlManager[29063]: Finished crawling of collection “WPMain” (id 205).
Feb 15 19:24:21 searchdaimon crawlManager[29063]: ERROR: Starting crawl of collection “WPMain” (id 205).

The loginurl is same as yours (different domain of course) http://wp.mydomain.com/wp-login.php and the url http://wp.mydomain.com

I verified the accounts by logging out, back in to the WP site.

I’m looking for logging options for the WP site to see if it will show anything, but have yet to find a way to log/trap problems with that login page.

I also checked the event log for our DC (single domain) - no errors.

I appreciate the effort you have put into this - if you have any suggestions on what I might check to help further identify the problem, I will do what I can.

Regards-
Tim

Quick follow-up.
I turned up logging on our DC to verify that the script was processing the wp_login.php page.

While logging in normally, I used an invalid password and verified that this event was logged at the DC.
I then used the same userID and invalid password in the Connector, but nothing was logged. So it appears that the script is not sending the authentication through the wp-login.php form.

While perl and php are definitely not my strong suits, I looked at the script:
# Login to Wordpress, using http post
$robot->post($opt->{‘loginurl’}, [ ‘log’ => $opt->{‘user’}, ‘pwd’ => $opt->{‘password’} ]);

I verified that the name for the input controls in the wp-login.php are the default ‘log’ and ‘pwd’ as used in your script, but based on the lack of an entry in the DC security event log when using an invalid password, it does not appear to be executing this.

If there are any additional debug/logging methods I can use to help with this, let me know.

Integration can sometime be quite a pain. Just one small differences between our two WordPress setup and it fails to work….

It is not always so easy to debug this without sitting in front of the system. If you want you can give me access to your ES or desktop computer so I can take a look with me own eyes. After several years working with integration I have picked up some tricks.

The ES has a phone home function in the admin gui that makes a ssh connection to us, enable me log in remotely (this is the best method). Or we can use a system from https://secure.logmein.com/ that enable me to see your desktop.

If this is ok by you, just use the chat function on the front page to ask for Runar. So can we take it from there.

I wanted to close this thread and let anyone else using WordPress know that it has been resolved!

I will leave it to Runar to provide any specifics and implementation requirements, but I am now able to continue my testing. We have a multi-site WP configuration, which we are using to replace SharePoint. A requirement we had was the ability to search not only all of the content, but linked / uploaded files as well.

ES is the only solution we have found that has been able to accomplish this task. Add in the AD integration, exchange, SMB and external website capabilites and it is truly impressive.

Well done to the ES team!

Therefore listed below are the reasons I believe most people make an application for the bloggers’ health program: The majority of bloggers tend to be misinformed; The majority of bloggers tend to be lazy; The majority of bloggers absence skills; The majority of bloggers absence Commitment; The majority of bloggers absence business way of thinking. legendsoflaw

From the actual BlackBerry towards the light bulb towards the telephone, Canadian creations are part of the normal life of the planet. But most finish up in US dependent companies. We depend down the very best ten Canadian technology inventions which stayed you’ll come to home. financeformanagers

Visit of Ruth Porat, because Chief Monetary Officer from Google, might be symbolized being an empowerment associated with women within tech advertising jobs. However, it’s sad this glory is actually dimmed down using the actual information showing the actual decline within the ratio associated with tech work held through women. medicomtech

Having your business known within the local area could be hard. Perhaps you have put the fundamentals in location? Follow these types of five easy local SEARCH ENGINE OPTIMIZATION principles to start to find the exposure your company needs. charterschoolpolicy

Perhaps you have faced the issue of finding the right canine house? Although thousands of styles flood the marketplace now, picking out a dog house could be tough, especially if you have a specific design in your mind or when you are interested in some genuinely unique canine house. thebeautyresource

Hollywood from the North. There’s valid reason why Vancouver is actually honoured with this type of glamourous pseudonym. In the end, it is a filmmaking centre for up to a hundred years. savenow4college

Traveling overseas is pricey. Some journey companies take your hard earned money and neglect to manage your own trip properly. Some strategies for contemplating choosing a organization that cares for that traveler below their view. bkkshopping

Faith, wish, and love would be the components associated with life. These 3 alone consider us from denote point within our lives, and provide us a chance to live the actual virtuous existence. With these types of three we could move within alignment towards the will associated with God. top—designer-brands

Finding journey deals within Europe is really a general topic however, many very obvious paths to follow along with. Lose them and will also be lost into a massive world associated with not fascinating opportunities. A large number of offers are available, with really tight costs one together. lifestyle-navi

Earlier throughout the 19th hundred years in Luxembourg, people took horse attracted carriages with regard to transportation. There had been obviously absolutely no wristwatches in those days. So metal carriage clocks were accustomed to keep an eye on time. womenfashionblog

Internet entrepreneurs discuss the device they uses within their SEO ways of log what kind of traffic stations they trigger when generating traffic to some new site. They additionally outline the actual default points they like about the content they generate. 7jewelry

Choosing the best person to have Infertility Guidance with could be a daunting job. But after you have chosen the best person, you’ll be amazed at that which you can discover. There tend to be many points to consider when selecting a therapist. shop24hr

Mobile Cherish Hunt Advertising is really a new advertising concept. It allows advertisers to place their manufacturers, ads as well as free special gifts onto a worldwide prize chart for cellular users to build up via GPS NAVIGATION. thepetcompanion

Today, the sports activities industry is considered to be the 11th largest business in Usa, producing a lot more than $213 billion dollars annually. It’s a growing as well as diverse area, including several segments such as professional groups, private as well as public businesses, en-architects

There tend to be many car insurance choices online. However before purchasing, try to consider various websites and obtain quotes through different insurance providers for your vehicle. Of course like a consumer, you need to have advisable of the cost ranges available on the market. dressydresses

Growing amount of people today depend on services which supply hassle free of charge recovery associated with assets as well as bring ways of long impending cases associated with locating investors, heirs in order to estates, traders, due refunds, and so on. visonictech

As technologies becomes a greater part individuals children’s life, pediatricians tend to be seeing increasingly more young sufferers for repeated stress accidental injuries. How may parents aid the prevention of injury? cardsforbusiness

Factors which influence the actual fee and the price of real house management. What’s the difference between your fee and also the cost? The charge is that which you sign within the real house management agreement whereas the price is the total amount you need to pay all year long to obtain service you’ll need. sdgl-edu

Traveling is definitely an interesting pastime. Travelers have the advantage of learning as well as gathering understanding of different locations. Not only they see as well as explore various places however they also find out about those locations intentionally in addition to unintentionally. house-blueprints

Medical Payments technology offers witnessed ongoing expansion throughout the first 8 weeks of come early july, as substaniated by pr releases about a few ninety items accumulated within BillingWiki. A exercise manager may fight to select the very best product from this type of large item variety. newharbortech