I’m unable to get the crawler to properly authenticate to our sharepoint site. We’re using HTTP Basic authentication, and I’ve confirmed that I can browse to http://myUser:myPassword@OurSharePointSite/_vti_bin/Webs.asmx?WSDL

I’m setting up the collection using ignoreperms=1, ip=<<our IP or name - I’ve tried both), and leaving site blank or setting it to “/”. I’ve tried setting username to both myName and myDomain\myName.

All combinations give me the same “401 Unauthorized at (eval 3) line 49” error.

In the crawlManager.log I see:
Jul 30 13:39:35 searchdaimon crawlManager[29809]: ERROR: Starting crawl of collection “mySharePoint” (id 198).
Jul 30 13:39:37 searchdaimon crawlManager[29809]: ERROR: documentError: level: 1
Jul 30 13:39:37 searchdaimon crawlManager[29809]: ERROR: 401 Unauthorized at (eval 3) line 49
Jul 30 13:39:37 searchdaimon crawlManager[29809]: ERROR: Error: Problems in crawlfirst_ld.
Jul 30 13:39:37 searchdaimon crawlManager[29809]: Finished crawling of collection “mySharePoint” (id 198).

You shouldn’t happen to have an “@” in the password like maxit in this thread: http://www.searchdaimon.com/forum/viewthread/609/ ?

No - pure alpha and numeric, with upper and lower case. 8 characters. Complexity shouldn’t be an issue with this one.

I’ve done some debugging and narrowed it down to a GetAllSubWebCollection call. I’ve found others with the same issue with this call, such as http://www.tech-archive.net/Archive/SharePoint/microsoft.public.sharepoint.portalserver.development/2004-05/0663.html but d,on’t know how to resolve it.

I also found some sample powershell code to call the service (http://www.c-sharpcorner.com/UploadFile/anavijai/how-to-get-all-the-sub-web-sites-using-sharepoint-2010-web-s/) and found that the call works fine when runn from the Sharepoint server itself, but not when run from anywhere else.

So it appears that this is a SharePoint issue rather than a SearchDaimon issue… but I have no idea how to work around it at this point.

I think I got it. According to this post http://msdn.microsoft.com/en-us/library/webs.webs.getallsubwebcollection(v=office.12).aspx (and my own brief test), replacing GetAllSubWebCollection with GetWebCollection resolves the issue.

Although calling GetWebCollection works from PowerShell on my own computer (whereas GetAllSubWebCollection returns a 401 error), unfortunately even GetWebCollection returns a 401 when called from the search engine… so now I’m stuck again. I suspect that running it as SharePoint admin may get around the issue, I was trying to avoid doing so for security purposes.

What version of SharePoint is this?

The crawler debug reports it as Sharepoint version: 12.0.0.4518

After looking at the version and our current patch status, I’ve decided to spend some time this weekend bringing our SharePoint server up to speed with the latest patches. With any luck, that will resolve the issue for us.

Sounds good, I will make sure the version we use for testing is updated also. I can also look into if there is a version of the newest SharePoint available on Amazon Web Services, so we can setup a public accessible SharePoint for testing. It is probably easier to ask around if anyone else know what the error is if one has something to test with

If we are lucky Microsoft have changed something that others already have found out how to resolve.

Bdw, you don’t have an isa (http://msdn.microsoft.com/en-us/library/ms811801.aspx ) server in front of your SharePoint?

No, no ISA. Our Sharepoint is for internal use only.

I’ve run into the same problem, did you solve it eventually?

We do have an ISA however.

I did not - but based on my research, I’m reasonably confident that bringing SharePoint up to date will resolve it (or at least get me past the current problem it’s having). Unfortunately, getting the required downtime for our SharePoint site has been problematic for me. I expect to be able to try it out in October.

What version of SharePoint are you on Slogstorm?

2007

This should work. We do have a SharePoint 2007 server in our test setup, and do specifically test for it to work.

Your server shouldn’t by any means be publicly accessible so I can have a look?