Hi,
I am just starting to evaluate what appears to be an excellent program.
 
I have set up a test crawl on a shared directory, and checked the output.  It seems that it is not indexing the content on my Corel Wordperfect files (it is indexing their filenames, but no content). 
 
Is there a list of file formats supported somewhere?  (In English? )
 
Are there plugins available for formats not supported?
 
Thanks
 
PS:  Not wanting to start on a bad note, but I must say, the Captcha test I had to pass to be able to register for this forum was TOO HARD. Took me maybe 20 attempts before I could convince it I was real.
 
I am sure it could be made a little less difficult, while still keeping the spambots at bay!

It seems that it is not indexing the content on my Corel Wordperfect files (it is indexing their filenames, but no content).

Wordperfect is supported, or at least some Wordperfect files (it may be several different versions of the Wordperfect format out there ). What endings are you using in the file names ?

Full list of supported file endings:

mp3, dot, rtf, wpf, sdw, doc, xls, zip, dwg, eml, jpg, png, tif, tiff, bmp, gif, eps, ai, psd, msg, docx, docm, dotx, dotm, xlsx, xlsm, xltx, xltm, xlsb, xlam,pptx, pptm, potx, potm, ppam, ppsx, ppsm, sldx, pdf, ppt, pot

PS:  Not wanting to start on a bad note, but I must say, the Captcha test I had to pass to be able to register for this forum was TOO HARD. Took me maybe 20 attempts before I could convince it I was real.

Thank you for the heads up on this one. Will look into it.

We are using a reCaptcha, which is a service from a 3-party. Maybe they have made it harder to solve lately? Hopes this don’t means that a lot of users have given up registering…...

Wordperfect uses .wpd
So, looks like that one isn’t read.  Pity, but not the end of the world. Is there any capability to add more filters?
Also, what about plain old .txt??!!

Now I think of it, what if there is no extension or it is wrong?  Eg if I have a .pdf, but it is named test.txt, will it be read?  ( I think linux reads content to determine format, rather than extension, doesn’t it?)

Wordperfect uses .wpd
So, looks like that one isn’t read.  Pity, but not the end of the world. Is there any capability to add more filters?

That explains it. I think we may have belivede that Wordperfect only used .wpf as file ending. You can add .wpd your self manual if you want:

Log in to console as root.

Open the Abiword runinfo file for editing: nano -w /home/boitho/boithoTools/fileFilter/abiword/runinfo

Add:

documentstype: wpd
command: /usr/bin/abiword—to=txt #file
outputformat: textfile

Save and close: ctrl-o, ctrl-x

Then on the web based administrator panel go to Manage services and restart the Document Management service.

Also, what about plain old .txt??!!

Now I think of it, what if there is no extension or it is wrong?  Eg if I have a .pdf, but it is named test.txt, will it be read?  ( I think linux reads content to determine format, rather than extension, doesn’t it?)

.txt and .htm* is also supported. For thus we don’t use a 3-party converter so they was missing from my list. Sorry about that.

File format conversions is based on file extension. If you rename test.pdf to test.txt it will be treated as text.

Thanks for your suggestion. Unfortunately, it does not seem to have worked.
I edited the file as described, saved and closed, reopened to ensure that the setting was properly saved, then restarted the document management service, then re-crawled my file share with the wordperfect documents on it.

same result - file names indexed, but no content.

I then restarted ALL of the services in the ‘manage services’ section, and re-crawled.  Still no luck - no content is being indexed for wordperfect documents.

I checked in the search results and the ‘cache’ version of the wordperfect files has no content. (the cache version of pdf and .doc files etc includes all the content)

Any other suggestions?

Also, and I am happy to open this as a new thread if you prefer, but how do I enable SSH? I see your answer to another post that it is disabled by default, however I prefer working in a putty session than using the VMWARE console (I am using ESXI server).

Also, and again, if you want me to move this to a new topic, happy to do so:  I have had a little experiment with permissions, and it seems to me that if I have permissions set on a collection as ‘public’ and permissions on another collection as “system”, and then I use the ‘private’ search page (ie, I am logged on as a specific user) then I cannot search the ‘public’ collection ??!!!  It seems strange (ok..wrong   ) that if I am logged in with credentials, I cannot access ‘public’ data and have to connect as public to see it?!  Is this actually how it works, or am I doing something wrong?

Sorry to bombard with questions.  I can’t find much documentation.  Is there some more documentation available somewhere other than the links below?

Thank you for your feedback! We do actual value it, and uses user feedback extensible when planing further development of the ES.

Documentation is available at http://www.searchdaimon.com/documentation/ and faq at http://www.searchdaimon.com/documentation/faq/ . But some of your question is unfortunately to technical to have made it into the documentation yet

For your Wordperfect problem; I did verify that Abiword can convert .wpd files, so that should have worked. I will set up a server to test it later today. I haven’t used Wordperfect my self sins the late 90’s. But did find some old .wpd over at http://www.opentestsearch.com/test-sets/ I plan to use for testing. Are your Wordperfect files also old, or are you using a more resent version of Wordperfect ?

You can enable ssh like this: http://www.searchdaimon.com/documentation/C36/#getting_ssh_access_for_the_boitho_user

  I have had a little experiment with permissions, and it seems to me that if I have permissions set on a collection as ‘public’ and permissions on another collection as “system”, and then I use the ‘private’ search page (ie, I am logged on as a specific user) then I cannot search the ‘public’ collection ??!!!  It seems strange (ok..wrong   ) that if I am logged in with credentials, I cannot access ‘public’ data and have to connect as public to see it?!  Is this actually how it works, or am I doing something wrong?

I believe that that is the way it works, and it’s both wrong and dumb. It will be straiten out in an update soon. The only exception is for websites collections. Thus should behave as expected.

Ps, yes, feel free to make a new topic for different, new questions. It make it easier for others if they browse around the forum for answers.

Wordperfect version:  The bulk of the files I want to index have been created with Corel Wordperfect X3, which came out in about 2006. (Latest ver seems to be X6.) 

SSH:  Thanks for the link. I have this working now. Much easier to work with, thanks.

Website collections…. having awful trouble getting mine to work.. will post in new topic!

But did find some old .wpd over at http://www.opentestsearch.com/test-sets/ I plan to use for testing.

I had a look and couldn’t find any true wordperfect files there. There were some files with extension .wpd, but I tried a couple at random, and they were actually MSWord files.

I would be happy to send you some Wordperfect files for testing if you would like?

Have gotten it to work!: http://ec2-50-17-12-100.compute-1.amazonaws.com/webclient2/indexanon.pl?query=enron&lang=en&x=0&y=0 (may not be up for long).

Unfortunately there where a couple of issues:

1) If serving the documents from a website, be sure that it is served with HTTP Response Header Content-Type “application/wordperfect” ! Use http://web-sniffer.net/ to verify if it’s not working. On my Red Hat server .wpd was missing as mime type and got served as text/plain. That wont work. Add it to /etc/mime.types or add a “AddType application/wordperfect .wpd” in your http.conf file.

2) The forum did translate the text I posted to add to the configfile to something that don’t work. It’s supposed to be this: http://www.searchdaimon.com/div/wpdsetup.txt at the end of the /home/boitho/boithoTools/fileFilter/abiword/runinfo file.

You can use this documents to test: http://bbh-001.boitho.com/div/wpdtest/ .

Remeber to restart the document management service, and use the Collections->Manage->Advanced management [Force full crawl now] button when recrawling after doing changes. Just normal crawl wont cut it because the ES will skip the document because is hasn’t changed.

I had a look and couldn’t find any true wordperfect files there. There were some files with extension .wpd, but I tried a couple at random, and they were actually MSWord files.

Did find some and uploaded them at http://bbh-001.boitho.com/div/wpdtest/

I would be happy to send you some Wordperfect files for testing if you would like?

Yes, if the above don’t resolves this issue I wold love to have some to test with. Please email me at: rb [at] searchdaimon [dot] com

Thank you for your super fast response.  I looked at your sample, and it sure seems to be working well. I think the error was mine.

I did have the correct entry in the /home/boitho/boithoTools/fileFilter/abiword/runinfo file.

What I didn’t do was “use the Collections->Manage->Advanced management [Force full crawl now] button when recrawling after doing changes.” 

I am very excited to try this, however my system tells me it is busy already crawling another collection of the same type.  I don’t want to stop that one either, so I will test tomorrow when it is finished and report back. 

Thanks again.

It works!  The step I had missed was forcing a full crawl - I just clicked the ‘crawl now’ link on the collections overview page.  I can confirm that the steps as instructed above allow you to index Wordperfect files. Thanks!

I have found that a lot of my wordperfect files also use the extensions:  .frm and .wpt   so I have added two more paragraphs into the /home/boitho/boithoTools/fileFilter/abiword/runinfo file:

I have restarted the docs management service, and am re-crawling now.  It is a rather large collection…

I will advise when it finishes whether this allows these other wordperfect files, with these .frm and .wpt extensions to be indexed as well.

Yes.  Putting those additional entries in the /home/boitho/boithoTools/fileFilter/abiword/runinfo file allowed the crawler to index the Wordperfect files with .frm and .wpt extensions.

Thanks! Great to get this straiten out.

I will add the settings for .wpd, .wpt and .frm to the official release so other users can benefit from this also.

PS:  Not wanting to start on a bad note, but I must say, the Captcha test I had to pass to be able to register for this forum was TOO HARD. Took me maybe 20 attempts before I could convince it I was real.

Thank you for bringing this to our attention. We have disabled thus tests for now.

We are using a 3-party service from reCAPTCHA/Google. They most have don some changes on there end. Did run some test today, and many of the captchas was just a blur, or had part of the word missing:

Seems like .msg is a supported format, but it doesn’t seem to be indexing for me. The Documents comes out blank when I look at it through Manage collection in the admin ui.

Thank you for your feedback. You have found a bug!

MSG files are supposed to be supported, but unfortunately it has stopped working because of a bug in an install script. We are planning an update to address this.

In the mean time you can install it manually by login in to the console as user root and run the following:

yum install perl-OLE-Storage_Lite.noarch perl-MIME-tools.noarch

Press “y” when prompted.

The root password will be viable on the console if you haven’t changed it.

After you have install as described above you will have to recrawl the collection where you have the MSG files. Do so from the administrator interface by going to manage for the collection and click on the “Force full crawl now” button under the “Advanced management” tab.

If you are unable to do this or it does not work, let me know so I can find another way of helping you.