Hi Folks

Hope someone can help. I am looking to do the following on a private network
- scan an entire network 192.168.1.0/24
- identify those systems that are running web servers on port 80 (or a range of other ports)
- crawl the sites with search daimon

Has anyone already done this or something similar and have already written a script/connector that could achieve the desired result? I would like to hear peoples ideas.

Thanks in advance

Richard

There is a tool in the ES admin gui to do this for file servers, but not for web servers. You could try to extend the intranet crawler but this is a undocumented function, so I think you time would be better spend finding an alternative method.

Have you considered using a tool like nmap to first scan thru the network to find all webservers, then manually enter the addresses into the ES admin gui?

Unfortunately the problem may be more complex than just identify which ip addresses has a webserver on it. The webservers may be using virtual hosting. If so, you will need to know the correct domain names to index all the content.

Thanks Runar, in fact this is exactly what I did last night. Its a shame there isn’t an option to put a network range and port range into the interface. I was thinking of finding the SMB Scan and modifying it, maybe later (has some one done this already?).

I basically wrote the following lines of shell using the output from nmap (I’m sure there is a more elegant and better way in say perl)

hostx> nmap -v -sn -oA net10.10.0.0 10.10.0.0/16
hostx> cat net10.10.0.0.gnmap | grep Up | cut -c 7-18 | sed s/“10.10”/“http:\/\/10.10”/g | sed s/”(”/”“/g | sed ‘s/” “/”“/g’ | sed ‘s/$/\/;/’ > uphosts.txt
hostx> tr -d ‘\n\’ < uphosts.txt > unclean-search.txt
hostx> cat -unclean-search.txt | sed s/” “/”“/g > search.txt

the final output is search.txt which gives me a file with contents in the form of the following
http://10.10.80.129/;http://10.10.80.161/;http://10.10.80.162/;http://10.10.80.163/;http://10.10.80.171/;http://10.10.80.209/;http://10.10.80.210/;http://10.10.80.211/;http://10.10.80.212/;http://10.10.80.225/;http://10.10.80.233/;http://10.10.80.241/;http://10.10.80.242/;http://10.10.80.243/;http://10.10.80.244/;http://10.10.80.245/;http://10.10.80.246/;http://10.10.80.247/;http://10.10.80.248/;http://10.10.80.249/;http://10.10.80.252/;http://10.10.81.17/;http://10.10.81.33/;http://10.10.81.34/;http://10.10.81.35/;http://10.10.81.36/;http://10.10.81.49/;http://10.10.81.60/;http://10.10.81.61/;

They key feature here is no newline. I then open the file, select the contents and past them into the URL line of the internet connector.

The only issue now is that the crawl stops if the host is no-longer accessible. I think I am going to clone the internet connector and modify it so that it continues to crawl regardless.

I understand and agree about your virtual host comment.

A little more background. The network I am looking to index is a community wireless project. Little is centralised and finding interesting things to look at is difficult. What better target for a search engine.

I would be interested to know if anyone else has any other ideas about how to solve this.

How good an insurance plan is hinges on how certainly it suits those. But, if you want a starting place, it will be good to read these two top health care providers first of all. www.tophealth.us

According to Law about Attraction, you’re such as giant magnetic pulling into you whatever you concentration on. You may well apply this approach basic principle for the Law about Attraction to create out the best quality in many people. www.lawcafe.us

Now just a day in the periods of monetary spending money on unnecessary things is mostly a very unwise quest. With all the recent belt tensing measures about governments in many countries shopping just for non-essential things is mostly a very bad option. www.shoppingguru.us

The easiest method to improve your personal property security can be to install an outdoors camera security and safety camera model. It skilled assistance to implement and lower priced than you might think. Read on to learn more about possibly the best there is normally. www.improveyourhomes.us

This is a creative travel around blogs which i read and additionally follow. They are simply written just by independent travel around writers, the directory include individuals that I look into as heavy-weights for travel weblog. www.drtravel.us

A cutting edge series to the world about fashion and therefore the top beautiful labels that comprise it. Each set up will concentration on a primary top model brand, the country’s mission, and your history. In this particular first set up we view Virgins, Saints, and Angels. www.fashionsaint.us

Marketing whatever business offline with home business cards and provide fliers have to be in your game plan. These really are two great techniques for carpet cleaning real country non-internet endorsing possibilities that one can take good thing about today. www.anybusiness.us

Law work opportunities are extremely satisfying and additionally fulfilling work opportunities around. Winning in instances is want striking gold considering the fact that you, that you really need way understandthat justice is actually served, and all resulting from your exertions, you look and feel absolutely awesome. www.lawcareer.us

It is normally never ahead of time to get started in your Holiday season shopping. The sharp shoppers know that one can already determining the awesome products would be this crash. Use this list to help you out avoid all the rush. www.shoppingnow.us

Inspite of the reasons why to choose a a bit more luxury at your house, the luxury home improvement 123 tips will give the choices on where do you start. www.luxuryhomeimprovement.us

By finding out multiple situations which might confront leaders with their career pathway, this page brings to focus forcefully which usually leadership style is not to be static in a leader all of which require individuals wearing varied leadership a terrific way to, suiting several situations. www.stylekeeper.us

For many considering signing up the jet-to-let investor set listed here the two main choices in the case of raising home finance, loans or possibly mortgages to invest in property out of the country. www.localfinance.us

Well, take the chance to uncover your rights thereafter flip all the antics back relating to the debt financial institution. This help from A debt negotiation program Legal Guides will highlight how. www.legalguide.us

The equivalent applies for home remodel; such smaller changes will offer your home an exciting new and greater look. It will also enhance all the living environment inside. A simple home improvement 123 guide can get done great matters for more suitable living. www.homeimprovementguide.us

Travelers have the luxury of learning and additionally gathering is crucial different venues. Not only make see and additionally explore varied places but additionally also discover about those venues intentionally combined with unintentionally. www.toptravelers.us

Free magazine-style web page themes can help you your ınternet site look refined and high quality. You don’t need a high-priced proprietor to update a web site. Here really are five reasons to get a new web page theme or possibly template. www.styleblog.us

B2B websites are most successful for developing new company communities and observe after improved interaction with today’s business correspondents. It equally helps for reducing made the effort cycle part of making products and services easily there to the purchasers. www.gobusiness.us

The equal laws appear to be today that in results yesterday and will be tomorrow. When everyone enact legislation today you will not regret change yesterday evening but will offer an upgraded experience just for tomorrow. www.lawtoday.us

It is normally of awesome importance the moment searching for home enchancment loans you ought to do a lot of legwork to ascertain you determining is best for your needs. If an individual has a large home improvement 123 project coming the finding the best quality home loan is just what you want. www.besthomeimprovement.us

Travelling is really an enjoyable feel. It claims to be an even more suitable experience once you prepare for use on your journey, and using purchasing travel cover, there are really a host about other actions to better organize your voyage. www.travel101.us

Part for the beauty about deciding to go into the fashion enterprise is daydreaming about establishing a finest fashion class. Every future designer or aspiring learner of model has thought of attending amongst the top model schools on the nation certain times or anothe www.topfashion.us

Creating Home business Plans is mostly a meaningful part of the development on the online or Real world business, no matter how large you should become. It forms a good all round foundation and additionally reference for the whole family involved. www.mrbusiness.us

The majority of us wish to buy cutting edge and unique products, but the job of driving and preparing to many shops to search expected items aggravate us. At this time, one can buy all the desired items directly from their home by making usage of Internet Researching. www.internetshopping.us

Home renovation is fun and additionally rewarding just for someone who’s prepared for those problems and additionally aggravation if you have a home enchancment project can lead to. Just confirm you understand genital herpes virus treatments are establishing. www.homeimprovement101.us

In this approach exciting cutting edge century the good news is new way to turn into a travel ingredient. The days for the expensive travel around agent courses get ancient the past, and doing work in a retail center booth isn’t really required now. www.protraveler.us

 
Find a great buzz of embellishing enthusiasm to make a home environment that times your creative imagination. Make style a giant impact not to mention production when ever decorating even on a powerful psychologically and mentally . wave. Take a decorative style even on a journey from elegance towards refine, shift in front and developed beautiful in house spaces. makestyle.us

B2B web sites are very useful for developing home based business communities and gaze after improved associations with active business correspondents. It even helps through reducing the moment cycle in the middle of making obtains easily for sale to the potential customers. dobusiness.us

A legal requirement career will provide prestige not to mention unlimited making money potential. But for being an attorney encompasses way more than who. This piece of writing guides you precise through the approach of legal requirement education not to mention practice. lawforum.us

Whether you do your monthly food store shopping or getting electrical purchases for a home work, a few secrets could help you save some profit while by it. Smart store shopping means bringing control of this situation to help get the best value you can actually afford. smartshopping.us

Consider hearing very much about travel and leisure clubs not too long ago and wondering the things they are? Becoming well informed about them could help you save a small fortune. abouttravel.us

swimwear t only his own orchestra piece 8222;death and transfiguration8220; but, rhythmically slightly adapted, also the beginning of the requiem by Johannes Brahms.A journey into the world of classical music, moving, inspiring and sensual - Seal and composition are merging. The virtuosity and tendernes,swimwear,s of the presentation is very touching.It should be noted that “Mondo” Duo in the Milan Scala occurred.The admission fee is �?12, for members of the Ammertal Beker culture 9 us tickets are available o,Club Dresses,nly at the evening box office.Date: Saturday, 10.12.KNC 16 B Time: 20:00 UhrOrt: horse stable, at the Grange 1, ammersbek
Ruthild, Enno and günther Oetjen from gron wohld are originals. They live and,womens blouses, work “Unter den Linden” 8211; where the legendary “Dorfkrug” of the Butte warder is rotated.In