From: "Gopher-Project" <gopher-project-bounces+rachael=telefisk.org@lists.alioth.debian.org> Date: Thu Dec 25 11:20:28 2014 Subject: Re: [gopher] A little Gopher crawler --===============7516014296009082221== Content-Type: multipart/alternative; boundary=001a113ed7fcb01b5f050b07be31 --001a113ed7fcb01b5f050b07be31 Content-Type: text/plain; charset=UTF-8 Hi again, So I wrote that crawler and improved upon it (it's still single threaded) but here's what I have to share. prologic@daisy Thu Dec 25 20:06:53 ~/tmp $ time ./gspider.py &> index real 0m6.244s user 0m0.120s sys 0m0.047s prologic@daisy Thu Dec 25 20:07:16 ~/tmp $ wc -l index 3039 index This is run on localhost on the same machine against my cgod python gopher server (see other thread). The performance is *MUCH* better :) cheers James James Mills / prologic E: prologic@shortcircuit.net.au W: prologic.shortcircuit.net.au On Mon, Dec 15, 2014 at 3:23 PM, James Mills <prologic@shortcircuit.net.au> wrote: > > Hi All and Kim (author of Gophernicus!), > > I wrote this little cralwer today: > https://gist.github.com/b781e02b0299fef1f3f6 > > I'm a bit disappointed in performance though of crawling my local > Gopehrspace (basically via localhost): > > prologic@daisy > Mon Dec 15 15:13:08 > ~/tmp > $ time ./gspider.py &> index > > real 5m27.825s > user 0m6.126s > sys 0m5.825s > > prologic@daisy > Mon Dec 15 15:18:51 > ~/tmp > $ wc -l index > 355 index > > Any comments? :) > > cheers > James > > James Mills / prologic > > E: prologic@shortcircuit.net.au > W: prologic.shortcircuit.net.au > --001a113ed7fcb01b5f050b07be31 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Hi again,<div><br></div><div>So I wrote that crawler and i= mproved upon it</div><div>(it's still single threaded) but here's w= hat I have to share.</div><div><br></div><div><div>prologic@daisy</div><div= >Thu Dec 25 20:06:53=C2=A0</div><div>~/tmp</div><div>$ time ./gspider.py &a= mp;> index</div><div><br></div><div>real<span class=3D"" style=3D"white-= space:pre"> pace:pre"> ce:pre"> div>Thu Dec 25 20:07:16=C2=A0</div><div>~/tmp</div><div>$ wc -l index</div>= <div>3039 index</div></div><div><br></div><div>This is run on localhost on = the same machine against</div><div>my cgod python gopher server (see other = thread).</div><div><br></div><div>The performance is *MUCH* better :)</div>= <div><br></div><div>cheers</div><div>James</div></div><div class=3D"gmail_e= xtra"><br clear=3D"all"><div><div class=3D"gmail_signature"><span style=3D"= border-collapse:collapse;color:rgb(136,136,136);font-size:13px"><br><font f= ace=3D"arial, sans-serif">James Mills / prologic</font><br><br><font face= =3D"arial, sans-serif"></font><font face=3D"'courier new', monospac= e">E:=C2=A0<a href=3D"mailto:prologic@shortcircuit.net.au" style=3D"color:r= gb(0,0,204)" target=3D"_blank">prologic@shortcircuit.net.au</a></font></spa= n><div><span style=3D"font-family:'courier new',monospace;color:rgb= (136,136,136);font-size:13px">W:=C2=A0</span><a href=3D"http://prologic.sho= rtcircuit.net.au" style=3D"font-family:'courier new',monospace;font= -size:13px;color:rgb(0,0,204)" target=3D"_blank">prologic.shortcircuit.net.= au</a><br></div></div></div> <br><div class=3D"gmail_quote">On Mon, Dec 15, 2014 at 3:23 PM, James Mills= <span dir=3D"ltr"><<a href=3D"mailto:prologic@shortcircuit.net.au" targ= et=3D"_blank">prologic@shortcircuit.net.au</a>></span> wrote:<blockquote= class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc soli= d;padding-left:1ex"><div dir=3D"ltr">Hi All and Kim (author of Gophernicus!= ),<div><br></div><div>I wrote this little cralwer today:=C2=A0<a href=3D"ht= tps://gist.github.com/b781e02b0299fef1f3f6" target=3D"_blank">https://gist.= github.com/b781e02b0299fef1f3f6</a></div><div><br></div><div>I'm a bit = disappointed in performance though of crawling my local Gopehrspace (basica= lly via localhost):</div><div><br></div><div><div>prologic@daisy</div><div>= Mon Dec 15 15:13:08=C2=A0</div><div>~/tmp</div><div>$ time ./gspider.py &am= p;> index</div><div><br></div><div>real<span style=3D"white-space:pre-wr= ap"> /span>0m6.126s</div><div>sys<span style=3D"white-space:pre-wrap"> 5.825s</div><div><br></div><div>prologic@daisy</div><div>Mon Dec 15 15:18:5= 1=C2=A0</div><div>~/tmp</div><div>$ wc -l index=C2=A0</div><div>355 index</= div></div><div><br></div><div>Any comments? :)</div><div><br></div><div>che= ers</div><div>James</div><div><div><div><span style=3D"border-collapse:coll= apse;color:rgb(136,136,136);font-size:13px"><br><font face=3D"arial, sans-s= erif">James Mills / prologic</font><br><br><font face=3D"arial, sans-serif"= ></font><font face=3D"'courier new', monospace">E:=C2=A0<a href=3D"= mailto:prologic@shortcircuit.net.au" style=3D"color:rgb(0,0,204)" target=3D= "_blank">prologic@shortcircuit.net.au</a></font></span><div><span style=3D"= font-family:'courier new',monospace;color:rgb(136,136,136);font-siz= e:13px">W:=C2=A0</span><a href=3D"http://prologic.shortcircuit.net.au" styl= e=3D"font-family:'courier new',monospace;font-size:13px;color:rgb(0= ,0,204)" target=3D"_blank">prologic.shortcircuit.net.au</a><br></div></div>= </div> </div></div> </blockquote></div></div> --001a113ed7fcb01b5f050b07be31-- --===============7516014296009082221== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Gopher-Project mailing list Gopher-Project@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project --===============7516014296009082221==-- Thread start (DIR) [gopher] A little Gopher crawler (DIR) Followup: Re: [gopher] A little Gopher crawler