From: "Gopher-Project" <gopher-project-bounces+rachael=telefisk.org@lists.alioth.debian.org>
Date: Thu Dec 25 11:20:28 2014
Subject: Re: [gopher] A little Gopher crawler
--===============7516014296009082221==
Content-Type: multipart/alternative; boundary=001a113ed7fcb01b5f050b07be31
--001a113ed7fcb01b5f050b07be31
Content-Type: text/plain; charset=UTF-8
Hi again,
So I wrote that crawler and improved upon it
(it's still single threaded) but here's what I have to share.
prologic@daisy
Thu Dec 25 20:06:53
~/tmp
$ time ./gspider.py &> index
real 0m6.244s
user 0m0.120s
sys 0m0.047s
prologic@daisy
Thu Dec 25 20:07:16
~/tmp
$ wc -l index
3039 index
This is run on localhost on the same machine against
my cgod python gopher server (see other thread).
The performance is *MUCH* better :)
cheers
James
James Mills / prologic
E: prologic@shortcircuit.net.au
W: prologic.shortcircuit.net.au
On Mon, Dec 15, 2014 at 3:23 PM, James Mills <prologic@shortcircuit.net.au>
wrote:
>
> Hi All and Kim (author of Gophernicus!),
>
> I wrote this little cralwer today:
> https://gist.github.com/b781e02b0299fef1f3f6
>
> I'm a bit disappointed in performance though of crawling my local
> Gopehrspace (basically via localhost):
>
> prologic@daisy
> Mon Dec 15 15:13:08
> ~/tmp
> $ time ./gspider.py &> index
>
> real 5m27.825s
> user 0m6.126s
> sys 0m5.825s
>
> prologic@daisy
> Mon Dec 15 15:18:51
> ~/tmp
> $ wc -l index
> 355 index
>
> Any comments? :)
>
> cheers
> James
>
> James Mills / prologic
>
> E: prologic@shortcircuit.net.au
> W: prologic.shortcircuit.net.au
>
--001a113ed7fcb01b5f050b07be31
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Hi again,<div><br></div><div>So I wrote that crawler and i=
mproved upon it</div><div>(it's still single threaded) but here's w=
hat I have to share.</div><div><br></div><div><div>prologic@daisy</div><div=
>Thu Dec 25 20:06:53=C2=A0</div><div>~/tmp</div><div>$ time ./gspider.py &a=
mp;> index</div><div><br></div><div>real<span class=3D"" style=3D"white-=
space:pre">
pace:pre">
ce:pre">
div>Thu Dec 25 20:07:16=C2=A0</div><div>~/tmp</div><div>$ wc -l index</div>=
<div>3039 index</div></div><div><br></div><div>This is run on localhost on =
the same machine against</div><div>my cgod python gopher server (see other =
thread).</div><div><br></div><div>The performance is *MUCH* better :)</div>=
<div><br></div><div>cheers</div><div>James</div></div><div class=3D"gmail_e=
xtra"><br clear=3D"all"><div><div class=3D"gmail_signature"><span style=3D"=
border-collapse:collapse;color:rgb(136,136,136);font-size:13px"><br><font f=
ace=3D"arial, sans-serif">James Mills / prologic</font><br><br><font face=
=3D"arial, sans-serif"></font><font face=3D"'courier new', monospac=
e">E:=C2=A0<a href=3D"mailto:prologic@shortcircuit.net.au" style=3D"color:r=
gb(0,0,204)" target=3D"_blank">prologic@shortcircuit.net.au</a></font></spa=
n><div><span style=3D"font-family:'courier new',monospace;color:rgb=
(136,136,136);font-size:13px">W:=C2=A0</span><a href=3D"http://prologic.sho=
rtcircuit.net.au" style=3D"font-family:'courier new',monospace;font=
-size:13px;color:rgb(0,0,204)" target=3D"_blank">prologic.shortcircuit.net.=
au</a><br></div></div></div>
<br><div class=3D"gmail_quote">On Mon, Dec 15, 2014 at 3:23 PM, James Mills=
<span dir=3D"ltr"><<a href=3D"mailto:prologic@shortcircuit.net.au" targ=
et=3D"_blank">prologic@shortcircuit.net.au</a>></span> wrote:<blockquote=
class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc soli=
d;padding-left:1ex"><div dir=3D"ltr">Hi All and Kim (author of Gophernicus!=
),<div><br></div><div>I wrote this little cralwer today:=C2=A0<a href=3D"ht=
tps://gist.github.com/b781e02b0299fef1f3f6" target=3D"_blank">https://gist.=
github.com/b781e02b0299fef1f3f6</a></div><div><br></div><div>I'm a bit =
disappointed in performance though of crawling my local Gopehrspace (basica=
lly via localhost):</div><div><br></div><div><div>prologic@daisy</div><div>=
Mon Dec 15 15:13:08=C2=A0</div><div>~/tmp</div><div>$ time ./gspider.py &am=
p;> index</div><div><br></div><div>real<span style=3D"white-space:pre-wr=
ap">
/span>0m6.126s</div><div>sys<span style=3D"white-space:pre-wrap">
5.825s</div><div><br></div><div>prologic@daisy</div><div>Mon Dec 15 15:18:5=
1=C2=A0</div><div>~/tmp</div><div>$ wc -l index=C2=A0</div><div>355 index</=
div></div><div><br></div><div>Any comments? :)</div><div><br></div><div>che=
ers</div><div>James</div><div><div><div><span style=3D"border-collapse:coll=
apse;color:rgb(136,136,136);font-size:13px"><br><font face=3D"arial, sans-s=
erif">James Mills / prologic</font><br><br><font face=3D"arial, sans-serif"=
></font><font face=3D"'courier new', monospace">E:=C2=A0<a href=3D"=
mailto:prologic@shortcircuit.net.au" style=3D"color:rgb(0,0,204)" target=3D=
"_blank">prologic@shortcircuit.net.au</a></font></span><div><span style=3D"=
font-family:'courier new',monospace;color:rgb(136,136,136);font-siz=
e:13px">W:=C2=A0</span><a href=3D"http://prologic.shortcircuit.net.au" styl=
e=3D"font-family:'courier new',monospace;font-size:13px;color:rgb(0=
,0,204)" target=3D"_blank">prologic.shortcircuit.net.au</a><br></div></div>=
</div>
</div></div>
</blockquote></div></div>
--001a113ed7fcb01b5f050b07be31--
--===============7516014296009082221==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Gopher-Project mailing list
Gopher-Project@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project
--===============7516014296009082221==--
Thread start
(DIR) [gopher] A little Gopher crawler
(DIR) Followup: Re: [gopher] A little Gopher crawler