[Klug-general] Bash Scripts / Curl Meetup with Stephen

[Nathan Friend]

Hi Stephen,
I've spit up your question, maybe some of the others Klug members can
add their thoughts...

1.1  I want to start by writing a script that spiders my website,
pulls down all the pages i have there
If this is for backup you could use Rrsync to copy the entire site to
another location.

1.2 pulls down all the new comments posted by site visitors.
How are the comments stored?  My site uses wordpress so the comments
are in the database, easy to query the database to get your comments.

1.3 scrapes all my site stats for the day.
Unless your really attached to your cpanel stats I recommend webalizer
for web site stats.  Which works OK in a terminal browser like w3m,
see attached screen grab.


Cheers,

Nathan.



from	Stephen Ryan <intrench at googlemail.com>
reply-to	Kent Linux User Group - General Topics <kent at mailman.lug.org.uk>
to	kent at mailman.lug.org.uk
date	Fri, Apr 15, 2011 at 3:32 PM
subject	[Klug-general] Bash / Curl Scripting Tuition 5
mailing list	kent.mailman.lug.org.uk Filter messages from this mailing list
mailed-by	mailman.lug.org.uk


Ok. here is some input in response to questions.

1. I want to start by writing a script that spiders my website, pulls
down all the pages i have there, pulls down all the new comments
posted by site visitors and then logs into my cpanel and scrapes all
my site stats for the day. All this content will be downloaded and
formatted into text files for my viewing. This way i don't even need
to load the browser up. Then from there i start working out how to
interact with my website in other ways.

2. My plan is to use Amazon EC2 free services to pull up a command
line, from where i can run my scripts.
http://aws.amazon.com/

3. I do not want to use any higher level language which i have to
upload to EC2. Native Linux and Bash would be my preferred choice and
of course i have to use Curl.

I hope that clarifies my requirement there.

cheers
Stephen

_______________________________________________
Kent mailing list
Kent at mailman.lug.org.uk
https://mailman.lug.org.uk/mailman/listinfo/kent

On Fri, Apr 15, 2011 at 3:36 PM, Julia Freeman <klug at quixotic.org.uk> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Thu, Apr 14, 2011 at 06:43:34PM +0100, Stephen Ryan wrote:
>>
>> I guess i just need to see it working because i have only seen it on youtube
>> videos and there are not that many videos about on Bash/Curl. My goal is to
>> write scipts to manage my website and it will help me learn more about the
>> web.
>>
>
> Would a linux shell account do the trick?
>
> J
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.4 (GNU/Linux)
>
> iD8DBQFNqFf/pQ4p2TvRQNARAqOgAJ4n8lJW4VJj4TZE7K/9+MHqLdLkhQCfeiPY
> P88RSPyxBxTjgmCJL65jwQA=
> =RG5e
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Kent mailing list
> Kent at mailman.lug.org.uk
> https://mailman.lug.org.uk/mailman/listinfo/kent
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: webalizer.tiff
Type: image/tiff
Size: 877520 bytes
Desc: not available
URL: <http://mailman.lug.org.uk/pipermail/kent/attachments/20110415/62bb5be7/attachment-0001.tiff>