Web Authoring Tools - webpage crawler for database auto-update

Author

webpage crawler for database auto-update

lluum@yahoo.com

2005-02-16, 11:19 pm

Hi, I need to visit a webpage daily to get some information and input
the information in my MS Access database. I am thinking of writing
something that can automate the task

1. visit the webpage (using my login and password)
2. download the webpage
3. parse the webpage
4. extract necessary information from the webpage
5. update my Access database.

I had a little experience in programming. I am thinking to write it by
myself. But I do not know where to start (e.g., what language to use).
Could you recommend a starter's webpage or a book. Please note that I
need to update Access database. I also have a future plan to migrate
to a MS SQL database.

I guess this product may already in the market. Any product
recommendation is welcome,too.

Thank you very much.

lluum@yahoo.com

2005-02-17, 6:52 pm

Maybe webpage crwaler is not the correct term. The term probably
should be web grabber. Anyway, its intended use is described above.
Thank you very much.

Peter Greenwood

2005-02-21, 6:51 pm

In article <1108606379.660537.305390@z14g2000cwz.googlegroups.com>,
"lluum@yahoo.com" <lluum@yahoo.com> writes:
> Hi, I need to visit a webpage daily to get some information and input
> the information in my MS Access database.
...
> 1. visit the webpage (using my login and password)
> 2. download the webpage
> 3. parse the webpage
> 4. extract necessary information from the webpage
> 5. update my Access database.

It isn't a web authoring tool you want, but ...

I would consider using perl; look at http://www.cpan.org/ports/ to get
started. You probably want the libwww-perl, DBI and DBD::ODBC modules; find
these at www.cpan.org if they don't come with the PERL distribution you get.

HTH

--
Peter Greenwood peterg@pgid.co.uk
http://pgid.demon.co.uk