Webmaster forum - Getting Data; Opinions Wanted

Author

Getting Data; Opinions Wanted

BenignVanilla

2006-09-24, 7:00 pm

In another thread, I am getting advice for how I can format and produce some
data for a third party web site to consume. The part I didn't explain is
that right now the incoming data is dumped in a CSV file, which I import
every 25 minutes or so, so I can always have the most up to date info.

I do this because I don't want to distribute an application that has my DB
login credentials within it, and I do not want to share my DB credentials
with the group that is publishing the data. It occured to me today, that
there may be another solution.

I am thinking of getting the publisher to add some code to their application
to do an HTML Post of the data to a page on my site. They produce a new row
every minute, so they could simply POST to a custom page, and I could update
the database that way.

I realize someone could hijack this link and put garbage in our database,
but are there any other concerns? Or is there a better way to solve this
issue?

BV.

Jerry Stuckle

2006-09-24, 7:00 pm

BenignVanilla wrote:
> In another thread, I am getting advice for how I can format and produce some
> data for a third party web site to consume. The part I didn't explain is
> that right now the incoming data is dumped in a CSV file, which I import
> every 25 minutes or so, so I can always have the most up to date info.
>
> I do this because I don't want to distribute an application that has my DB
> login credentials within it, and I do not want to share my DB credentials
> with the group that is publishing the data. It occured to me today, that
> there may be another solution.
>
> I am thinking of getting the publisher to add some code to their application
> to do an HTML Post of the data to a page on my site. They produce a new row
> every minute, so they could simply POST to a custom page, and I could update
> the database that way.
>
> I realize someone could hijack this link and put garbage in our database,
> but are there any other concerns? Or is there a better way to solve this
> issue?
>
> BV.
>
>

Why not use an RSS feed? Then you can pull it from them at your leisure.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

Andy Dingley

2006-09-24, 7:00 pm

BenignVanilla wrote:

> I am thinking of getting the publisher to add some code to their application
> to do an HTML Post of the data to a page on my site.

That's a "push" model - always painful.

Assuming that the supplier of data has a web server that's reliably
"up" (this could be simple rented static HTML space) and also that
they're remotely competent at running simple scripts when nneeded, then
you'd be better off using a "pull" model. The difference is which side
of the interface controls the asynchronous polling times.

Look at RSS, Atom and especially the RSS 1.0 Syndication module
http://web.resource.org/rss/1.0/modules/syndication/
also
http://www.intertwingly.net/wiki/pie/
It's a good format to use, and there's also some useful information to
gain from seeing how the "how often to download it" information can be
handled. Atom's publishing protocol is very good, but heavyweight, RSS
2.0's is crude and fairly useless, RSS 1.0 Syndication is just about
right.
You can use these same syndication ideas for publishing in CSV as in
RSS or XML.

TTo set up "pull", then have all the "pushing" taking place within the
producer's own machinery. Have them query the hardware as needed,
making the nesw data set and then uploading it to the public server.
Set the creation timestamps on this carefully: precisely and accurately
according to standard network time. Make sure the updateFrequency
property is set accurately too, although this is usually a one-off
setting and can be set manually (about half the actual update time is
good).

On your server, retrieve the content and update your pages with it.
Revisit for a new copy when the updateFrequency tells you, or as soon
after that as you can manage. You shouldn't be quicker than that.

If you want (and this gets pointlessly complex for a simple aggregator
of one feed at one update frequency), then modify your own aggregatpr's
revisiting rate based on the claimed updateFrequency and the time you
actually note changes happening. Most manual blogging sites will tell
you not to revisit more than once an hour, but they might not actually
post more than once a day. So back off your frequency until you're
running at half-day intervals (haf the actual change time).

Ignore the "skipHours" markup - that was a bad idea.

BenignVanilla

2006-09-24, 7:00 pm

"Jerry Stuckle" <jstucklex@attglobal.net> wrote in message
news:-tKdnTLO6578po7YnZ2dnUVZ_sGdnZ2d@comcast.com...
<snip>
>
> Why not use an RSS feed? Then you can pull it from them at your leisure.

I pull the data from them now at my leisure. I am concerned we may have a
client in the future that does not have the capabilities to post the files
as this client is, and then we'll need a new solution. So I was kind of
thinking out loud.

BV.

Jerry Stuckle

2006-09-24, 7:00 pm

BenignVanilla wrote:
> "Jerry Stuckle" <jstucklex@attglobal.net> wrote in message
> news:-tKdnTLO6578po7YnZ2dnUVZ_sGdnZ2d@comcast.com...
> <snip>
>
>
>
> I pull the data from them now at my leisure. I am concerned we may have a
> client in the future that does not have the capabilities to post the files
> as this client is, and then we'll need a new solution. So I was kind of
> thinking out loud.
>
> BV.
>
>

Yep, that's why I suggested an RSS feed. It works quite well for
machine-to-machine communications and can be available any time you want it.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================