Screen Scraping Applications and Ideas - Pls comment

MidnightCoder · Dec 4, 2005

Hi there,

I have designed a few scripts in PHP and Perl that scrape information off a website and post it to a local mysql database.

I first started using this to scrape stock prices. I'd scrape the info off a stock exchange website every 20 mins and post it to the database. I then setup a couple of procedures that monitored the prices and could email me if the prices changed.

I wanna use this stuff for other purposes though - do you have any bright ideas where this technology could be used? I just thought I'd start a little discussion here - so if you have any ideas feel free to comment.

Jayne · Dec 4, 2005

Hi,

I haven't a clue what you are on about (i'm a none techy)

But Welcome to the forum

Jayne

cjd · Dec 4, 2005

Time dependent information......?

Unsold holidays, hotel beds and flights spring to mind. If you could scrape Ryan Air for empty seats 4 hours before take off or Travel Lodge at 9pm then SMS people interested in last second deals.........

"Forget Lastminute try Lastsecond" (c)

(I want paying for this one!)

MidnightCoder · Dec 5, 2005

Catchy slogan!

And I like the idea.

Slept on the idea myself and I've come up with a few ideas of using this stuff for competitive research purposes... i.e monitoring stock levels, prices and info from a competitor's website. Any thoughts?

I think the last minute hotel booking market is saturated though - but it might be handy for personal use!

flights are tricky to scrape - but I think there's a website out there that does it already .... will have a quick scout around.

there's an idea - scrape flight arrival and departure times & provide txt updates? or rail times? or bus times .... hmmm.

Have any of you seen this: Google SMS
http://www.google.co.uk/sms/

Very handy for finding driving directions.... and free to use (depending on your txt message tarriff)

MidnightCoder · Dec 5, 2005

Uh-oh.... this stuff's been outlawed!

"Court forbids 'scraping' of American Airlines web site"

http://www.out-law.com/page-3398

Jayne · Dec 5, 2005

Hi,

I don't think I like the idea of people nicking bits off my web site, not very nice business practice.

Jayne

mattk · Dec 5, 2005

Surely most information is now available via RSS rather than the somewhat covert "scraping"!

Srivvy · Dec 20, 2005

I agree with mattk

Sites that are happy for their content to be used would either state this or syndicate via RSS.

Otherwise, they would be losing site loyalty to your service wouldnt they?

What are your views on the ethics / legality of this or am I missing an important point?

confused · Dec 20, 2005

I would very much like to "aqquire" prices off some of my suppliers websites, not for anything sinister, but theres nothing worse than being face to face with a customer, and checking online prices hoping they wont see the trade prices ! I would love an offline price list that I could modify once in a while.
Not quite on the same scale as your own reasons, but the same idea.

WelshPixels · Dec 20, 2005

I know of at least two major websites that dont mind this practice. Amazon and e-bay both say its possible but wont provide support for it.

There is a big market with gamblers placing clever arrangements of bets so they cant loose. For example take a football match, with one bookie you will get better odds on team "A" and in another you will get better odds in team "B". It becomes possible to place bets on both teams and come out ahead no matter the score.

The bets can get a lot more complicated than this for horse races etc.

So there must be a market to scrape odds of a variety of bookies sites

Just a thought

Search

Search

Screen Scraping Applications and Ideas - Pls comment

MidnightCoder

Jayne

cjd

MidnightCoder

MidnightCoder

Jayne

mattk

Srivvy

confused

WelshPixels

Similar threads

Latest Articles