Screen Scraping Applications and Ideas - Pls comment

Discussion in 'IT & Internet' started by MidnightCoder, Dec 4, 2005.

Thread Status:
Not open for further replies.
  1. MidnightCoder

    MidnightCoder UKBF Newcomer Free Member

    12 0
    Hi there,

    I have designed a few scripts in PHP and Perl that scrape information off a website and post it to a local mysql database.

    I first started using this to scrape stock prices. I'd scrape the info off a stock exchange website every 20 mins and post it to the database. I then setup a couple of procedures that monitored the prices and could email me if the prices changed.

    I wanna use this stuff for other purposes though - do you have any bright ideas where this technology could be used? I just thought I'd start a little discussion here - so if you have any ideas feel free to comment.
     
    Posted: Dec 4, 2005 By: MidnightCoder Member since: Dec 4, 2005
    #1
  2. Jayne

    Jayne UKBF Newcomer Free Member

    2,921 8
    Hi,

    I haven't a clue what you are on about (i'm a none techy)

    But Welcome to the forum :D

    Jayne
     
    Posted: Dec 4, 2005 By: Jayne Member since: Aug 6, 2005
    #2
  3. cjd

    cjd UKBF Legend Full Member - Verified Business

    15,438 3,067
    Time dependent information......?

    Unsold holidays, hotel beds and flights spring to mind. If you could scrape Ryan Air for empty seats 4 hours before take off or Travel Lodge at 9pm then SMS people interested in last second deals.........

    "Forget Lastminute try Lastsecond" (c)

    (I want paying for this one!)
     
    Posted: Dec 4, 2005 By: cjd Member since: Nov 23, 2005
    #3
  4. MidnightCoder

    MidnightCoder UKBF Newcomer Free Member

    12 0
    Catchy slogan!

    And I like the idea.

    Slept on the idea myself and I've come up with a few ideas of using this stuff for competitive research purposes... i.e monitoring stock levels, prices and info from a competitor's website. Any thoughts?

    I think the last minute hotel booking market is saturated though - but it might be handy for personal use!

    flights are tricky to scrape - but I think there's a website out there that does it already .... will have a quick scout around.

    there's an idea - scrape flight arrival and departure times & provide txt updates? or rail times? or bus times .... hmmm.

    Have any of you seen this: Google SMS
    http://www.google.co.uk/sms/

    Very handy for finding driving directions.... and free to use (depending on your txt message tarriff)
     
    Posted: Dec 5, 2005 By: MidnightCoder Member since: Dec 4, 2005
    #4
  5. MidnightCoder

    MidnightCoder UKBF Newcomer Free Member

    12 0
    Posted: Dec 5, 2005 By: MidnightCoder Member since: Dec 4, 2005
    #5
  6. Jayne

    Jayne UKBF Newcomer Free Member

    2,921 8
    Hi,

    I don't think I like the idea of people nicking bits off my web site, not very nice business practice.

    Jayne :D
     
    Posted: Dec 5, 2005 By: Jayne Member since: Aug 6, 2005
    #6
  7. mattk

    mattk UKBF Newcomer Free Member

    1,692 700
    Surely most information is now available via RSS rather than the somewhat covert "scraping"!
     
    Posted: Dec 5, 2005 By: mattk Member since: Dec 5, 2005
    #7
  8. Srivvy

    Srivvy UKBF Newcomer Free Member

    93 4
    I agree with mattk

    Sites that are happy for their content to be used would either state this or syndicate via RSS.

    Otherwise, they would be losing site loyalty to your service wouldnt they?

    What are your views on the ethics / legality of this or am I missing an important point?
     
    Posted: Dec 20, 2005 By: Srivvy Member since: May 24, 2005
    #8
  9. confused

    confused UKBF Newcomer Free Member

    1,751 15
    I would very much like to "aqquire" prices off some of my suppliers websites, not for anything sinister, but theres nothing worse than being face to face with a customer, and checking online prices hoping they wont see the trade prices ! I would love an offline price list that I could modify once in a while.
    Not quite on the same scale as your own reasons, but the same idea.
     
    Posted: Dec 20, 2005 By: confused Member since: Dec 4, 2005
    #9
  10. WelshPixels

    WelshPixels UKBF Newcomer Free Member

    115 0
    I know of at least two major websites that don’t mind this practice. Amazon and e-bay both say its possible but won’t provide support for it.

    There is a big market with gamblers placing clever arrangements of bets so they can’t loose. For example take a football match, with one bookie you will get better odds on team "A" and in another you will get better odds in team "B". It becomes possible to place bets on both teams and come out ahead no matter the score.

    The bets can get a lot more complicated than this for horse races etc.

    So there must be a market to scrape odds of a variety of bookies sites

    Just a thought
     
    Posted: Dec 20, 2005 By: WelshPixels Member since: Nov 19, 2005
    #10
Thread Status:
Not open for further replies.