Aggregators

[Lucky8]

I'm intrigued by these businesses which aggregate information on other websites/databases and re-present them in a different way. In particular I'm thinking about information content (eg advice), and products (eg clothing).

Has anyone here created such a business? Is there any off the shelf aggregator software - or what is involved in creating one, platform-wise/technically?

And who pays who what generally in this relationship?

 

[Darren_Ssc]

Mostly using affiliate datafeeds. The retailer pays a commission if a sale occurs within a set timescale from 24 hours to 30 days, generally. Amazon, for example, have an api with real time date rather than a daily feed.

Not worth doing unless you can add significant value or already have an established audience since the product descriptions are duplicated on many different sites.

It used to be an easy way to make money since the process could be automated with feeds uploaded via a cron job every morning. Obviously, since this offers no real value, Google started ignoring a lot of these sites. There is also a lot of dodgy stuff going on with merchants using every trick in the book to avoid payment.

 

[Lucky8]

Not worth doing unless you can add significant value

Yep. If you thought you could add value - and I'm thinking more on the information content side here not the clothing/product side - not just by re-presenting it, but re-formatting + adding brand new content that increased the value/usefulness of the original content, what would you technically need to create, and with what?

 

[Darren_Ssc]

Yep. If you thought you could add value - and I'm thinking more on the information content side here not the clothing/product side - not just by re-presenting it, but re-formatting + adding brand new content that increased the value/usefulness of the original content, what would you technically need to create, and with what?

You would need to pull the product descriptions and re-write them by hand. Easy enough to do in theory but a very tedious job and not necessarily guaranteed to work.

The biggest problem, aside from adding something unique, is also managing the feeds themselves and filtering out all the guff. You don't really want to be writing descriptions for every single product since it would never be worth the effort.

 

[Lucky8]

I'm thinking that to start with it could be scruffy and manual in the background, then semi/automated as it grows and develops.

Ideally they'd be some sort of business relationship with the content provider where they submit the agreed content (so don't need to filter out guff as 100% of what they provide/allow access to is to be re-displayed) in a way that could be re-formatted and added to (so might need to filter out guff at this level, fields that aren't needed, headings and paragraphs that aren't helpful), or their "feeds" allow the aggregator to scrape the content and take what they need.

What tech is needed to do all this?

 

[Darren_Ssc]

I'm thinking that to start with it could be scruffy and manual in the background, then semi/automated as it grows and develops.

Ideally they'd be some sort of business relationship with the content provider where they submit the agreed content (so don't need to filter out guff as 100% of what they provide/allow access to is to be re-displayed) in a way that could be re-formatted and added to (so might need to filter out guff at this level, fields that aren't needed, headings and paragraphs that aren't helpful), or their "feeds" allow the aggregator to scrape the content and take what they need.

What tech is needed to do all this?

You don't need to worry about the tech. You need to worry how to generate enough millions in sales in order to get the retailer to bother talking to you.

Your idea for the retailer to do all the work for you is a great one. Now ask what they need you for?

 

[Lucky8]

I'm asking more on the content side rather than retailers, I have an idea about a proposition which I'm researching, so I am worrying about that don't worry.

But I'm also 'worrying' about the tech - what basically is needed and what could I get away with starting out?

 

[NickGrogan]

Wordpress, a plugin to pull the rss feeds and convert them to posts, and another plugin to share the posts on Twitter or wherever.

You could put another plugin in the middle to "rewrite" the posts if you wanted to.

 

[mattk]

There are three ways to do this. As @Darren_Ssc says, the most basic way is by hand.

The second way, is via screen scraping. These are tools which look for tags or series of text in a file and strip out the relevant sections, headings, sub-headings, main bodies etc. In days of yore, screen scraping was done in JavaScript, but there are now to let you point and click scraping.

Finally is the most sophisticated, which is via data transfer. This will likely be XML or CSV where you provide a schema (details of what needs to be sent) and the source site produces the file. As @NickGrogan says, RSS is an example of this, as is something like Google Shopping.

Once you have the data it is up to you how you transform and display it. Typically, you would transform it using your backend code (I'd use c#) and then store it in a database. You then build a page template which outputs the data in a consistent format, no matter what the source.

 

[Lucky8]

Thanks everyone, that gives me a good overview to start off.