Server specification for music upload site

[francisgane]

Hi all,

I'm in the early stages of developing a website that is centred around users being able to upload their music to our site for review.

Since its inception I have been worrying about the server costs as I envisage each user may upload between 1-7 tracks a month (each track around 7MB-ish) so upto around 50MB per user per month. When you start to multiply the number of users up you start needing 50GB per month for 1000 users and so on.

I don't know a lot about server technologies/specs/how to figure out my requirements so am hoping a server-savvy member can help!

Thanks,

Francis

 

[KarlAustin]

Hi,

First things first, you need to be realistic. We tend to find that most people over-estimate how popular their new venture will be by at least a factor of 10, if not 100. It's far better to start off smaller, keep your cash flow free and grow later rather than to spend a boatload of money on a setup you find you're only using 1% of for the first 12 months. Lack of cash-flow is what kills most businesses.

More important than the hardware is actually your software - you need to design it from the off to be easily scalable in the future, that way your initial hardware choice isn't going to matter so much, be it a small dedicated server or a VPS.

Downloads of static files don't really tax servers at all, it's the dynamic stuff that puts the load on the server. Although if you've got 10,000 users trying to download a track at the same time then that is going to place some stress on your disks - but you're unlikely to find you have that situation for some time, and even then it's likely to only be a handful of popular tracks getting downloaded - so it's a problem easily solved with some intelligent caching.

If you want to chat about it, feel free to message me, it's something we tend to specialise in, helping customers scale out their platform without spending a small fortune in the process.

Thanks,

 

[francisgane]

Hi Karl,

Thanks for the considered response. If you noticed though, this site will be an upload site not a download site. In fact, there probably won't be any downloads available from the site at all.

The question I have been worrying over is the amount of disk space that could get used up if it gets popular. Also, I'm not sure of the implications for the upload bandwidth of the site as I don't want it to take people forever to upload a 3 minute track as it could kill the site if people get fed up of long upload times.

What do you think the spec behind a site like dnbshare dot com is as their whole purpose is to accept music uploads?

May well be in touch about your services when we come to development this summer Karl.

Thanks,

Francis

 

[KarlAustin]

Hi,

Apologies, I was making an assumption on the downloads.

Generally when it comes to hosting, the upload/download speeds are the same, it's all symmetrical - although for the person doing the uploading from DSL/Cable etc. that is most likely not the case.

In terms of storage, if it is going to be reviewed but then rarely looked at again, how long do you need keep each track for? If it's pretty much write once, read once then it opens up your storage options some more - although if you were expecting many simultaneous uploads then you'd still want to look at some form of intelligent caching so that the writes happen to very fast storage before going off to the backing store for more permanent storage. That though is something you can look at in the future - it's far easier to say, "We've grown a lot quicker than expected, we might be a little slow for a couple of weeks" than to say, "Sorry, we've spent too much in the beginning and now can't keep going" (I'll never get an award for salesman of the year!)

I'd not want to hazard what their spec is, as I don't know anywhere near enough about them to do that, so it'd be wild speculation and of no real use to you. Disk I/O will most likely be your main bottle-neck, but then that's true for most of our customers these days, but fortunately it's not as expensive a problem to solve as it once was.

Cheers,

 

[astutiumRob]

I'm in the early stages of developing a website that is centred around users being able to upload their music to our site for review.

Will you be retaining the files after the review ?
Are the reviews by other users or you ?

With "upload" generally being 1/50th of the "download" capacity for home users, it's going to a few minutes take for a 3 min track to transfer - that's going to be down to their connection not the server connection, so unless you're expecting hundreds/thousands of simultaneous uploads, it's a non-issue at this stage

What are you best-guesses for the number of users ?

 

[Dominic Taylor]

As long as you plan your app, you can scale easily - separate uploads by letter to store in different locations as and when needed, etc.

Unless you don't mind initial high costs for a dedicated system with a few 2Tb drives, you'll be fine with a virtual server or similar which you can resize as needed.

Below that you could always start with a simple shared hosting account too.

Of course if you don't need to keep data on the site for long, then almost anything will do capacity-wise.

 

[Jolt.co.uk]

For something like this where scalable storage is more important than customer service or reliability, have you considered Amazon S3?

 

[Dan_HiHosting]

Depending on your needs, this sounds like a new venture so I'd recommend you actually start with shared hosting.

If you're still developing your website, this will let you test the site live at a low cost, and will also be fine for uploads until the time comes that it really takes off, if that happens.

Seeing as the uploads aren't going to be downloaded by other users, unless you plan on sharing them with your team over a long period of time, you could simply download them to your computer after a period to free up disk space. That way you'd be fine with a normal amount of storage.

I'd actually recommend against focusing on the technology and big specs to start off with. What's more important in my opinion is that you work with someone who's going to offer the support and advice you'll need when developing your site and business, and that'll help you easily move up to higher spec services when the demand for your site grows and you need to.

I always think it's important to avoid unnecessary expense when developing a new site, as a lot will depend on your users and the direction you take as it grows and develops.

Good luck with it, and do let us know your site if you like, sounds interesting

 

[francisgane]

Will you be retaining the files after the review ?
Are the reviews by other users or you ?

With "upload" generally being 1/50th of the "download" capacity for home users, it's going to a few minutes take for a 3 min track to transfer - that's going to be down to their connection not the server connection, so unless you're expecting hundreds/thousands of simultaneous uploads, it's a non-issue at this stage

What are you best-guesses for the number of users ?

Probably will retain the files after review so that they can be listened to any time on the artists profile page.

Reviews will be submitted by site users/visitors.

Users wise, I am unsure to say the least as to expected numbers. I'd like to think that we'd financially be breaking even at around 100 paying users (and in the region of 5000-ish free users) but it will need to be scalable to accommodate thousands if it works out as planned.

 

[KarlAustin]

Hi,

I'm a little confused, you say there won't be any downloads but users will be reviewing the tracks - that's going to require them to listen to the track surely? Which involves a download - even if you're playing it inside an HTML media object or Flash player etc. it's still effectively a download

I'd also not underestimate the number of people who may listen multiple times if they like what they've heard.

Thanks,

 

[francisgane]

For something like this where scalable storage is more important than customer service or reliability, have you considered Amazon S3?

Not heard of S3. Tell me more. Are you saying that customer service or reliability is bad with S3?

 

[francisgane]

Hi,

I'm a little confused, you say there won't be any downloads but users will be reviewing the tracks - that's going to require them to listen to the track surely? Which involves a download - even if you're playing it inside an HTML media object or Flash player etc. it's still effectively a download

I'd also not underestimate the number of people who may listen multiple times if they like what they've heard.

Thanks,

That's correct. I didn't think of it in terms of raw data, and yes if they're streaming a whole track in order to review then it's effectively the same as downloading it isn't it?

Would this make a big difference to the spec of the hardware/software I'd need?

 

[francisgane]

Depending on your needs, this sounds like a new venture so I'd recommend you actually start with shared hosting.

If you're still developing your website, this will let you test the site live at a low cost, and will also be fine for uploads until the time comes that it really takes off, if that happens.

Seeing as the uploads aren't going to be downloaded by other users, unless you plan on sharing them with your team over a long period of time, you could simply download them to your computer after a period to free up disk space. That way you'd be fine with a normal amount of storage.

I'd actually recommend against focusing on the technology and big specs to start off with. What's more important in my opinion is that you work with someone who's going to offer the support and advice you'll need when developing your site and business, and that'll help you easily move up to higher spec services when the demand for your site grows and you need to.

I always think it's important to avoid unnecessary expense when developing a new site, as a lot will depend on your users and the direction you take as it grows and develops.

Good luck with it, and do let us know your site if you like, sounds interesting

By shared hosting do you mean your bog standard cheap hosting? This would be ideal if you think I could get away with it to start with.

 

[KarlAustin]

That's correct. I didn't think of it in terms of raw data, and yes if they're streaming a whole track in order to review then it's effectively the same as downloading it isn't it?

Would this make a big difference to the spec of the hardware/software I'd need?

All comes back to how many tracks you think will be listened to each day - obviously if you have 240 tracks listened to in a day that is very unlikely to be 10 every hour, there will be peaks in it depending on your user base. For general web traffic they would be between 8-9AM (when people get to work), steadily rising, another peak at lunch time, slowing a little, with another peak at 4-6PM (when people have had enough of working or just got home) - Although in your case I'd expect to maybe see your peak time around 8-11PM.

 

[Dan_HiHosting]

By shared hosting do you mean your bog standard cheap hosting? This would be ideal if you think I could get away with it to start with.

Yes exactly. I'd start with that and then upgrade as you need to. Any decent host will be able to provision you a VPS and move your site to it as soon as you need it.