Solutions to allow large file transfer across the net

[NetwiseHosting]

Hi All,

Cutting a long story short, we have been passed a client that is pulling their hair out about their current setup where they need to regularly send a ~8GB file from one site to another.

They have a remote worker creating a video file that then needs inspecting and then later burning to DVD at a later date. Currently they are using dropbox but due to having relatively poor broadband speeds and the size of the file they have quoted that it can take 12 hours or more to send the file.

I don't believe the services we offer are going to help unfortunately but I was wondering if anyone had any suggestions as I would like to at least pass on some advice if possible?

I am going to suggest the option of the remote worker using a remote desktop session to a PC at the end destination/burning point but apart from that I don't see too many other options and as the upload speed is the killer. Any ideas guys?

Thanks in advance and hope you are all well,

Matthew

 

[OldWelshGuy]

As you say, a chain is only as strong as its weakest link, and that link is the connection speed.

 

[cmcp]

Yip, you're stuck with their upload speed. Wireless dongle or new / additional broadband point probably required, imo.

 

[Posilan]

Hi All,

Cutting a long story short, we have been passed a client that is pulling their hair out about their current setup where they need to regularly send a ~8GB file from one site to another.

They have a remote worker creating a video file that then needs inspecting and then later burning to DVD at a later date. Currently they are using dropbox but due to having relatively poor broadband speeds and the size of the file they have quoted that it can take 12 hours or more to send the file.

I don't believe the services we offer are going to help unfortunately but I was wondering if anyone had any suggestions as I would like to at least pass on some advice if possible?

I am going to suggest the option of the remote worker using a remote desktop session to a PC at the end destination/burning point but apart from that I don't see too many other options and as the upload speed is the killer. Any ideas guys?

Thanks in advance and hope you are all well,

Matthew

If it has to go via the Internet, and the connections cannot be increased, the only idea I can think of is to re-encode the video file with a different codec to compress it down smaller?

Steve

 

[NetwiseHosting]

Cheers for your quick responses,

Very true - an additional broadband connection, especially one that could take advantage of Annex M for that little more upload may be an option. This coupled with a multiple WAN router/load balancer could be a good solution, at least it should cut a decent chunk off of their upload time.

Matthew

 

[NetwiseHosting]

I did suggest couriering the data encrypted on USB of similar but they didn't seem to enthused by this idea! I will raise this again, especially as there are so many ways of protecting the data during during transit.

Regarding encoding the video differently, great suggestion I will make sure this is raised as an option too.

Much appreciated guys.

 

[internetspaceships]

"Matt is it real time streaming?"

Ignore that comment it's daft. What's the time delay between the file being made and then being utilised at the other end?

 

[smo]

Your best bet is to suggest they use something like Spideroak backup.

The very efficient compression and encryption makes files much smaller and completely secure, if you were to configure a sync between folders at location A and B then it would make life easy and probably quicker too (due to compression)

On top of this they would also have a reliable secure backup of the files too

 

[NetwiseHosting]

Hi Guys,

Jon: good question, I don't believe it is all that quick. 24 hours at least, but I will confirm this tomorrow - it sounds like you have a suggestion

Smo: will have a look into this. The option of encoding the video appropriately in addition to high compression may be the answer.

I will find out more and revert with my findings - thanks again for your advice.

Matthew

 

[FreelanceSoftwareDeveloper]

It's a tough one.

8GB must be in a raw format, if it's going onto a DVD then it possibly can't be compressed due to the loss of quality, you wouldn't record an album in a studio then send mp3's to a CD producer.

You say the source is a remote worker so possibly only has a single internet connection. If multiple (separate) connections were available you could use a torrent.

Does the client have to download the file after the upload is complete taking a lot more time, if so then a peer-to-peer transfer would save some of that time.

 

[JDX_John]

I doubt very much it's a raw format. Maybe losslessly compressed but even DVDs use lossy compression. You can compress smaller than DVD size at DVD quality but since a DVD is normally 5Gb (I think) sending an 8Gb file seems overkill even if you're sending the DVD image itself.

Can you tell us more about duration and content... is it to be played on regular DVD players or on more modern systems only?

 

[decworld]

You cant speed it up, but you send it in a more reliable manner.

Create a torrent of it, private of course, and use this method to download.

 

[Posilan]

Create a torrent of it, private of course, and use this method to download.

How will that be faster? :|

 

[decworld]

Because transferring one 8Gb file has a high probability of failing during transfer. A torrent will be multiple smaller chunks.

Thats why I said reliability, not speed.

 

[Posilan]

Because transferring one 8Gb file has a high probability of failing during transfer. A torrent will be multiple smaller chunks.

Thats why I said reliability, not speed.

Yes but as speed is of the essence here, using bittorrent to transfer would just exasperate the situation. Torrents have high overheads so would just slow the transfer down further.

FTP with resume enabled would be far better if reliability of the connection was the issue.

Steve

 

[thewebguy2012]

transfer via SSH it is much faster than ftp and more reliable

 

[Posilan]

transfer via SSH it is much faster than ftp and more reliable

Huh? SSH (SCP) is around half as fast as FTP as it has to transmit roughly double the packets :|

If looking for network performance, NFS is probably the best but it's academic anyway as the bottleneck here is the ADSL upload speed...

 

[stugster]

transfer via SSH it is much faster than ftp and more reliable

I came here to laugh at this.

 

[lynxus]

Dont under estimate the bandwidth of a lorry load of DVD's.. Latency can vary from same day to a few but the bandwidth is huge!

 

[Mortime Business Software]

Books can be split into chapters; software applications into files, classes, etc.; musical compositions into movements; movies into scenes; and so on. Is there any way that your video could similarly be split into parts? If so, then each part could be sent separately.

All the best,

 

[leemason]

Have a look a Replify: http://www.replify.com/. It provided application optimisations. I don't know what size of organisation they are and it might be a bit over overkill but it should speed up such copies if you were to run this on both sites. It has to run in a VMware virtual machine so you could run VMware server on some existing hardware and host it on that.

 

[leemason]

I've also just had a search online and there are some Open Source alternatives to Replify (which I think is not that cheap). Here is one: http://wanproxy.org/. I've not used it but it might be worth checking it out.

 

[thebusinesscookbook]

Hi All,

Cutting a long story short, we have been passed a client that is pulling their hair out about their current setup where they need to regularly send a ~8GB file from one site to another.

They have a remote worker creating a video file that then needs inspecting and then later burning to DVD at a later date. Currently they are using dropbox but due to having relatively poor broadband speeds and the size of the file they have quoted that it can take 12 hours or more to send the file.

I don't believe the services we offer are going to help unfortunately but I was wondering if anyone had any suggestions as I would like to at least pass on some advice if possible?

I am going to suggest the option of the remote worker using a remote desktop session to a PC at the end destination/burning point but apart from that I don't see too many other options and as the upload speed is the killer. Any ideas guys?

Thanks in advance and hope you are all well,

Matthew

Hi Matthew,

Is there any reason that after inspection, the remote worker can't burn the DVD?

I'm thinking along your lines of having a RD connection to allow inspection and then for the DVD to be burnt by the remote worker. Of course, I guess it depends on what happens to the DVD next...

Mike

 

[NetwiseHosting]

Hi All,

Apologies for the delay in replying - I have a meeting with the client next week so will relay this information and combined ideas onto him, but so far these are the possibilities of assistance:

1. Secondary internet connection to decrease the upload bottleneck
2. Use a FTP transfer method instead of the current HTTP service
3. Look into the WAN optimisation software/solutions
4. Compression combined with a relevant codec

I will establish the playback information, quality needed and the device the video is intended to be played on so we can find out if the file could be reduced in size - but thank you again one and all for your responses, much appreciated and I will keep you updated!
All the best,

Matthew

 

[NetwiseHosting]

Books can be split into chapters; software applications into files, classes, etc.; musical compositions into movements; movies into scenes; and so on. Is there any way that your video could similarly be split into parts? If so, then each part could be sent separately.

All the best,

Hi there, very true, but I'm not sure how splitting the file up is going to increase the speed overall - could you expand on that?

Hi Matthew,

Is there any reason that after inspection, the remote worker can't burn the DVD?

I'm thinking along your lines of having a RD connection to allow inspection and then for the DVD to be burnt by the remote worker. Of course, I guess it depends on what happens to the DVD next...

Mike

Hi Mike, good point. Rather then the file being sent to the local site, the local site could possibly remote to the remote worker and perform any insepction from there, but yes I suppose that is dependant on what happens next. I will find out!

 

[vmhosts]

I would like to add to Lee's recommendation. WAN optimisation may help you here.

Replify is a neat solution but its man advantages are when transferring the same file across WAN links ie multiple people accessing the same file

Try giving www.sliver-peak.com they provide a free fully functional appliance for slower links http://www.vx-xpress.com/

Any optimisation will depend on the file types. Highly compressed files will be difficult to further optimise

Hope this helps

 

[Mortime Business Software]

Hi there, very true, but I'm not sure how splitting the file up is going to increase the speed overall - could you expand on that?

Sure. You're quite correct. The speed of the actual transmission would not be increased. In fact, transmission may take slightly longer because of the extra network overhead. However, most of the creation and inspection could be carried out simultaneously, and while transmission is taking place.

For example, suppose part 1 has been created. The creator sends part 1 to the inspector, and then gets on with creating part 2 while part 1 is in transit or being inspected. This process could continue until all the parts have been created, inspected and corrected. This would make most of the transmission time redundant because most of the time, all three activities, creation, transmission and inspection, would be occurring simultaneously.

All the best,