« Commercial Podcaster but poor, terms changed | Main | Long interview with me at Indieish »

March 15, 2006


John S. Jacob

The Debian project uses a program called "Jigdo" (short for Jigsaw Download) that might match your requirements. Jigdo can download pieces of the target file from multiple sources. Jigdo supports upgrading a file by downloading only the changed parts. Unfortunately Jigdo doesn't have a pretty GUI yet -- it's in development but isn't stable.



If i'm not incorrect I believed BitTorrent was based on debian's Jigdo... So it might not help.

Using Bittorent behind a NAT is just a matter of mapping ports from 6881 to 6999. The fact is if you want to use a P2P system, being behind a NAT is going to be a problem, because you'll need to map ports.

Kirk McElhearn

You don't need to zip the files to torrent them - I download Grateful Dead and other live concerts all the time via BT, and they are just folders containing multiple files. And that's behind NAT as well.

My torrent client (Transmission, on OS X) doesn't ask to save individual files - in fact, I've set a DL location, and it doesn't ask at all.

But you're right about the average user not being comfortable with BT or your other solutions. Probably the best way is to create individual zip files of, say, 1 GB each, put them on an HTTP page, and let users download them one at a time. Since most users won't have the bandwidth to do that very quickly (like in one day), it would probably be much easier for them anyway.

Eric Redlinger

I also use Transmission on OS X which lets you select any TCP port you want (including 80) for use with BT. It *always* works for me no matter what the network topography.

Nathan Jones

You probably need a best-of-both-worlds approach: make the music available via both a download manager (BitTorrent, Firefox plugin, or otherwise), and as individual downloads in a web browser.

The question is, what size chunks should you use for the individual downloads? My thought was that you'd have a page listing all artists, with one zip archive for each. That way, people can start by getting the music they are most interested in, then come back for more of what they've bought when they're ready. However, that's a very big number of zip files to download for customers who really do want to get them all. (But at least they'd have the option of using a download manager to automate the process.)

Michael Kohne

I'd go with a custom download app (either write your own or find one). You need to give a good use experience for this, or you're not going to get many takers. If you can find an existing app, then great, but it sounds like you haven't come up with one yet.

One thing to consider: for Windows I know it's possible to get a 'compiler' that bundles up a bunch of Perl into an executable. This would make it possible for you to do something like a perl script that wraps wget.

Good luck on this one.

Don Marti

If you do go the custom app route, please use regular HTTP so that people who want to use "wget --mirror" can do that too.

Tried Coral?

Kirk McElhearn

Re custom download apps: I've had nothing but bad (or mediocre) experiences with them. I think it's best to stick to simple technologies, using HTTP, or perhaps FTP for those who know how to deal with it. Offer BT as well, again, for those who know, since it'll save you bandwidth, but if you want to attract un-geeky users, you need to go the simplest route.

John Buckman

Re custom download apps: I've had nothing but bad (or mediocre) experiences with them. I think it's best to stick to simple technologies, using HTTP, or perhaps FTP for those who know how to deal with it. Offer BT as well, again, for those who know, since it'll save you bandwidth, but if you want to attract un-geeky users, you need to go the simplest route.

I'm agreeing with you, and think I'll go with your earlier suggestion of a few very big ZIP files, limited to under 2GB each, that way everyone can get them, and worst case it'll be something like 10 zips, so you won't even need a download manager to click them all.



About how many files are from the various genres on the site? You may want to consider splitting along those lines, since people may want to get cracking on one of the genres before any of the others (For me, I'd start with the electronica and classical before I even touched the Folk music.)

And I agree against the custom download app. Downloading is already hard enough without some home-grown application bollixing up the works.


I agree with the as many options as possible. have you looked into rsync? I use it to move large filesystems around and keep them up to date across the net. It might be too much of a geek factor though.

One thing to consider is that if someone is shelling out $150 for music in one shot their tolerance for 'geeky' might be a bit higher than normal.

If there is going to be a subscription and continuous updates then a synchronizing solution is definitely needed.

Kirk McElhearn

I'll agree with Craig's post about genres - I'd certainly start with classical, then move on to others, so genre-based archives are probably a good idea. If a given genre is too big, make two archives: classical1 and classical2. And, if updates are made, then you can always just create a new update for each genre, or a single update for everything.

Kirk McElhearn

Oh, and one more thing... You might want to put links on the download page to freeware/shareware download managers for different platforms. This would help those who have dodgy connections, and who might otherwise have trouble restarting downloads.

Greg M. Johnson

Uh, what's wrong with a 20h download? I just downloaded some live Linux DVD's and they took over a day each via bittorrent.

But the advice is good to break it up into smaller sizes via 4-5 genres.

Matt May

So, this is a little Luddite of me, but how about burning 6-8 DVDs (or 3-4 DVD-DLs), shipping them snail-mail, and giving people who buy the package a couple hundred downloads that expire after 7 days just to tide them over? That way, you don't get pummeled for bandwidth or have to build new infrastructure, they get immediate access to some of the music and copies for the archive, _and_ you get to see which artists they were interested in hearing first.

I don't tend to believe that anybody is going to need all 30GB up front anyway. I know how hard it is to have a couple hundred songs in an audition queue, so 7000 would be more than a bit overwhelming.

Gordon Smith

I've been thinking about delivery methods since I first heard about the "Fill your iPod" project. Living in a rural area, and using a 2-way 512/64 satellite link to connect to the world, I only get a 5 GB per month quota (plus 6c per MB thereafter). Downloading anything of significant volume, torrent or not, seems unrealistic for people in my position (never mind those still on dial-up).

My preference would be for DVDs, sent snail-mail, at additional cost. I'd happily pay $200+ for the high-quality Magnatune collection, plus annual update fees.

Emil Perhinschi

You could use CVS or Subversion; both can handle binary files and anonymous connections, synchronization etc. and there are GUI clients ... or you could have somebody make a dumbed down CVS client to move the files to the iPod after sync.

The comments to this entry are closed.