Downloading 7000 files: helpful suggestions needed, because Bittorrent isn't going to work

I've started working on the "Fill your ipod for $150" project, and had planned on using bittorrent and the ibiblio "Osprey" torrent server, which is meant for just this http://osprey.ibiblio.org/

I'm soliciting ideas on ways I can make 30gb of separate MP3s easily downloadable, because it doesn't look like bittorrent will work.

I'm not able to get any of the files from http://torrent.ibiblio.org/ using my bittorrent clients (azareus or bits on wheels) and that makes me nervous. Granted, I'm behind a NATed DSL connection, which I know is not ideal for bittorrent, but I was hoping it would at least work. And, I'm sure many other people will also have internal TCP/IP addresses and DSL or corporate networks, so whatever I use needs to actually work. Also, one of my bittorrent clients (azareus) pops a "Save as" dialog for each filename in a torrent file, which would be awful for 7000 mp3s.

The problem is that I can't give people a 20gb zip file to download. That would

- take 20h to download
- not unzip on many operating systems, due to a 2gb zip file limit that is widely present
- not enable incremental updates

I was hoping that by using bittorrent, I would get:

- each mp3 would be downloaded separately, so you'd get some music very quickly
- incremental updates would be possible, since only the music you didn't have would be downloaded
- no unzipping or multi-gig files

However, I suspect bittorrent is too particular to be used widely as the main way of distributing content, so I'm soliciting other suggestions. Bandwidth really isn't a problem for me, that's not the reason I was looking at bittorrent.

One possibility, which is what microsoft does with many installers (and emusic with its downloader), is to give people a small program which then downloads via HTTP all the files. I'm open to that, if it ran on all platforms, but I don't love it. Better would be a Java app, running in the browser, so it'd work for most people, but saving files locally is, I assume problematic from a a browser-based java app, and more specifically, this sounds like a lot of work.

What http://mp3search.ru does is put all the mp3 links on one page, and asks the user to use a "download helper" application which sucks down all the mp3s on an html page. That's probably not great either, especially with 7000 mp3s.

Another option is for more to provide a batch file, which uses "wget" (or curl) to download all the files. That would be easy to develop, would allow incremental downloads, but would be ugly to look at, and many novice users would be nervous about this.

Yet a third option is a browser plugin, there are several that support p2p, or I could write a simple app using the tcl plugin, but I suspect users wouldn't like to have to permanently install a browser plugin just to download our music.

So... I've not yet found a good solution to this dilemma -- so I'm soliciting ideas on ways I can make 30gb of separate MP3s easily downloadable.

Comments

[John S. Jacob]

The Debian project uses a program called "Jigdo" (short for Jigsaw Download) that might match your requirements. Jigdo can download pieces of the target file from multiple sources. Jigdo supports upgrading a file by downloading only the changed parts. Unfortunately Jigdo doesn't have a pretty GUI yet -- it's in development but isn't stable.

http://atterer.net/jigdo/

[JMG]

If i'm not incorrect I believed BitTorrent was based on debian's Jigdo... So it might not help.

Using Bittorent behind a NAT is just a matter of mapping ports from 6881 to 6999. The fact is if you want to use a P2P system, being behind a NAT is going to be a problem, because you'll need to map ports.

[Kirk McElhearn]

You don't need to zip the files to torrent them - I download Grateful Dead and other live concerts all the time via BT, and they are just folders containing multiple files. And that's behind NAT as well.

My torrent client (Transmission, on OS X) doesn't ask to save individual files - in fact, I've set a DL location, and it doesn't ask at all.

But you're right about the average user not being comfortable with BT or your other solutions. Probably the best way is to create individual zip files of, say, 1 GB each, put them on an HTTP page, and let users download them one at a time. Since most users won't have the bandwidth to do that very quickly (like in one day), it would probably be much easier for them anyway.

[Eric Redlinger]

I also use Transmission on OS X which lets you select any TCP port you want (including 80) for use with BT. It *always* works for me no matter what the network topography.

[Nathan Jones]

You probably need a best-of-both-worlds approach: make the music available via both a download manager (BitTorrent, Firefox plugin, or otherwise), and as individual downloads in a web browser.

The question is, what size chunks should you use for the individual downloads? My thought was that you'd have a page listing all artists, with one zip archive for each. That way, people can start by getting the music they are most interested in, then come back for more of what they've bought when they're ready. However, that's a very big number of zip files to download for customers who really do want to get them all. (But at least they'd have the option of using a download manager to automate the process.)

[Michael Kohne]

I'd go with a custom download app (either write your own or find one). You need to give a good use experience for this, or you're not going to get many takers. If you can find an existing app, then great, but it sounds like you haven't come up with one yet.

One thing to consider: for Windows I know it's possible to get a 'compiler' that bundles up a bunch of Perl into an executable. This would make it possible for you to do something like a perl script that wraps wget.

Good luck on this one.

[Don Marti]

If you do go the custom app route, please use regular HTTP so that people who want to use "wget --mirror" can do that too.

Tried Coral?
http://www.coralcdn.org/

[Kirk McElhearn]

Re custom download apps: I've had nothing but bad (or mediocre) experiences with them. I think it's best to stick to simple technologies, using HTTP, or perhaps FTP for those who know how to deal with it. Offer BT as well, again, for those who know, since it'll save you bandwidth, but if you want to attract un-geeky users, you need to go the simplest route.

[John Buckman]

Re custom download apps: I've had nothing but bad (or mediocre) experiences with them. I think it's best to stick to simple technologies, using HTTP, or perhaps FTP for those who know how to deal with it. Offer BT as well, again, for those who know, since it'll save you bandwidth, but if you want to attract un-geeky users, you need to go the simplest route.

I'm agreeing with you, and think I'll go with your earlier suggestion of a few very big ZIP files, limited to under 2GB each, that way everyone can get them, and worst case it'll be something like 10 zips, so you won't even need a download manager to click them all.

-john

[Craig]

About how many files are from the various genres on the site? You may want to consider splitting along those lines, since people may want to get cracking on one of the genres before any of the others (For me, I'd start with the electronica and classical before I even touched the Folk music.)

And I agree against the custom download app. Downloading is already hard enough without some home-grown application bollixing up the works.

[Justin]

I agree with the as many options as possible. have you looked into rsync? I use it to move large filesystems around and keep them up to date across the net. It might be too much of a geek factor though.

One thing to consider is that if someone is shelling out $150 for music in one shot their tolerance for 'geeky' might be a bit higher than normal.

If there is going to be a subscription and continuous updates then a synchronizing solution is definitely needed.

[Kirk McElhearn]

I'll agree with Craig's post about genres - I'd certainly start with classical, then move on to others, so genre-based archives are probably a good idea. If a given genre is too big, make two archives: classical1 and classical2. And, if updates are made, then you can always just create a new update for each genre, or a single update for everything.

[Kirk McElhearn]

Oh, and one more thing... You might want to put links on the download page to freeware/shareware download managers for different platforms. This would help those who have dodgy connections, and who might otherwise have trouble restarting downloads.

[Greg M. Johnson]

Uh, what's wrong with a 20h download? I just downloaded some live Linux DVD's and they took over a day each via bittorrent.

But the advice is good to break it up into smaller sizes via 4-5 genres.

[Matt May]

So, this is a little Luddite of me, but how about burning 6-8 DVDs (or 3-4 DVD-DLs), shipping them snail-mail, and giving people who buy the package a couple hundred downloads that expire after 7 days just to tide them over? That way, you don't get pummeled for bandwidth or have to build new infrastructure, they get immediate access to some of the music and copies for the archive, _and_ you get to see which artists they were interested in hearing first.

I don't tend to believe that anybody is going to need all 30GB up front anyway. I know how hard it is to have a couple hundred songs in an audition queue, so 7000 would be more than a bit overwhelming.

[Gordon Smith]

I've been thinking about delivery methods since I first heard about the "Fill your iPod" project. Living in a rural area, and using a 2-way 512/64 satellite link to connect to the world, I only get a 5 GB per month quota (plus 6c per MB thereafter). Downloading anything of significant volume, torrent or not, seems unrealistic for people in my position (never mind those still on dial-up).

My preference would be for DVDs, sent snail-mail, at additional cost. I'd happily pay $200+ for the high-quality Magnatune collection, plus annual update fees.

[Emil Perhinschi]

You could use CVS or Subversion; both can handle binary files and anonymous connections, synchronization etc. and there are GUI clients ... or you could have somebody make a dumbed down CVS client to move the files to the iPod after sync.