Skip to Content.
Sympa Menu

ch-scene - Re: The Future Of Music

ch-scene AT lists.ibiblio.org

Subject: RTP-area local music and culture

List archive

Chronological Thread  
  • From: grady <grady AT ibiblio.org>
  • To: RTP-area local music and culture <ch-scene AT lists.ibiblio.org>
  • Subject: Re: The Future Of Music
  • Date: Wed, 06 Oct 2004 13:03:48 -0400



bendy(a)lostpla[dot]net wrote:

grady wrote:
> Or, my personal current fave meme of

the week: using a wget script to scrape mp3s from multiple mp3 blogs and then store them in a folder hierarchy that indicates which mp3 blog they came from. You get 100s of new mp3s to listen to, and if you notice something that interests you, you can figure out where it came from & go looking for more . . .


Do example scripts exist or this memevaporware?

Aw, c'mon: http://www.google.com/search?q=wget+mp3+blog

Here's a sample of some of the methods I used for some of the blogs I scraped. And yes, I feel 12% guilty for not taking the time to read most of 'em before slurping. Then again, most of them just read like blogs. I will say that this is primarily an experiment to see if any of the bloggers are consistently on-point enough for me to start checking them on a regular basis. I don't really want to spend my days slogging through 400 new mp3s/week if 390 of them are bad eurodisco.

(these are all for the win32 port of wget, so minor syntactical differences may or may not exist if you're on a different platform)

(note that in each case, I'm calling the URLs of the blogs from textfiles that are referenced on the cmd-line - you'll have to make your own)

(note also that those cmd-lines are being wrapped by the mailer & you'll need to ensure that they wind up all on one line in your shell before you execute them):

domains where you'll want to go offsite (to random other sites) but only 1 level:

http://www.fatplanet.com.au/mp3_archive.html
http://www.livejournal.com/community/talkiewalkie/
http://fingertipsmusic.blogspot.com/
http://www.3hive.com/

wget -np -t1 --limit-rate=800000 -w1 -nc -x -U "Mozilla/5.001 (windows; U; NT4.0; en-us) Gecko/25250101" -A .mp3,.ogg --follow-ftp -r -l1 -erobots=off -H -i extsites.txt



domains where everything is in a single browsable directory & this is it, man:

http://www.o-dub.com/sounds/soulsides/
http://www.moistworks.com/mp3s/
http://www.bellybongo.se/mp3/
http://tofuhut.racknine.net/
http://members.aol.com/beastmystical/music_files/
http://saturn5.com/~moebius/muzik/

wget -t1 -np --limit-rate=800000 -w1 -nc -x -U "Mozilla/5.001 (windows; U; NT4.0; en-us) Gecko/25250101" -A .mp3,.ogg -r -l4 -erobots=off -i ezsites.txt



domains where everything is onsite but scattered all over:

http://www.christopherporter.com/
http://www.scissorkick.com/
http://www.bubblegum-machine.com/culchah.html
http://www.londonlee.com/blog.html

wget -t1 -np --limit-rate=800000 -w1 -nc -x -U "Mozilla/5.001 (windows; U; NT4.0; en-us) Gecko/25250101" -A .mp3,.ogg --follow-ftp -r -l4 -erobots=off -i intsites.txt


domains where everything is offsite but on a predictable other server:

http://www.pop77.com/blog/
http://www.tangmonkey.com/blogs/music/
http://teachingtheindiekidstodanceagain.blogspot.com/
http://music.for-robots.com/
http://copycommaright.blogspot.com/
http://the-big-ticket.blogspot.com/
http://vinyljourney.blogspot.com/

offsite server list for the above:

http://s93123386.onlinehome.us
http://homepage.mac.com/sceldred/.Public/Steve/
http://turbine.slackworks.com/robots/blair/
http://homepage.mac.com/mr.gilbert/.Public/
http://home.comcast.net/~lobstar72/
http://projectmersh.com/

wget -t1 -np --limit-rate=800000 -w1 -nc -x -U "Mozilla/5.001 (windows; U; NT4.0; en-us) Gecko/25250101" -A .mp3,.ogg --follow-ftp -r -l1 -erobots=off -H -D s93123386.onlinehome.us,homepage.mac.com,turbine.slackworks.com,home.comcast.net,projectmersh.com -i knownsites.txt


I don't really want/need to provide a wget tutorial at the moment, so if you're wondering what all those switches do, do a wget --help & find out. Note that you can remove the rate-limit & the wait between downloads & etc if you want to be a bastard. You can also remove the browser-spoofing; I will freely admit that I only did that to avoid getting caught by any anti-bot/slurp snares that might have been out there.

Yeah, I know, I suck.

xo

Ross




Archive powered by MHonArc 2.6.24.

Top of Page