Shell Scripts Can Do Anything!

So I found this podcast that I like called FLOSS Weekly. It is all about open source software and the whole culture surrounding it. So I listen to a few episodes with topics that particularly catch my eye. It is released under the creative commons so they have no problem providing a nice easy download link. But the webpage sucks so going to each episode will take like a year to get to. So I download just a few before I figure I need a better way to do this.
 
First, I start by figuring out where the file is, with Firefox this is as difficult as: right clicking on the link, then selecting “Copy Link Location”. Then, I head over to a command line and see what happens when I punch it in. I use wget, a simple tool for downloading stuff at the command line (comes with every install of Linux I have ever used), and get this:
$ wget http://www.podtrac.com/pts/redirect.mp3/twit.cachefly.net/FLOSS-076.mp3
--2009-07-08 00:56:27-- http://www.podtrac.com/pts/redirect.mp3/twit.cachefly.net/FLOSS-076.mp3
Resolving www.podtrac.com... 69.16.232.36
Connecting to www.podtrac.com|69.16.232.36|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://twit.cachefly.net/FLOSS-076.mp3 [following]
--2009-07-08 00:56:27-- http://twit.cachefly.net/FLOSS-076.mp3
Resolving twit.cachefly.net... 205.234.175.175
Connecting to twit.cachefly.net|205.234.175.175|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 25931432 (25M) [audio/mpeg]
Saving to: `FLOSS-076.mp3'
 
100%[============================>]
25,931,432 1.84M/s in 13s
 
2009-07-08 00:56:41 (1.87 MB/s) - `FLOSS-076.mp3' saved [25931432/25931432]

The gibberish flies past my eyes, and I scroll up to see what happened. The download was a little slow and there was some weird redirection, but it worked. So I try changing one number near the end of the filename, and it worked. So I did it again, and again, and again. I have 9 or so episodes now, and I certainly don’t want to keep doing this up/backspace/number maneuver, it could give anyone carpal tunnel, another 60 times, so I write a script. For the unacquainted a script is a simple little program that automates tasks.
 
Since my task is just a repeat of what I am doing at a command line, and I am using a BASH shell, I think that bash it my best bet for this task. I do a quick Google search for “bash for loop” and find a page full of examples. I pick an example that makes the most sense to me, and I quickly cobble together this which I called getfloss:
#!/bin/bash
for (( b=0; b<=10; b++ ))
do
 or (( c=0; c<=10; c++ ))
 do
  wget -nc http://www.podtrac.com/pts/redirect.mp3?http://twit.cachefly.net/FLOSS-0$b$c.mp3
 done
done

Pretty much it says, repeat the next steps 10 times while counting in $b, repeat the next step 10 times while counting in $c, then download the file "FLOSS-0$b$c.mp3" replacing all the $b and $c with numbers.
 
I gave the file execute permissions using "chmod +x getfloss", then I ran it buy typing "./getfloss". Finally I have my script to download all of FLOSS weekly, and it took me a whole minute and a half. The downloading took longer, but I got up and got some juice and it was done when I got back.
 
Here is a copy of getfloss, I am releasing this trash under the BSD license. I know that if you are not used to programming that is a real heavy example, and if you are used to programming this is easy sauce. This is meant to spark ideas in people, and not necessarily be a useful solution for other problems. If you have never written a script before and think you might be interested take a look at the Advanced BASH Scripting Guide. If anyone is having trouble automating their problems, let me know, maybe there is some way to do it, I am usually good at figuring this stuff out.
 
Now to tweak my script and hit up some porn sites with it...

Share
  1. I understood this better in Greek. :P

    • This one is not for everyone, if you have never programmed anything this more arcane than a ….. very arcane thing. Did you at least get that I downloaded a bunch of stuff by writing a script?

      • I did get that. I have programmed a calculator in Visual Basic before. I got a C+(+) in that class. Notice the run on joke in there. I'll stop now.

  2. Or just get Curl. Because then you can say:

    curl -O 'http://www.podtrac.com/pts/redirect.mp3?http://twit.cachefly.net/FLOSS-001-076.mp3&#039

    And we even still get credit for the downloads with PodTrac (important when we have sponsors).

    Seriously, having discovered Curl, I ignore Wget except to mirror a section of a website.

    • Sweet! I will redo the download like that so that you get credit, I didn't know the ins and outs curl I just wanted to get the job done as fast as possible.

      Actually I did what you said, and it just downloaded a bunch of html (which look like mp3 files at first glance) files which look like they would download the mp3's. I suspect you got credit for it though.

      Nope, I suspect the commenting system botched the URL in your command because I am getting a very hacked together looking 404 page.

      With the command: curl -O 'http://twit.cachefly.net/FLOSS-001-076.mp3&#39;
      it only 404's on the first 5 then the rest work, does that still give you credit?

  3. Wouldn't:
    wget -nc http://www.podtrac.com/pts/redirect.mp3?http://tw

    Do effectively the same thing?

    I suppose if you don't know about brace expansion your way works, but it seems like serious overkill to me. Also, why'd you use two nested loops instead of a single for() loop counting from 1 to 76?

    • That seems like it would work, I thought that bash used square braces for that kind of thing….
      I guess that is why I used two loops, I know how loops work and I am fuzzy on the details of the brace expansion, can you recommend any good reading so that I can learn more?

      PS you may want to sign up for an intense debate account, it will follow you to any site that uses it, and will let you posts through moderation more easily.

Leave a Comment


NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>