Tuesday, November 4, 2008

obama fever

OBAMA WON! Truly a day to remember. So, in the spirit of science funding not being slashed to death I am going to release one of my first python scripts that downloads all images off of the NASA Astronomy Picture of the Day (apod).

I tried keeping the code commented so that it could also serve as a good learning tool. One thing I'd like to include is some way of saving the description of the image. This could be done by saving it into a text file with the same filename as the image, but I suspect there is a better way (using ImageMagick to embed the information?). If you have an idea or preference, post a comment! So, here it goes:


#!/bin/env python
#
# Copyright 2008 Michael Gorelick
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see:
# http://www.gnu.org/licenses/gpl.html.

import re, urllib, sys, os

### Check/Do Help
help = """NASA Image Getter - Michael G. (GPL 2008)
./%s [-h] [imagepath]
imagepath - path to save images
-h, --help - this help"""%sys.argv[0]

if "-h" in sys.argv or "--help" in sys.argv:
exit(help)
elif len(sys.argv) == 1:
print "Not enough program arguments"
exit(help)

### CONFIG
allowedChars = '[-a-zA-Z0-9*:\'.,&!?\(\)//\n"+;_ ]'
allowedFiles = '[jpg|jpeg|png|gif|mov|avi|mpeg|mpg]'
url = "http://antwrp.gsfc.nasa.gov/apod/archivepix.html"
pbarwidth = 80
try:
imagePath = sys.argv[1]
os.makedirs(imagePath)
except Exception, e:
#errno == 17 means the directory already exists.
if e.errno != 17:
exit(e)
print "Saving images to %s"%imagePath

##############################################
## This section looks for links to image pages
print "Searching for image pages in", url
html = urllib.urlopen(url).read()
pagePattern = re.compile('([0-9]{6}).html">(' + allowedChars + '*)')
pages = pagePattern.findall(html)
totalpages = len(pages)
print "Found", totalpages, "image pages."

#############################################
## Now we go to the pages and extract a list
## of images and download
try:
alreadyHave = open(imagePath + "/tracker.log").readlines()
except IOError, e:
alreadyHave = ""
totalhave = len(alreadyHave)
try:
tracker = open(imagePath + "/tracker.log", 'a+')
except IOError, e:
exit(e)

base = url[0:url.rfind('/')]
imagePattern = re.compile('<a href="image/(' + allowedChars + '*.' + \
allowedFiles + ')"', re.IGNORECASE)
sanitizePattern = re.compile('[//\n\r\t]')
imageGet = urllib.URLopener()

for i, page in enumerate(pages):
try:
if page[0]+"\n" not in alreadyHave:
#Extract information from image page
title = sanitizePattern.sub("", page[1])
currentpage = base + "/ap" + page[0] + ".html"
print "Getting " + title
content = urllib.urlopen(currentpage).read()
image = imagePattern.findall(content)

#Parse data
filename = imagePath + "/" + (title + \
image[0][image[0].rfind('.'):]).replace(" ","_")
print "\tFilename: " + filename
url = base + "/image/" + image[0]
print "\tURL: " + url

print "\n[" + "="*(pbarwidth*(i-totalhave)/(totalpages-totalhave)) \
+"@"+"="*(pbarwidth*(i-totalpages)/(totalhave-totalpages)-1)+"]",
sys.stdout.flush()

#Download
imageGet.retrieve(url, filename)
tracker.write(page[0] + "\n")

print "\r" + " "*(pbarwidth+2),"\r",
sys.stdout.flush()
except IndexError, e:
print "\tImage not found\n"
except IOError, e:
print "\t404 Error\n"

tracker.close()
print "Done"

4 comments:

  1. so how to run it ?

    ReplyDelete
  2. Just like any ordinary python script... save it to a file, say apod.py, then run `python ./apod [directory]` where directory is where you want to save the images.

    ReplyDelete
  3. ever heard about PEP 8?

    ReplyDelete
  4. heh, I've been programming for a long time so going to new formatting standards isn't the easiest. In my defense, the 2 space tabs instead of the 4 space is mainly because I do a lot of programming on my eepc which doesn't have the screen size for large tabs.

    As far as I can see however, that and my one line import seem to be the only offenses. Oh, and once or twice I took away the space around an operator to save line-space. Do you see any other problems?

    ReplyDelete