4chan download script

Jag brukar ju vanligtvis skriva på svenska, men eftersom detta inlägget riktar sig mot en internationell publik tänker jag ta tillfället i akt och skriva det på engelska.

The 4chan download script

This is an update of the 4chan download script written by Daniel Triendl, http://blog.pew.cc/blog/4chan+download+script/

The modified script downloads every image file in a 4chan thread, preserving the original file names (not the incrementing numbers given by 4chan). Perfect for downloading entire sets of pictures or other original content. Tested on a few different boards but should theoretically work on all. Anyway, here it is:

#!/bin/bash
# A bash script for downloading all images in a 4chan thread to their original
# filenames. Updates every 30 seconds until canceled or the thread disappears.
#
# Copyright 2008, 2010 Daniel Triendl, Anton Eliasson
# http://blog.pew.cc/blog/4chan+download+script/
# http://antoneliasson.wordpress.com/2010/08/03/4chan-download-script/
#
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
#

if [ "$1" = "" ]; then # no arguments
 echo "Usage: `basename $0` <4chan thread url> [optional: download directory]"
 exit 1
fi

if [ "$2" = "" ]; then # only one argument
 LOC=$(echo "$1" | egrep -o '([0-9]*)$' | sed 's/\.html//g' ) # find out the thread number
else
 LOC=$2 # use download dir specified by user
fi
echo "4chan downloader"
echo "Downloading to \"$LOC\" until canceled or 404'd"

if [ ! -d $LOC ]; then
 mkdir -- $LOC
fi

cd -- $LOC # new directory named after the thread number

while [ "1" = "1" ]; do
 TMP=`mktemp` # TMP is the html thread
 TMP2=`mktemp` # TMP2 will be a list of all image addresses
 TMP3=`mktemp` # TMP3 will be a list of all original file names

 # get thread
 echo "Updating..."
 wget -q -O "$TMP" "$1"
 if [ "$?" != "0" ]; then
 echo "Update failed, exiting"
 date
 rm $TMP $TMP2 $TMP3
 exit 1
 fi

 # get file list, space separated
 grep -E -o 'http://images.4chan.org/[a-z0-9]+/src/([0-9]*).(jpg|png|gif)' "$TMP" | uniq | tr "\n" " " > "$TMP2"

 # get original file name list, space separated (spaces in filenames changed to underlines)
 sed 's/ /_/g' "$TMP" | grep -E -o '<span_title=\"[[:alnum:]_-]*.*(jpg|png|gif)' | awk -F \" '{print $2}' | tr "\n" " " > "$TMP3"

 COUNT=`cat $TMP3 | wc -w` # total number of files/names
 for ((i=1; i<=$COUNT; i++)); do
 wget -nv -nc -O `cut -d ' ' -f $i $TMP3` `cut -d ' ' -f $i $TMP2` # now download all files, one by one
 done

 rm $TMP $TMP2 $TMP3

 echo "Waiting 30 seconds before next run"
 sleep 30
done;

This should run on any Linux-based OS using the bash shell. Feel free to contact me if you find any bugs and/or improve the script.