download torrents from rss feed

[sunchiqua]

Is it possible for you to share the feed URL with others ( it would be easier to understand, what exactly you need ) ?

[abhiroopb]

http://showrss.karmorra.info/rss.php?user_id=1007

Thanks!

[abhiroopb]

Update of my feed:

http://showrss.karmorra.info/rss.php...&hd=0&proper=0

[abhiroopb]

I think I have figured out the terminology. What I am looking to do is essentially "Broadcatching" (http://en.wikipedia.org/wiki/Broadcatching).

There is a lot of information on it on the web, but other than the Java app TED there does not seem to be anything particular useful for Ubuntu.

Ideally, I'd want a script that runs every 10-15 minutes and searches the specified feed for any new torrent files, and if there are any new torrent files it should download it to a folder which is then being scanned by Transmission (creating an autodownload function).

[unutbu]

abhiroopb, are you looking for a script to monitor a url web page, check for new torrents (differently named files?) and then issue a command based on the new files found?
If that is all it takes, I think I can write a script to do that.

Is this the page you want monitored:
http://showrss.karmorra.info/rss.php...&hd=0&proper=0

and what is the command you'd like issued based on the new feeds found?

[abhiroopb]

Thats essentially it, unutbu.

Basically, I have configured that particular website to create a personalised RSS feed for me to only provide me with the torrents I require. So, every single torrent that shows up in that feed would have to be downloaded to my specially assigned "torrents" folder that is being watched by Tranmission.

It seems simple enough, but I have no idea how to go about it.

Thanks a lot for your help.

[unutbu]

Okay, once I have the list of new torrents, such as

http://torrent.zoink.it/True.Blood.S...ztv%5D.torrent

What does one do with this? Do you download this file to your computer and put it in a certain directory? Which directory? (As this makes abundantly clear, I am a torrent noob...)

[abhiroopb]

Thats pretty much all you have to do, once it appears in the specific directory, as long as your torrent client is running it should automatically add it.

All I would need the script to do is this:

1. Search RSS feed every few minutes
2. Download NEW .torrent file (this is obviously important as I don't want the same .torrent file being downloaded over and over again).
3. Place newly downloaded .torrent file in specified directory.

Then transmission (my bittorrent client of choice would download the torrent after scanning the specified directory.

[unutbu]

I know you do not need such explicit directions abhiroopb, but I'm writing it this way just in case others find this useful...

• Install the python-beautifulsoup package
• Save this to ~/bin/tormon.py
  
  

  #!/usr/bin/env python
  
  import urllib2,urlparse
  from urllib2 import HTTPError,URLError
  from BeautifulSoup import BeautifulSoup
  import os
  import optparse
  
  __usage__='''
  tormon.py -O ~/test/tormon -u "http://rss.feed"
  '''
  
  class Main(object):
      '''
      tormon checks an rss feed for new torrents. When it finds a new .torrent, to
      downloads it to a specified output directory, where (presumably) a monitoring
      torrent program will download the corresponding file.    
      '''
      def parse_options(self):
          usage = 'usage: %prog [options]'+__usage__
          parser = optparse.OptionParser(usage=usage)
          parser.add_option(
              '-O', '--output_dir', dest='output_dir', 
              help='directory into which new torrents are saved', 
              metavar='DIR')
          parser.add_option(
              '-f', '--filetype', dest='filetype',
              action='append',
              default=[],
              help='admissible file types', 
              metavar='TYPE')
          parser.add_option(
              '-d', '--downloaded_torrents', dest='downloaded_torrents',
              default=os.path.expanduser('~/.downloaded_torrents'),
              help='log of already downloaded torrents', 
              metavar='FILE')
          parser.add_option(
              '-e', '--error_log', dest='error_log',
              help='log of torrents tormon failed to download', 
              metavar='FILE')
          parser.add_option(
              '-b', '--batch', dest='batch',
              help='file containing list of rss feed urls', 
              metavar='FILE') 
          parser.add_option(
              '-u', '--url', dest='url',
              action='append',
              default=[],
              help='url of the rss feed', 
              metavar='URL')
          parser.add_option(
              '-m','--mark_all_downloaded', dest='mark_all_downloaded',
              action='store_true', 
              default=False,
              help="mark all torrents as already downloaded")
          parser.add_option(
              '-M','--match_by_filename', dest='match_by_filename',
              action='store_true', 
              default=False,
              help="recognize downloaded files by filename, not URL. Matching by URL is the default.")        
          (self.opt, args) = parser.parse_args()
          if self.opt.batch:
              for line in open(self.opt.batch,'r'):
                  line=line.strip()
                  if line and not line.startswith('#'):
                      self.opt.url.append(line)
          if not self.opt.output_dir:
              self.opt.output_dir=os.path.expanduser('~/Desktop')
          if not self.opt.filetype:
              self.opt.filetype=['.torrent']
          if not self.opt.error_log:
              self.opt.error_log=self.opt.downloaded_torrents+'.errors'
          try:
              os.makedirs(self.opt.output_dir)
          except OSError:
              if not os.path.exists(self.opt.output_dir):
                  print('tormon failed to create directory %s'%self.opt.output_dir)
                  exit(1)
      def load_list_of_already_downloaded_torrents(self):
          try:
              self.downloaded=open(self.opt.downloaded_torrents,'r').read().split()
          except IOError:
              self.downloaded=[]
          try:
              self.errors=open(self.opt.error_log,'r').read().split()
          except IOError:
              self.errors=[]
      def update_downloaded(self,url):
          self.downloaded.append(url)
          try:
              self.errors.remove(url)
          except ValueError:
              pass        
      def download_torrent(self,url):
          try:
              sock=urllib2.urlopen(url)
          except (HTTPError, URLError):
              # print('tormon failed to download %s'%url)
              if url not in self.errors:
                  self.errors.append(url)
          else:
              filename=self.url2filename(url)
              target_file=os.path.join(self.opt.output_dir,filename)
              print('Downloading %s'%target_file)
              content=sock.read()
              sock.close()
              fh=open(target_file,'w')
              fh.write(content)
              fh.close()
              self.update_downloaded(url)
      def url2filename(self,url):
          return os.path.basename(urlparse.urlparse(url)[2])
      def has_been_downloaded(self,url):
          if self.opt.match_by_filename:
              filename=self.url2filename(url)
              return (filename in [self.url2filename(link) for link in self.downloaded])
          else:
              return (url in self.downloaded)
      def parse_rss_feed(self):
          for url in self.opt.url:
              print('RSS feed: %s'%url)
              try:
                  sock=urllib2.urlopen(url)
              except (HTTPError, URLError):
                  print('tormon failed to download %s'%url)
              else:
                  content=sock.read()
                  sock.close()
                  soup=BeautifulSoup(content)
                  links=([link.nextSibling for link in soup.findAll('link')]+
                         [link['href'] for link in soup.findAll('a')]+
                         [link['url'] for link in soup.findAll('media:content')])
                  for link in links:
                      if (any([link.lower().endswith(ending)
                               for ending in self.opt.filetype])
                          and not self.has_been_downloaded(link)):
                          if self.opt.mark_all_downloaded:
                              print('Marking %s as downloaded'%link)
                              self.update_downloaded(link)
                          else:
                              self.download_torrent(link)
      def save_list_of_already_downloaded_torrents(self):
          fh=open(self.opt.downloaded_torrents, 'w')
          fh.write('\n'.join(self.downloaded))
          fh.close()
          fh=open(self.opt.error_log, 'w')
          fh.write('\n'.join(self.errors))
          fh.close()
      def __init__(self):
          self.parse_options()        
          self.load_list_of_already_downloaded_torrents()
          try:
              self.parse_rss_feed()
          except KeyboardInterrupt:
              pass
          finally:
              self.save_list_of_already_downloaded_torrents()
  if __name__=='__main__':
      Main() 

• Make it executable:
  

  chmod +x ~/bin/tormon.py

• Test it:
  

  tormon.py -O /path/to/downloaded/dot/torrent/files -u "http://rss.feed"

  Putting the url of the rss feed in quotes protects the url from being misinterpreted by the shell.
  
  • /path/to/downloaded/dot/torrent/files should be the directory where you wish the .torrents to be saved.
  • If you omit the -O flag, the files will be placed in ~/Desktop.
  • You should see a list of the torrents downloaded.
  • If you run the same command again, nothing should be downloaded unless a new torrent has been added to the rss feed.
  • This command creates /path/to/downloaded/dot/torrent/files if it does not already exist.
  • It will save a list of the urls of successfully downloaded torrents to ~/.downloaded_torrents.
  • I tried to make the script general enough to be of use to other people too. If some else has a rss feed they wish to monitor, they can use the command
    

    tormon.py -u "http://rss.feed"

  • To monitor more than one rss feed, you can use the -u flag multiple times:
    

    tormon.py -u "http://rss.feed1" -u "http://rss.feed2"

  • ~/.downloaded_torrents is a plain text file. If you wish to download the same torrent again, just delete the url from this file. If you wish to "blacklist" a certain torrent, you can just add the url to this file.
  • If you want to save the list of downloaded torrents to a different file, you can use the command
    

    tormon.py -d /path/to/downloaded/torrents/file -u "http://rss.feed1"

• If everything seems to be working properly, then set up the cron job: Run
  

  crontab -e

  A text editor (e.g. nano) will pop up.
  
  Add this to the top of the file, changing USER to your username (e.g. abhiroopb)
  

  PATH=/sbin:/bin:/usr/sbin:/usr/bin:/home/USER/bin
  MAILTO=USER

  Put this anywhere under the "PATH=..." line.
  

  */15 * * * * tormon.py -u "http://rss.feed"

  This will run the tormon.py command once every fifteen minutes.
  
  Make sure there is at least one blank line at the end of the crontab file. (It's a quirk of the cron program.)
  Save and exit the text editor.
  
  If you have a mail transfer agent (e.g. exim4) installed, then the MAILTO=USER line in the crontab will tell cron to send you the output of the command in a (local) email message.
  You then get to see what torrents have been most recently downloaded.
  
• That's it. See if it works. I'll be expecting your bug reports. Enjoy!

[abhiroopb]

Hi there.

WOW! That is amazing! Its perfect, and it means I don't have to be running a clunky java programme in the background all day long.

I really love what you can do with linux!

Thanks again, like I said this is exactly what I was looking for, and with the help of the linux community it took me less than a day to find it!

NB: I have PMed you about something else.