Offer to access my test usenet indexing site

oneman

VIP Member
VIP Member
Joined
Aug 23, 2007
Messages
7,504
Reaction score
1,284
Location
Essex
As you probably know nzbmatrix and newzbin(2) are now both history and may of the remaining sites have either stopped accepting new members, charge too much, are overwhelmed, are missing features (e.g. API/RSS) and/or not indexing correctly.

Second problem is the speed at which posts are being removed from distribution. AFAIK all the main usenet providers are now responding to take down notices so you potentially have a few hours to grab a post before its removed.

So I've been playing with setting up a test usenet (newsgroup) indexing site for my own use using newznab. The aim is very targeted, its not a general purpose indexing site that is attempting to index everything out there that has ever been posted.

My aim is to provide a real time automated feed to support sickbeard and couchpotato. This means as files are posted then as long as your sickbeard/couchpotato server is running when the files get posted (for example overnight for US shows) then you should be able to grab the posts before they get taken down.

To this end
- I am only index groups which focus on TV and Movies.
a.b.boneless
a.b.etc
a.b.hdtv
a.b.hdtv.x264
a.b.highspeed
a.b.mma
a.b.multimedia
a.b.nl
a.b.teevee
a.b.tv
- Limiting the groups allows me to index quicker then most sites. I am using Astraweb and my releases in general are appearing about 30 minutes to an hours ahead of nzbsrus and a couple of hours ahead of binsearch.
- My plan is to only store 30 days worth of NZB files, I've backfilled around 14 days for testing. I may extend this depending on how much storage it takes. But if you want to search the old posts then go to binsearch or one of the other sites.
- I have RSS, API and SABNZBD support enabled
- The server has not got unlimited resources. Lets just put it this way, its a slow box.
- There is currently a know issue with obfuscated headers. Basically in an attempted to get around the issue of take down notices, the groups which upload the files have been replacing the contents with a random code in the subject and filename (and sometimes encrypting the files as well). I am working on a process which can match this code with real contents so the post can be correctly indexed but its probably going to take time.

Bearing all that in mind, I am offering access to the service to members here for free. There are two conditions, I would like know what you plan on downloading (like your sickbeard list) and more importantly I would like feedback on success rate for automatic feeds.

If you are interested in taking part then PM me. I will be adding people gradually so as not to overwhelm the poor server.
 
I've been working on 60 days retention (backfilled) and at the moment I'm looking at about 100Gb storage for about 40 groups.

The obfuscation is a real pain, so I would be interested if you get around this.
 
Last edited:
I've been working on 60 days retention (backfilled) and at the moment I'm looking at about 100Gb storage for about 40 groups.

The obfuscation is a real pain, so I would be interested if you get around this.

I've got around 30 days but only the main groups for TV and movies. Also only bother indexing posts over 100mb. That's around 20k NZB which take up 2GB and database is around 6GB.

As for hashed headers, I've have deep password analysis switched on, this means it also pulls down a part of the rar files and thus generates a file list. About 1/2 the hashed posting have the real contents unhashed in the rar file. As I am running sphinx, it indexes the files inside the rar so at least they are manually searchable. Next step is to take the unhashed file names and update the release details with them. Then that would allow API search for apps like sickbeard.
 
Back
Top