Newznab on Ubuntu 11.10¶

This guide covers installing Newznab+ on a fresh copy of Ubuntu 11.10 64-bit server and assumes a basic knowledge of using Linux via the command line.

TODO: Install and configure Memcache?

Install Ubuntu¶

Grab a copy of Ubuntu from their website and install it. Once the install is complete, log in, fire up a terminal window, do a full upgrade and reboot (entering your password when prompted):

sudo apt-get update
sudo apt-get -y upgrade
sudo apt-get -y dist-upgrade
sudo reboot

You’ll probably want to install SSH if you don’t have it already:

sudo apt-get install -y ssh

Additionally, we’re going to be compiling some code from source, so we’ll need the tools to do so:

sudo apt-get install build-essential checkinstall

Install Newznab¶

Once Ubuntu is installed and updated, we can begin installing Newznab. Before we get started though, we need to decide where to install Newznab. Typically, web-related stuff goes in /var/www/, so lets put Newznab in /var/www/newznab. Let’s create the directory and set it writeable by our user for now:

sudo mkdir -p /var/www/newznab
sudo chmod 777 /var/www/newznab

Now we can begin installing the prerequisites.

Prerequisites¶

Newznab as a few dependencies, so let’s start with installing the required software. According to the install docs, Newznab needs the following software:

PHP 5.2+

GD Imaging Library w/PHP integration

PEAR

Database: MySQL (or Percona)

Web Server: Apache (or Nginx)

Its worth mentioning a few things at this point. The first is that the default version of MySQL that comes with Ubuntu is kind of old (version 5.1). Therefore, we’ll also cover installing a higher-performance version (Percona version 5.5) instead. Additionally, Apache is listed as a requirement, however it is possible to use a different web server instead. Therefore, we’ll go over how to use the blazing-fast Nginx as well for those who don’t want to use Apache.

The following software is optional, but we’ll also cover installing it and setting it up:

Unrar

FFmpeg

Lame

MediaInfo

Sphinx

By the end of this guide we’ll have a fully working Newznab+ install with all optional components working as well.

PHP¶

Let’s start by installing PHP and required PHP extensions:

sudo apt-get install -y php5 php5-dev php-pear php5-gd php5-mysql php5-curl

The install docs also say that PHP needs a few tweaks. Using your favorite text-editor (I’m using nano) open up php.ini:

sudo nano /etc/php5/cli/php.ini

Scroll down and look for register_globals, make sure it is set to:

register_globals = Off

Next, look for max_execution_time, and set it to (2 minutes):

max_execution_time = 120

Next, look for memory_limit, and either set it to 1024M, or -1 for no limit:

memory_limit = 1024M

While you’re here, you should also set PHP’s date.timezone (a list of available timezone settings can be found here):

date.timezone = Europe/London

That should do it for PHP for the time being (we’ll come back to it when we configure the web server).

Database¶

You have a few options here, we’ll cover two:

Use MySQL 5.1

Use Percona 5.5

I’d recommend that you use Percona, but the choice is yours. Pick one and follow one of the sections below (do not install MySQL and Percona).

MySQL 5.1

If you decided to use MySQL, simply install it via aptitude:

sudo apt-get install mysql-server-5.1 mysql-client-5.1 libmysqlclient-dev

Percona 5.5

Installing Percona requires a little extra work upfront, but in the long run it is generally worth it as Percona is a high-tuned fork of MySQL. The first step is to add Percona into aptitude:

gpg --keyserver  hkp://keys.gnupg.net --recv-keys 1C4CBDCDCD2EFD2A
gpg -a --export CD2EFD2A | sudo apt-key add -

That retrieves and installs the GPG keys needed to add Percona’s aptitude repository. Now we need to tell aptitude about the repositories:

sudo sh -c "echo  \"\n#Percona\" >> /etc/apt/sources.list"
sudo sh -c "echo  \"deb http://repo.percona.com/apt lenny main\" >> /etc/apt/sources.list"
sudo sh -c "echo  \"deb-src http://repo.percona.com/apt lenny main\" >> /etc/apt/sources.list"

Now update and install it:

sudo apt-get update
sudo apt-get install -y percona-server-client-5.5 \
                        percona-server-server-5.5 \
                        libmysqlclient-dev

It’ll ask for a password to use for the MySQL root user, so pick one and remember it.

Web Server¶

OK, now for the web server. Once again, you have multiple choices and we’ll cover two:

Apache

Nginx

Apache is “easier” and generally more flexible, but it also tends to be a system hog. So, if you can’t make up your mind it is probably best to take the easy route and install Apache. If you feel like squeezing the most out of your machine and you are a little more skilled with configuring software, Nginx might be a good choice.

Again, pick one and follow the steps below (don’t install both).

Apache

First install it:

sudo apt-get install -y apache2

You’ll also need to configure the php.ini for Apache, so follow the section above about configuring PHP (from above), but this time, edit the file /etc/php5/apache2/php.ini:

sudo nano /etc/php5/apache2/php.ini

Let’s create the site configuration file for Apache. Open up a new file:

sudo nano /etc/apache2/sites-available/newznab

Here is a template you can use for this file. You should change settings where appropriate for your setup:

<VirtualHost *:80>
    ServerAdmin webmaster@localhost
    ServerName localhost    # You might want to change this

    # These paths should be fine
    DocumentRoot /var/www/newznab/www
    ErrorLog /var/log/apache2/error.log
    LogLevel warn
</VirtualHost>

Now we need to disable the default Apache settings, enable the one we just created for Newznab and enable modrewrite:

sudo a2dissite default
sudo a2ensite newznab
sudo a2enmod rewrite
sudo service apache2 restart

Nginx

First install it:

sudo apt-get install -y nginx

Now, for Nginx there are multiple ways to serve PHP files. Probably the best is php-fpm, which is basically a daemon that runs and serves PHP to Nginx. So, let’s go ahead and install that now:

sudo apt-get install -y php5-fpm

You’ll also need to configure the php.ini for FPM, so follow the section above about configuring PHP (from above), but this time, edit the file /etc/php5/fpm/php.ini:

sudo nano /etc/php5/fpm/php.ini

Just for good measure, restart the daemon:

sudo /etc/init.d/php5-fpm restart

Let’s create the configuration file for Nginx. Open a new file:

sudo nano /etc/nginx/sites-available/newznab

Here is a template you can use for this file. You should change settings where appropriate for your setup:

server {
    # Change these settings to match your machine
    listen 80 default_server;
    server_name localhost;

    # Everything below here doesn't need to be changed
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    root /var/www/newznab/www/;
    index index.html index.htm index.php;

    location ~* \.(?:ico|css|js|gif|inc|txt|gz|xml|png|jpe?g) {
            expires max;
            add_header Pragma public;
            add_header Cache-Control "public, must-revalidate, proxy-revalidate";
    }

    location / { try_files $uri $uri/ @rewrites; }

    location @rewrites {
            rewrite ^/([^/\.]+)/([^/]+)/([^/]+)/? /index.php?page=$1&id=$2&subpage=$3 last;
            rewrite ^/([^/\.]+)/([^/]+)/?$ /index.php?page=$1&id=$2 last;
            rewrite ^/([^/\.]+)/?$ /index.php?page=$1 last;
    }

    location /admin { }
    location /install { }

    location ~ \.php$ {
            include /etc/nginx/fastcgi_params;
            fastcgi_pass 127.0.0.1:9000;

            # The next two lines should go in your fastcgi_params
            fastcgi_index index.php;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    }
}

Let’s make sure that the Nginx logs directory exists and is writeable:

sudo mkdir -p /var/log/nginx
sudo chmod 755 /var/log/nginx

Now, let’s disable the default Nginx site handler and enable our newznab configuration:

sudo unlink /etc/nginx/sites-enabled/default
sudo ln -s /etc/nginx/sites-available/newznab /etc/nginx/sites-enabled/newznab

Finally, let’s restart Nginx:

sudo service nginx restart

Extras¶

unrar¶

Unrar is used to extract files from releases. Installing unrar is trivial:

sudo apt-get install -y unrar

FFmpeg¶

Once again, the version of FFmpeg in Ubuntu’s repository is old and crusty, so lets build a newer one and also add x264 support. Start by grabbing the source code we need:

wget http://ffmpeg.org/releases/ffmpeg-0.8.7.tar.gz
wget ftp://ftp.videolan.org/pub/videolan/x264/snapshots/last_stable_x264.tar.bz2
sudo apt-get install -y libfaac-dev libjack-jackd2-dev \
                        libmp3lame-dev libopencore-amrnb-dev \
                        libopencore-amrwb-dev libsdl1.2-dev libtheora-dev \
                        libva-dev libvdpau-dev libvorbis-dev libx11-dev \
                        libxfixes-dev texi2html yasm zlib1g-dev \
                        libdirac-dev libxvidcore-dev

Let’s build x264 first:

mkdir x264
tar --strip-components=1 -jxf last_stable_x264.tar.bz2 -C x264
cd x264
./configure --enable-static
make
sudo checkinstall --pkgname=x264 --pkgversion="3:$(./version.sh | \
                  awk -F'[" ]' '/POINT/{print $4$5}')" \
                  --backup=no --deldoc=yes --fstrans=no --default

Now, onto FFmpeg:

cd ../
tar xvfz ffmpeg-0.8.7.tar.gz
cd ffmpeg-0.8.7/
./configure --enable-gpl --enable-libfaac --enable-libmp3lame \
            --enable-libopencore-amrnb --enable-libopencore-amrwb \
            --enable-libtheora --enable-libvorbis --enable-libx264 \
            --enable-nonfree --enable-postproc --enable-version3 \
            --enable-x11grab --enable-libdirac --enable-libxvid
make
sudo checkinstall --pkgname=ffmpeg --pkgversion="5:0.8.7" --backup=no \
                  --deldoc=yes --fstrans=no --default
hash x264 ffmpeg ffplay ffprobe

Lame¶

Lame is used for processing audio samples:

sudo apt-get install -y lame

MediaInfo¶

MediaInfo is used to gain information about various types of media files found in releases. In order to install it we simply need to add the MediaInfo aptitude repository:

sudo apt-get install -y python-software-properties
sudo add-apt-repository ppa:shiki/mediainfo
sudo apt-get update
sudo apt-get install -y mediainfo

Sphinx¶

Sphinx is used for full-text searching. It is insanely fast and if you really want your Newznab+ install to fly, it is highly recommended. Unfortunately the version in Ubuntu’s aptitude repository is horribly old, so we’ll need to build a newer version.

Let’s download the source and extract it:

wget http://sphinxsearch.com/files/sphinx-2.0.2-beta.tar.gz
tar xvfz sphinx-2.0.2-beta.tar.gz
cd sphinx-2.0.2-beta

Download and extract libstemmer_c:

wet http://snowball.tartarus.org/dist/libstemmer_c.tgz
tar --strip-components=1 -zxf libstemmer_c.tgz -C libstemmer_c

Configure it:

./configure --prefix=/usr/local --with-libstemmer

Now we’re ready to compile Sphinx. For this step you can speed up the compilation on a multi-core system. If you have a 4-core system, for example, you can do (replace j4 with the number of cores your machine has):

make -j4

Once that finally finished, install it:

sudo checkinstall --pkgname=sphinx --pkgversion="2.0.2-beta" --backup=no \
                  --deldoc=yes --fstrans=no --default

Now we have a nice new version of Sphinx installed in /usr/local. Binaries are installed in /usr/local/bin.

Newznab Source¶

Finally, we can now begin installing Newznab! We’ll be grabbing the latest and greatest version, so we’ll need Subversion installed first:

sudo apt-get install -y subversion

Now we can check out the Newznab code:

svn co svn://svn.newznab.com/nn/branches/nnplus /var/www/newznab

At this point we might as well set the permissions on a couple of directories as well:

sudo chmod 777 /var/www/newznab/www/lib/smarty/templates_c
sudo chmod 777 /var/www/newznab/www/covers/movies
sudo chmod 777 /var/www/newznab/www/covers/anime
sudo chmod 777 /var/www/newznab/www/covers/music
sudo chmod 777 /var/www/newznab/www
sudo chmod 777 /var/www/newznab/www/install
sudo chmod 777 /var/www/newznab/nzbfiles/

Configure Newznab¶

Now that Newznab is installed, we need to configure it. Configuration is done via a web browser.

Run Installer¶

It is now time to configure Newznab. This is done via a web-based installer. Open up http://localhost/install in a web browser (or whatever the address/IP address is of your server) and follow the guided steps.

Enable Groups¶

Head over to /admin/group-list.php in your web browser and pick some groups to index by clicking “activate” on a few groups.

Set Paths and Options¶

Make sure to set your newznab ID.

We need to let Newznab know where all the extra software is that we installed earlier, so head over to /admin/site-edit.php in your browser, scroll down to the “3rd Party Application Paths” section and update the fields:

Unrar Path

/usr/bin/unrar

Mediainfo Path

/usr/bin/mediainfo

Ffmpeg Path

/usr/local/bin/ffmpeg

Lame Path

/usr/bin/lame

If you’d like to enable audio previews, check Save audio preview. It is worthwhile to do rar checking, so set Check For Passworded Releases to Deep.

That’s it for configuration right now, but don’t close your browser yet as we’ll be coming back to the configuration page when configuring Sphinx.

Indexing¶

Back at the command line its time to fire up the binaries and releases update scripts:

cd /var/www/newznab/misc/update_scripts

The update_scripts folder contains a lot of scripts. The most important ones are update_binaries.php and update_releases.php. If you have any experience with screen or tmux, it is highly recommended that you use one of these to run the update scripts, as it will allow you to monitor the update process, observe and resolve issues; this is especially important for newcomers to Newznab. With that said, Newznab also ships with an init-sytle script that can be installed to make Newznab run more or less as a daemon that will start and stop with startup and shutdown, respectively.

Screen or tmux

If you want to go the screen or tmux route, you’ll need to pick one and install it:

# Install screen...
sudo apt-get install -y screen

# ...or tmux
sudo apt-get install -y tmux

In the nix_scripts directory there is a useful script called newznab_screen.sh that runs update_binaries.php and update_releases.php, in addition to a few other scripts, continuously and automatically. First, we need to modify it however, so lets change dir and make a copy:

cd /var/www/newznab/misc/update_scripts/nix_scripts
cp newznab_screen.sh newznab_screen_local.sh

Now open newznab_screen_local.sh in a text editor and modify NEWZNAB_PATH near the top to point to our installation path:

nano newznab_screen_local.sh

Set NEWZNAB_PATH:

export NEWZNAB_PATH="/var/www/newznab/misc/update_scripts"

Now we can run the script via screen:

screen sh newznab_screen_local.sh

You should see the script download headers for the groups that you have enabled and then run various stages that will attempt to group and catalogue the headers. For now, just leave the script running and detach from screen by typing cntl a d.

Sphinx¶

As mentioned previously, Sphinx is a fast full-text indexer. By default, it is disabled in Newznab, so go ahead and enable it by visiting /admin/site-edit.php and setting “Use Sphinx” to “Yes”. While there, you’ll also notice that there are a few other configuration options for Sphinx. By default, Sphinx will index all of the release information, however, there are three other optional indexes: NZB contents, NFO contents and release files. Enabling these optional indexes will add increased processing time, so you will likely want to experiment to see what combination works best for your hardware. For now, you don’t have to enable any of the optional indexes.

To get Sphinx running, we need to generate a sphinx.conf file. To do this we’ll use the nnindexer.php script in misc/sphinx:

cd /var/www/newznab/misc/sphinx
./nnindexer generate

The script will print out the location of the sphinx.conf file, which by default will be /var/www/newznab/sphinx/sphinx.conf. This path needs to be entered into the “Sphinx Configuration Path” setting located at /admin/site-edit.php.

Now we need to start the search daemon and create the indexes and restart the daemon:

./nnindexer.php daemon
./nnindexer.php index full all
./nnindexer.php index delta all
./nnindexer.php daemon --stop
./nnindexer.php daemon

Summary of Installed Software¶

PHP v5.3.6

Pear v1.9.2

MySQL v5.1.58 or

Percona v5.5.17

Apache v2.2.20 or

Nginx 1.0.5

FFmpeg v0.8.7

MediaInfo v0.7.50

Lame v3.98.4

unrar v4.00-beta3

Sphinx 2.0.2-beta