Saturday, 19 November 2016

So I now have a Pi powered newsgroup indexer and a Pi powered newsgroup server, time for a Pi powered downloader, I'm going to set this up for Torrents and Usenet use, I've added a USB drive to it and mounted it at:

Following on with my setting up of a Raspberry Pi as a Newsgroup Indexer, I've realised that the next big issue is with news servers themselves. I don't really want a load of newsgroup just a few with a reasonable retention. Some of the paid for servers are suffering from missing posts so I've decided to use a Raspberry Pi with a 500GB drive in it and set up my own. If it works well, I'll eventually add on a bigger drive.

I've installed Raspbian Lite and mounted the external drive at /var/news, one problem is that the drive doesn't seem to mount at boot, this has been fixed by modifying the cmdline.txt file, run:

You can't use localdomain here, so just put your Samba workgroup domain in, then we need to edit another two files so we can connect on the local network:

sudo nano /etc/hosts.allow

#-- leafnode beginleafnode: 127.0.0.1#-- leafnode end

Edit this so it looks like

#-- leafnode beginleafnode: 192.168.0.1#-- leafnode end

Changing the 192.168.0.1 to whatever your IP address range is and then:

sudo nano /etc/hosts.deny

#-- leafnode beginleafnode: ALL#-- leafnode end

Comment out the middle line so it looks like this:

#-- leafnode begin

# leafnode: ALL

#-- leafnode end

Now it's time to fetch a list of newsgroups:

sudo fetchnews -vvv

You can now connect to the server with news reader, I'm going to use Pan but there are loads out there, this is how you do it with Pan

Subscribe to a newsgroup and then go and request the latest headers, this will only give one leafnode entry at the moment, now we want to change some settings before we go any further, just enter:

sudo nano /etc/news/leafnode/config

I'm going to change the expire entry at the top to 400, that's just over a year

Going down the screen I'm changing the initial fetch to 1000, keep going down the screen and set maxfetch to 2000, maxcrosspost to 10 and maxage to 1000. If you're on a slow connection it may be worth you dropping some of these settings

We then run:

sudo fetchnews -vvv

This time it will populate the group(s)

You may find that your current news server doesn't hold all the groups you want so you can edit the leafnode config file again and add some other servers in, here's some you can add in;;

Now run sudo fetchnews -vvv again and it will pull down a list of groups from these servers, make sure that you ask your usenet client to refresh the groups list.

Last thing we have to do is make it update the groups by itself, there is a file in the leafnode folder that you can use but I've done it this way and it works for me:

sudo nano /bin/getnews

Paste the following into it:

texpire -vfetchnews -vvvecho

Let#s make it executable with:

sudo chmod +x /bin/getnews

And we'll get cron to run this every 15 minutes with:

sudo crontab -e

Paste this into it:

*/15 * * * * /bin/getnews

Then save it, give it a final reboot and you're all sorted, you're very own news server. You can't add a username and password for access as far as I'm aware so it's not a good idea to make it accessible from the Interwebs.

If you reboot at some point the lockfile may stop leafnode updating, I've created another file called clearlock with:

sudo /bin/clearlockThis goes into it:

rm /var/spool/news/leaf.node/lock.file Again, we make it executable with:

Sunday, 6 November 2016

A lot of the newsgroup indexing sites are either being shut down or are becoming invite only which is a shame as there's a lot of useful stuff in the groups that isn't only illegal downloads. There's quite a few Linux, freeware and shareware groups out there.

It's time to convert a Raspberry Pi into a usenet indexer, I tried this with an original Pi and it was far too slow but hopefully the model 2 will do the job. All configuration is done from a Linux desktop, if you haven't got an ssh command then you'll need to get something like Putty.

An external hard drive of some sort, I've used an old 320Gb and got hold of a case from Amazon, it's

nothing exciting but it works, I ordered the case on Saturday evening and it arrived on Sunday lunchtime, good going there Amazon.

A USB hub also makes life easier, nice cheap one from Maplin. You're also going to need an ethernet lead to connect it to your router, £1 in Poundworld.

Some coffee and a sausage inna bun will help.

It's time to make a start, put it all together but don't worry about connecting the external drive yet, we need to get hold of Raspbian Jessie Lite from here. Extract it and then copy it to a micro SD card, easiest way is to use a cross platform app called Etcher.

Once it's finished, put the card into the Pi and give it some power. Have a look at the connected devices in the router config and ssh into the pi with ssh pi@192.168.0.253 or whatever yours says, password to connect is raspberry. We now run:

sudo raspi-config

You should see this:

Scroll down to advanced, then down to update, once that's completed go back down to advanced options and put in your own hostname, mine is called Orac as the Pi is in a clear case just like Orac in Blakes 7, under advanced we want to select boot to console requiring user to login and enable ssh. Once last thing under advanced is memory split, set this to 16 to give as much RAM to the running of the Pi as possible. Last thing we do is select Expand Filesystem at the top.

Select finish and wait for a reboot, you should now be able to connect with

ssh pi@orac

or whatever you've called your pi.

Now we create a normal user with

sudo adduser username

then add them to the superuser group with

sudo adduser username sudo

Reboot with

sudo shutdown -r now

Now you should be able to login with ssh orac or ssh username@orac.

Let's remove the default Pi login with

sudo deluser pi

We need some helpful utilities now, so let's enter:

sudo apt-get install rsync mc

And now it's time to connect up the external drive.

Type dmesg at the command line and it should come up as /dev/sda or something similar, we now need to partition it so we enter:

sudo fdisk /dev/sda or whatever yours came up as.

Then D to delete any existing partitions, then N to create a new one, followed by P for primary partition, then enter 3 times followed by w to write the changes. There may be other options here depending on how many partitions there were on the drive when you started.

It's all on one line, we need to change the root=/dev/... bit to read root=/dev/sda1 or whatever your USB stick shows, we also need to add a delay to the end with rootdelay=5, it should look like the one below in the end.

And the line referring to the sd card needs to be commented out, mine looks like this now

proc /proc proc defaults 0 0

/dev/mmcblk0p1 /boot vfat defaults 0 2

# /dev/mmcblk0p2 / ext4 defaults,noatime 0 1

/dev/sda1 / ext4 defaults,noatime 0 1

We now should be running from the external drive, type df -h to check the space, so now we move onto getting Newznab installed, there's a few utilities we need that don't seem to be in the raspbian repository, I've precompiled these so you can get them with these commands:

Make sure you put in a good strong mysql passqword if you're going to get access to this from the Interwebs.

Now we make some changes to the php configuration file:

sudo nano /etc/php5/cli/php.ini

Under the resource limits section we need to change the maximum execution time to 120:

max_execution_time = 120

Then under Module Settings we need to change the time zone settings, in my case I'm the UK so I'm going to change it to:

date,timezone = Europe/London

Save that and then time to edit another one:

sudo nano /etc/php5/apache2/php.ini

Under the resource limits section we need to change the maximum execution time to 120:

max_execution_time = 120

and

memory_limit = -1

Then under Module Settings we need to change the time zone settings, in my case I'm the UK so I'm going to change it to:

date,timezone = Europe/London

Now we create the Apache config file for Newznab

sudo nano /etc/apache2/sites-available/newznab.conf

And paste the following into it:

<VirtualHost *:80>

ServerAdmin webmaster@localhost

ServerName localhost

DocumentRoot /var/www/newznab/www

ErrorLog /var/log/apache2/error.log

LogLevel warn

</VirtualHost>

Change the port if you want it to run on a different port.

One more change to the Apache configuration file before we go on:

sudo nano /etc/apache2/apache2.conf

Find this bit

<Directory /var/www/>

Options Indexes FollowSymLinks

AllowOverride None

Require all granted

</Directory>

And change it to this:

<Directory /var/www/>

Options Indexes FollowSymLinks

AllowOverride All

Require all granted

</Directory>

We now need to get hold of Newznab itself, you can get it from here. Newznab classic works but there are some limitations, if you are serious just pay the money for a better product and you get free upgrades too.

Get the file downloaded and then we need to extract it and copy it to the right place,

As I have a paid for version I'm going to use a slightly different approach, for this you need to install subversion with the following command:

sudo apt-get install subversion

Then to get Newznab you enter the following command:

sudo svn co svn://svn.newznab.com/nn/branches/nnplus /var/www/newznab

At this point you will be asked for the root password for your system and the SVN login details that have been emailed to you by the Newznab team.

Now it's time to activate the changes:

sudo a2ensite newznab

sudo a2dissite 000-default

sudo a2enmod rewrite

sudo service apache2 restart

Change some folder permissions with:

sudo chmod 777 /var/www/newznab/www/lib/smarty/templates_c

sudo chmod 777 /var/www/newznab/www/covers/movies

sudo chmod 777 /var/www/newznab/www/covers/music

sudo chmod 777 /var/www/newznab/www/covers/tv

sudo chmod 777 /var/www/newznab/www

sudo chmod 777 /var/www/newznab/www/install

sudo chmod 777 /var/www/newznab/nzbfiles/

sudo chmod 777 /var/www/newznab/www/covers/anime

sudo chmod 777 /var/www/newznab/www/covers/tv

And that's it for the moment, you now need to finish setting up Newznab from a browser with:

http://servername/install

Once you've done all that, run

sudo chmod 0777 /var/www/newznab/nzbfiles/tmpunrar

And install phpmyadmin with:

sudo apt-get install phpmyadmin

Go to the site admin options and edit site, it works much better if you put in your own Rotten Tomatoes api, your own Amazon settings and your own Newznab ID, the section that asks for the path to some files should show this:

/usr/bin/unrar

/usr/bin/mediainfo

/usr/bin/ffmpeg

/usr/bin/lame

Once that's all completed, go to the View Groups option and make some groups active.

Then it's back to the command line with

sudo -i

cd /var/www/newznab/misc/update_scripts

Then run php update_binaries_threaded.php

after that it's

php update_releases.php

That should start populating the indexer, it's up to you how you want to run this, you can it manually but I run a loop with a 10 minute pause each time it restarts.

I've made a file with

sudo nano /bin/newznabrun

Put this into it

while :

do

# Beginning Update

cd /var/www/newznab/misc/update_scripts/

php update_binaries_threaded.php

php update_releases.php

echo "Press [CTRL+C] to stop.."

sleep 360

done

Save the file and then run

sudo chmod +x /bin/newznabrun

You can then type sudo newznabrun to get it going.

That's it all done, hope it all works for you.

Big thanks to the howtogeek website for help with these instructions, I've added a few bits and pieces to get to this stage.

Sunday, 10 July 2016

I'm going to admit that I actually like the Unity desktop, there, I feel better now. I've used Ubuntu 14.04 for a while but as it gets older and you add more and more ppas to keep the apps up to date it's become slower and less stable, the latest issue has been after a load of updates I rebooted into a 1024x768 screen, easily sorted by installing the AMD drivers.

I decided once more to try and install 16.04 or one of the Mint derivatives, each one gave me the same issue, just a blank screen when trying to install. I wasn't going to give up, during install I pressed the down arrow key when the initial screen appeared, select UK then F6 and selected the nomodeset option. I could now install Ubuntu 16.04. I rebooted and got my lovely 1024x768 resolution, the fix to get this sorted was to modify the grub file to give me a 1440x900 screen with slow software video drivers. I looked up the installation of the AMD drivers, followed the instrctions, rebooted and it died in the most spectacular fashion with lots of lovely dots and so on.

I've given up, this was what Windows 3.1 was like trying to get the right resolution, it shouldn't need this much messing around. I decided to abandon Ubuntu and booted up Fedora Mate, no bottom or top bar visible on Fedora 24 although it worked on 23, the only other distro I had on my pendrive was PCLinuxOS, the latest preview version, it's pretty much a beta. I booted this, everything was there, I installed it, everything worked, it had automatically installed the right drivers for me. Even the bluetooth settings worked, on Ubuntu 14.04 you had to manually change the device name by editing a file. The most impressive thing is that the num lock worked and was turned on when I powered up, I didn't have to do this manually every time I started up the machine.

There's a few little quirks, you need to go into the Configure Your Computer option and select the Setup a network interface option to set the hostname, this requires a reboot.

You also need to go into the software centre and run the localisation utility to change your language, by default it's US, another reboot is needed, there's no sudo but this makes the machine more secure.

Installing Virtual Box is a breeze, you click on the Virtual Box Manager link, it installs the non free version so USB works, no messing round, it just works.

The other good thing is that it's a rolling release, no need to re-install, you download the updates and you have the latest version.

I used to use PCLinuxOS when Mandriva became to buggy, it never really let me down and it hasn't again, it uses rpm files but synaptic to manage them, if you want to get into Linux, try it.

Tuesday, 21 June 2016

It was time to setup a way of getting secure access to my network while away and also to get a secure connection to the Interwebs while away from home, it also means that I can watch BBC Iplayer as if I'm at home.I'm installing this on an old machine running Debian Jessie, it should work on a Raspberry Pi too, the only things I have installed during the initial setup process as the standard Debian utilities and ssh server so I can do everything remotely.I've found lots on instructions out there but the one from this website was the easiest to follow, I've modified it slightly to make it easier to get at the keys.I've modified a few things myselfFirst thing is to ensure we are up to date, lets switch to root for the installsuthenapt-get updateapt-get upgradeTime to start installing stuffapt-get install openvpn easy-rsa

Then copy some example files over to make the job easier

cp -r /usr/share/easy-rsa/ /etc/openvpn

mkdir /etc/openvpn/easy-rsa/keys

Now we edit the certificate variables

nano /etc/openvpn/easy-rsa/vars

# These are the default values for fields

# which will be placed in the certificate.

# Don't leave any of these fields blank.

export KEY_COUNTRY="changeme"

export KEY_PROVINCE="changeme"

export KEY_CITY="changeme"

export KEY_ORG="example"

export KEY_EMAIL="changeme@example.com"

export KEY_OU="changeme"

# X509 Subject Field

export KEY_NAME="server"

Time to generate some stuff and go and have a coffee, on a Pi, this may take some time

openssl dhparam -out /etc/openvpn/dh2048.pem 2048

Now we make the server certificate keys:

cd /etc/openvpn/easy-rsa. ./vars./clean-all./build-ca./build-key-server serverLet's copy them to where they belongcp /etc/openvpn/easy-rsa/keys/{server.crt,server.key,ca.crt} /etc/openvpn

Now time to make some changes to the network settings:

echo 1 > /proc/sys/net/ipv4/ip_forward

And let's make the changes permanent with:

nano /etc/sysctl.conf

Look for the following bit:# Uncomment the next line to enable packet forwarding for IPv4# net.ipv4.ip_forward=1

Then remove the # from the second line so it looks like this:# Uncomment the next line to enable packet forwarding for IPv4net.ipv4.ip_forward=1Now we make the server config file:nano /etc/openvpn/server.confPaste this lot into the empty file, this will run the VPN server on port 1194.port 1194proto udpdev tunca ca.crtcert server.crtkey server.key dh dh2048.pemserver 10.90.10.0 255.255.255.0ifconfig-pool-persist ipp.txtpush "redirect-gateway def1 bypass-dhcp"push "dhcp-option DNS 8.8.8.8"client-to-clientduplicate-cnkeepalive 10 120cipher AES-128-CBCcomp-lzouser nobodygroup nogrouppersist-keypersist-tunstatus logs/status.loglog-append logs/openvpn.logverb 3

Now we make the log files:mkdir -p /etc/openvpn/logstouch /etc/openvpn/logs/{openvpn,status}.logAnd let's do some firewall configuration:iptables -t nat -A POSTROUTING -s 10.90.10.0/24 -o eth0 -j MASQUERADEiptables-saveNow let's restart the server to put the changes into place:systemctl restart openvpn@server.serviceNow the original instructions came with a script file to help you create new keys for each user and device, pointless changing it.nano /etc/openvpn/gen-client.sh

Save it and then make it executable with:chmod +x /etc/openvpn/gen-client.shNext we have to create the template file for this to use:mkdir -p /etc/openvpn/clients/.tmp/nano /etc/openvpn/clients/.tmp/.tmp.ovpnPaste this in, change example.com for your external IP or server addressclientverb 1dev tunproto udpport 1194remote example.com 1194 udpremote-cert-tls serverresolv-retry infinitenobindpersist-keypersist-tuncomp-lzocipher AES-128-CBCNow, let's make some keys:cd /etc/openvpn/

replace username with your username, I'm going to install this onto an S5 so it will be freds5 or something.

./gen-client.sh username

To make it easy to get the files off the server and onto my device, I've decided to install Samba and setup the client keys folder as a Windows share, this is how this is done.apt-get install samba samba-commonOnce it's finished we edit the Samba config file:nano /etc/samba/smb.confChange the workgroup name at the top of the file and you can also add:netbios name = servername under it if you want.Add the bottom add the following[VPNKeys] path = /etc/openvpn/clients/ browseable = yes public = yes writeable = no

Restart the server with:

service smbd restart

Just got to change the folder permissions to make sure we can get the files off:

chmod 0777 -R /etc/openvpn/clients

Onto my phone now, I've installed OpenVPN Connect from the play store, then I've copied the files from the Windows share into dropbox, then saved them into a folder on the phone called VPN, you could just install a file browser and do the same.

Then import the .ovpn file into OpenVPN connect and click on connect.One last thing, make sure you give the server a static IP and forward port 1194 on the router.