Rsync over ssh - Quick Reference

This is for me - I always forget how to upload and download from a server with rsync over ssh.

Reading man files sucks.. Simple use cases are painful to piece together within the homogenous help files of ultimate doom.

Uploading…

rsync -avhz -e ssh { relative local path } {SSH CONFIG NAME}:{ SERVER PATH }/

Make sure you have the relevant server name matching in your ~/.ssh/config and you’re all set.

Dogeify: Bringing joy to your web with pictures of dogs.

For those of you unfamiliar with Dogecoins and the ‘Doge’ meme in general just click on the cute dog or drag it to your bookmarks bar if you want to have the ability to add cute dogs to any site you want.

So I thought the world could do with more Doge…

Click here for MOAR DOGE

I took a little time out to write some javascript to celebrate One Byte Too Many’s official support of Dogecoin. The script is called Dogeify.js and is available completely open source and free over at github.

It doesn’t do much, but it adds a cute Shiba Inus dog with a random head tilt as well as some general awesomeness to all your site links. If I get much interest in the project I will continue to add to it.

How to use this on other sites.

The dog picture above is actually a ‘live bookmark’. You may drag this picture up to your bookmarks bar and then henceforth you will have ‘Dogeify abilities’ on any future site you visit. Simply click your new awesome bookmark and the Shibe-dog will appear and spread happiness to all corners of your life.


Arch Linux Sysadmin: Pacman: Fix for 'invalid or corrupted package (PGP signature)'

Hi :)

I often get this error when doing a pacman -Syu and I always forget how easy it is to fix. In the past i have resorted to messing with pacman.conf and removing the PGP key checks entirely but this is unnecessary.

Simply run the following command to update your keys and you should be set.

sudo pacman-key --refresh-keys

Hope that helps.. as always feel free to send a few DOGE/BTC my way and remember, if you are in the Cambridgeshire region you can get full linux support here - Linux Support Services in Cambridge

Arch Linux Network Manager Activation failed - fix

I was getting a problem that whenever I activated NetworkManager my ethernet card would drop its ip address. Doing a sudo systemctl status NetworkManager would give me the following output.

Jan 28 11:11:19 desiced.cedeon.local NetworkManager[2317]: <info> Activation (enp3s0) Stage 4 of 5 (IPv6 Configure Timeout) scheduled...
Jan 28 11:11:19 desiced.cedeon.local NetworkManager[2317]: <info> Activation (enp3s0) Stage 4 of 5 (IPv6 Configure Timeout) started...
Jan 28 11:11:19 desiced.cedeon.local NetworkManager[2317]: <info> (enp3s0): device state change: ip-config -> failed (reason 'ip-config-unavailable') [70 120 5]
Jan 28 11:11:19 desiced.cedeon.local NetworkManager[2317]: <info> NetworkManager state is now DISCONNECTED
Jan 28 11:11:19 desiced.cedeon.local NetworkManager[2317]: <info> Marking connection 'Wired connection 1' invalid.
Jan 28 11:11:19 desiced.cedeon.local NetworkManager[2317]: <warn> Activation (enp3s0) failed for connection 'Wired connection 1'
Jan 28 11:11:19 desiced.cedeon.local NetworkManager[2317]: <info> Activation (enp3s0) Stage 4 of 5 (IPv6 Configure Timeout) complete.
Jan 28 11:11:19 desiced.cedeon.local NetworkManager[2317]: <info> (enp3s0): device state change: failed -> disconnected (reason 'none') [120 30 0]
Jan 28 11:11:19 desiced.cedeon.local NetworkManager[2317]: <info> (enp3s0): deactivating device (reason 'none') [0]

Not very informative.

After much digging around I finally found the solution. It seems I had the dhcpcd.service enabled.

`sudo systemctl stop dhcpcd.service`

Fixed the issue. :)

XBMC: Certain files play fast with no sound (AC3 audio codec with pulseaudio passthrough)

I was having an issue with certain media files which would play double speed with no sound. After much searching around it appears that only files with AC3 audio codec were affected. I confirmed that this was the case for me too.

The problem lies with the ‘passthrough’ audio device in XBMC. Mine was set to PulseAudio (default). After much tweaking around and research it appears that there is some kind of problem with Pulseaudio not passing through the AC3 stream correctly.

The Fix

This problem was remedied by killing pulseaudio and using the ALSA fallback.

1) Edit /etc/pulse/client.conf

autospawn = no
daemon-binary = /usr/bin/true

2) Have a look in the following paths and remove pulseaudio files to prevent autoloading by others.

/etc/X11/xinit/xinitrc.d/pulseaudio
/etc/xdg/autostart/pulseaudio.desktop
/etc/xdg/autostart/pulseaudio-kde.desktop

~/.config/autostart/pulseaudio.desktop

References

Arch Wiki - Always the best source


Gnome: Nautilus: How to hide partitions from the 'devices' area on the left pane

I have a lot of partitions such as windows system partitions that I never use from Linux yet it always annoyed me that they were showing up automatically in nautilus and if I happened to click on them accidentally I would get a mount dialog that I could not cancel.

Solution - add udev rules to ignore them and reboot.

Heres a copy of my file, modify yours accordingly…

% cat /etc/udev/rules.d/99-hide-disks.rules

ACTION!="add|change", GOTO="hide_partitions_end"
SUBSYSTEM!="block", GOTO="hide_partitions_end"
KERNEL=="loop*|ram*", GOTO="hide_partitions_end"

KERNEL=="sda1", ENV{UDISKS_IGNORE}="1"
KERNEL=="sda5", ENV{UDISKS_IGNORE}="1"
KERNEL=="sda6", ENV{UDISKS_IGNORE}="1"
KERNEL=="sda8", ENV{UDISKS_IGNORE}="1"

LABEL="hide_partitions_end"

Child Safety: How to sandbox your children's web traffic cheaply using a Raspberry Pi

Forget the rubbish (ISP Porn Block) proposal put forward by our P.M.! You can create a cheaper, more robust, secure and granular solution for less than £30.00 (+ some network equipment you may or may not have)

Picture of a Raspberry Pi You can pick up one of these for under £30

Heavy Disclaimer Although the Raspberry Pi described here is super cheap, you will have to have some slightly specialized network equipment to do this. In particular you will need a VLAN aware network switch and a VLAN aware Wifi Access Point at a minimum. These used to be costly but are getting cheaper. I HIGHLY recommend the:

Q: OK so what exactly is this thing and what can I do with it?

A: Quite simply you will have absolute control of your kids’ network traffic while at the same time enjoying complete freedom of your own. It will be like having two separate internet connections to your home and you can say goodbye to slow downs whenever your kids are on you tube.

It allows you to do many cool things such as:

  • Schedule access times for your children’s internet.
  • Set up a transparent porn/ content filter for your kids internet.
  • Log & capture your childrens network traffic.
  • Monitor speeds & bandwidth usage and set caps on data.
  • Separate the rest of your network from your kids (protecting your home office etc)
  • Block certain sites, filter adverts, and many more things!

All this can be done with free, libre, open source software and a cheap raspberry pi computer that you can get for under £30.00!

My rough network topology

My network topology.  We are only interested in the left hand side.

My network topology. We are only interested in the left hand side.[/caption] Note the left hand side of the above diagram and in particular the dashed blue line. This represents your children’s wireless network connection to the internet. Basically what we will be creating is known as a ‘One Armed Router’. It’s a router with only a single network port, yet it is able to capture traffic at the hardware level using the power of VLANs , and then change the traffic before spitting it out again. This happens in both directions effectively creating a sandbox. VLANs are used by ISP’s and enterprises; they are a way to have two or more networks sharing the same physical wires. The white circular object shown above is the Ubiquiti Unifi Wifi access point I described above. It allows you to create multiple separate wifi connections which is key because you can have a separate password for each and then give your kids their own wifi connection. All their traffic becomes the blue dashed line shown above and is sent to the Raspberry Pi by the switch yet your own green traffic passes freely through. You can even have your wifi connection completely hidden so that only the sandboxed connection shows up, this is also great for guests & strangers who want to use your internet because all your green network including your LAN (in my case) is completely separated

Q: OK I’m ready to do this. How do we get started? A: First of all you will need the following:-

  • A Raspberry Pi (case optional)
  • A spare SD card (at least 4GB)
  • About 90mins of time (+ some download time)
  • Some VLAN capable network hardware. (Check your router/network gear. Admittedly at the time of writing these are fairly rare, especially as most people stick with the BT Home Hub/router that your ISP has provided which mostly suck. If you don’t have these things then please scroll to the bottom where I will try to convince you to invest in some top gear and if you live in the south east of england I will even come and install it for you. Just head over to my business website https://www.onebytetoomany.co.uk )

1) Install Arch Linux onto the SD Card Instructions for this can be found here :- http://archlinuxarm.org/platforms/armv6/raspberry-pi

2) Boot up and Log in to your Raspberry Pi as root, either physically or via SSH. (If you use windows you can download a tool called PuTTY http://www.chiark.greenend.org.uk/~sgtatham/putty/ )

3) Set up your raspberry pi for VLAN.

$ cd /etc/netctl
$ cp examples/vlan-static .
$ cp examples/ethernet-dhcp .
$ (vi or nano) vlan-static

# [edit the file to look like this ]
Description='Virtual LAN 32 on interface eth0'
Interface=eth0.32
Connection=vlan
BindsToInterfaces=eth0
VLANID=32
Address="10.13.37.1/24"
Gateway="10.13.37.1"
DNS=("10.13.37.1")
ExecUpPost=("route del default dev eth0.32")
Hostname="RpisRCool"

$ (vi or nano) ethernet-dhcp
# [edit the file to look like this ]
Description='A basic dhcp ethernet connection'
Interface=eth0
Connection=ethernet
IP=dhcp

$ netctl enable ethernet-dhcp
$ netctl enable vlan-static

$ netctl start ethernet-dhcp
$ netctl start vlan-static

4) Enable forwarding of packets between the two virtual network cards we just set up on the Pi

$ echo net.ipv4.ip_forward = 1 &gt;&gt; /etc/sysctl.conf
$ echo 1 >> /proc/sys/net/ipv4/ip_forward # so we don't have to restart.

5) Set up the iptables firewall to act as a Network Address Translation (NAT) router between the two virtual network cards.

$ cd
$ touch onearm.sh
$ vi onearm.sh
# [make a file like this (we can expand it later but for now this will do]
ETH="eth0.32"
WAN="eth0"
SUBNET_LOCAL="10.13.37.0/24"

function clean {
iptables -F
iptables -t nat -F
}

function onearm {
iptables -t nat -A POSTROUTING -o $WAN -j MASQUERADE
}

clean
onearm

# [EXIT & SAVE the file ]
:wq (for vi)

$ ./onearm.sh # run our script
$ iptables-save > /etc/iptables/iptables.rules

6) Set up your hardware to VLAN tag the port your Pi is plugged into as VLAN ID = 32.

Refer to your manual. If you are using a Ubiquiti Toughswitch then you simply log into the switch, go to the VLANs tab and set the port to ‘T’ and the VLAN ID to 32 or whatever you chose your VLAN ID to be.

7) Set up your Wifi access point and create a separate SSID & Network on VLAN 32 (or whatever you chose as your tag).

On the Ubiquiti Unifi this can be done through the management interface, simply create a network and set it’s VLAN ID to be the same. Make sure the port on the switch is similarly set up as the Pi with tag enabled on the port.

8) Enable a DHCP Server on your VLAN’d Raspberry Pi interface so that the kids get given an IP address from their sandboxed network (10.37.37.0).

$ vi /etc/dnsmasq.conf

interface=eth0.32
domain=somedomainofmychoosing.local,10.37.37.0/24
dhcp-range=10.13.37.30,10.13.37.60,12h
:wq

$ systemctl enable dnsmasq
$ systemctl start dnsmasq

Thats it!. You now have a sandboxed connection running through a separate network (10.13.37) and your kids have their own network on 10.13.37.0 which gets ‘N.A.T.’ted through the Raspberry Pi onto the normal network which then gets N.A.T.ted again and out onto the public internet. Double firewall.

Drop me a mention on twitter @cedeon if you need any help!.

Now you have the basic functionality set up you can do lots of cool things with it. I’ll leave it up to your imagination but here are a few ideas:

  • Set up a cron job to turn off eth0.32 at night and then turn it back on again in the morning.
  • Install a traffic analysis tool like IPTraf ( http://iptraf.seul.org/ ) and have a look at whats going on with your kids online.
  • Install a content filter like safesquid / OpenDNS / squidguard
  • Install ‘snort’ intrusion tool
  • Make some advanced firewall rules & traffic shaping rules.

Q: I’ve looked into doing something similar before but lots of people say that the Raspberry Pi is useless and too slow to be a router, besides don’t I have to get a second network adapter for it?

A: No and No! The raspberry pi is absolutely perfect for this because it has more than adequate speed to handle current broadband internet speeds that your kids require. My set up doesn’t use a second network adapter (normally you need a second USB NIC plugged in to the Pi) There is a USB/Network bottleneck on the Raspberry pi which my version avoids and you will easily get upto 40Mbit/s throughput. The other beautiful thing about the network design is that any slow downs at all will have no affect on the rest of the network. Furthermore you will usually want your kids’ traffic to be slowed down a bit so that you can maintain a high quality of service on your own internet traffic so this is another reason why a raspberry pi works here.

Q: You suck, your title says cheap but then your solution relies on enterprise grade network equipment and VLAN tagging!

A: I’m sorry this is true. I wish it was as simple as plugging a raspberry pi into a BT Home hub. If home routers were better then it would be this simple but that is not the case. There are ways of doing the same thing with a Pi with an extra Network port (NIC) or a device which has two NICs but I haven’t gone into that here. Like hi-fi separates, you do get lots of benefits to buying a separate network switch and wifi access point like the Ubiquity ones I have mentioned at the top of the post. You get the following benefits:

  • You can have a completely separate network & even a publicly accessible network running simultaneously
  • You can reposition your Wifi access point separately from your ADSL or Cable modem and put it more centrally in the house.
  • You get various security benefits and can do advanced security things such as creating black holes, honey pots etc.
  • You get network level virus separation and compartmentalization so that if your kids do silly things and get themselves hacked, the rest of the computers on your network are unaffected.

Basic Linux Sysadmin: Untangling Linux Log Files

Note & TL;DR: This post mainly relates to the new Systemd journal style logs and not the traditional text based logs. Note the Arch Linux logo to the left :), if you are running Debian then this probably won’t apply to you until 2050 :p (if ever if the Upstart folks have their way).

Introduction

I’ve been a Linux user for over a decade now, and I would say it has been my primary go to desktop OS for more than half of that time. I spend a lot of my day on the command line and I believe I would be pidgeon-holed well into ‘Advanced User’ territory. Still, not a day goes by where I don’t learn something new about Unix based systems, which is great. One of my most avoided areas of Linux research has always been the syslog. This is due to few reasons, firstly it’s always looked ‘hacky’ to me and not fun at all. Maybe some people love ‘grepping’ and ‘catting’ and mixing weird and wonderful awk and sed pipelines together, not me. Personally I find it uncomfortable, probably because for some reason my brain is incapable of retaining knowledge about string formatting and regular expressions- I don’t know why because I’m perfectly fine with absorbing numerous programming languages. I seem to get a feeling of dyslexia wash over me as soon as I see too many symbols without ample white space and line breaks in between. The second reason I’ve avoided the dreaded syslog up until now is simply because I’ve never had a need to. I’ve always taken a clumsy approach to system administration- if it doesn’t work simply take a proverbial sledgehammer to the PC and have a do-over :). This has worked fine for me in the past because I have no one to answer to and I can manage my systems however I please. This is all due to change soon with the launch of my new business

One Byte Too Many. Time to learn me some syslog!.

Diving into some logs.

Disclaimer: I’m going to write this chronologically, in the order that I discovered certain things. This may not be compatible with other brains! If so I apologize for that.

1. utmp,wtmp,btmp have always annoyed me.. wtf are these!?

I remember blindly running cat on these files and having them smear crap all over my prompt. Weirdly up until now I’ve never bothered to find out what they are. It turns out they are actually quite simple binaries that log user log-ons and log offs. They can all be accessed with the ‘last’ command like this:

last -f /var/log/wtmp

also the ‘who’ command uses these files and is particularly useful. If you want to find out more about who and are as nerdy as me you could run :-

alias doctor="man" && doctor who

Sorry, I couldn’t resist.

2. Moar login logs? srious!?

Yes next we have lastlog and faillog. Both have respective command line tools.

Lastlog is quite simple and the command lists all the users on your system and besides them puts the a timestamp of the last successful log in as well as which tty/pts port and also a nice reverse dns if you logged in over ssh. The other important thing to note about lastlog is that it looks a lot more juicy than it actually is because of it’s size. It is a sparse file and as such will not tend to change in size. This concept threw me a little because on the face of it it looks like its filled with juicy log data.

Faillog is a bit of a wierd one. To quote the opening line of the man page:

“faillog displays the contents of the failure log database (/var/log/faillog). It can also set the failure counters and limits.”

So it seems that the userspace tool is a kind of hybrid log viewer and settings manager. Naturally the userspace tool is the same as the filename ‘faillog’.

3. Journalctl

This is worthy of an entire blog post by itself. It’s a beast that I havn’t as yet been able to tame. I normally just use

journalctl -xn 20

Which gives me the last 20 lines of log files in a descriptive manner. Also:

systemctl status [service name]

will give you the tail logs of the specific systemd service. You can also do

journalctl /name/of/bin/executable

and that will show you what you expect to see.

An aside

I hope you are with me when I say that linux syslogs are a real mess and not at all intuitive. In its present state we have a whole range of logging styles in the same folder from plaintext files with no .log extension, plaintext files with a .log extension, binary files mixed in the same folder, plaintext files in subfolders, rotated and gzipped files in subfolders and base folder. Its a hodgepodge of mayhem that looks like it was designed by sadists. I personally think Lennart Poettering’s hugely criticised journalctl work is the right idea but unfortunately its one of those 1 steps back for 2 forward manoeuvres that we now have to struggle to get through in the meantime. This interim period of having half binary, half plaintext is not easy on my brain and every time I type ‘man journalctl’ I want to take my shovel and thrust it towards my own face. I mean sure.. i guess its intuitive if you precisely know what you want to search for and know systemd like the back of your hand but its in no way nice at first glance. If I get enough feedback i’ll troll through and do a proper write up on it.

DERP!: Diminished Eavesdrop Raspberry Pi by One Byte Too Many

alt text

The Need

Ever since the whistle blower Edward Snowden released chilling information about government spying and collection of massive amounts of our personal data by ‘Five Eyes’ countries (USA, Canada, UK, Australia, and New Zealand) against our will, a lot of us have felt extremely disheartened by our governments’ actions. Many of us believe that privacy is not only a fundamental right but also a benefit for all people, not just the ones that have malicious things to hide. It’s an ironic contradiction that the governments are so opaque about their own surveillance yet feel the people should be of the mindset that if they aren’t doing anything wrong then they have nothing to hide. According to the actions of our governments we should be content with trading our freedoms for increased national security even when this increased level of national security has not been quantified to us.

Whatever your personal opinions to this recent news are, we still do have constitutional rights to protecting the sanctity of privacy and we also have many technologies and tools to help us achieve this, particularly on computers. My aim is to lower the cost of entry to these tools by providing easy to use ‘plug & play’ style internet privacy & encryption devices.

Please note: The following is not a one stop ‘Snake Oil’ anonymity solution! Please learn all you can about the technologies that I have used, none of them are one stop solutions. Nevertheless these technologies can go a long way to help with pulling power back in your own hands and should be treated as one of your many privacy tools.
The Goal

To create a device which helps to mitigate the surveillance capabilities of internet service providers, governments & malicious people in general in order to protect against intrusion of privacy.
The Device

The intention is to build a portable transparent Tor proxy. To quote Wikipedia, “Tor directs Internet traffic through a free, worldwide volunteer network consisting of more than three thousand relays[6] to conceal a user’s location or usage from anyone conducting network surveillance or traffic analysis“

My device will encapsulate this technology in a black box, easy to use form.

It will be comprized of the following parts.

  • Raspberry Pi Type B.
  • Black Case
  • USB Wifi Dongle
  • SD/Micro SD Card pre-loaded & configured with custom software
  • (Optional) Ethernet & Power Cables, which complete the prerequisites needed for plug and play.
  • (Optional) Real Time clock add on.

All parts to be assembled and ready to go out of the box.

Features

  • Must be easy to use, portable, & zero low configuration
  • Provides a complete Wifi Hardware solution, i.e. can be used as a Wifi dongle.
  • Device must be completely independently verifiable by using Free and Open Source Software throughout.
  • Device must fail closed i.e. when Tor fails, no traffic gets through at all. (security feature)
  • Device must be capable of updating itself.
  • I must share all future add ons (shown below) to the project with everyone that owns a device, free of charge.

Future Goals & add ons.

  • VPN Connection to this proxy so that it can be placed in a large LAN environment and still JustWork™
  • Non persistant logging on the Rpi. (Maybe mount var as ramfs, or kill logging completely). Same for DNS caching.
  • Html web app front end configuration – for ease.
  • LXC Containment or DOCKER containers with full merkle tree hash checking for self analysis for the mega paranoid.
  • Rapid re-provisioning from scratch with provisioning tools like Puppet. This will provide a complete automatic update solution

Wait!… This sounds familiar!

Yes this is true :( .Technically Adafruit Industries have created a similar product with their Onion Pi . I’m claiming prior art as I had a similar idea when I first got my Rpi pokes tongue

Regardless of originality my product differs in the following ways:-

  1. Pre-configured out of the box. My product comes with everything you need pre-installed and ready to use with one command on an SSH terminal.

  2. Bespoke open source Tor monitor and set up software. I have written custom software to set up and monitor the connection with as much ease of use as possible to make this product newbie friendly.

  3. Free updates forever. All my work is in a Git repository and can be provisioned to your box automatically through Puppet. All of this is open source and as the project evolves you will always be able to pull down the latest updates and features.

  4. Network configuration.
    My product is set up to be a DHCP server on the ethernet port passing traffic through a firewall to a wifi client on the wifi port. The Onion Pi is a software wifi access point and SOCKS5 proxy. My product does not require SOCKS proxy enabled software, nor does it create a wifi hotspot. My product is designed to be a personal device that you would take with you two a public wifi hotspot and therefore it only allows a single tor connection through the ethernet port. This is just how it’s setup although all the hardware is the pretty much the same as the Onion Pi so you can certainly run it as a wifi access point if you wish.

If you have any grievances please aim them here. All my work is copyleft or completely free so I hope the work i’ve done on this gets used for greater good elsewhere.

How it works

Diagram of the D.E.R.P.

Diagram of Derp

This diagram shows how the DERP re-routes internet traffic through it’s transparent Tor Proxy.

The DERP works as follows. You plug an ethernet cable into the ethernet port and attach the other end to your computer. The DERP then gives your computer an IP address and sets itself up as a gateway. Traffic which then comes from your computer via the ethernet port will then pass by the DERP’s internal firewall and get re-routed to ports 9040 and 9053 where the Tor service is waiting and listening. The tor service then ‘Torify’s’ the data and routes it through the Tor network. This is all done completely transparently. You can check your new IP address by visiting a service like http://icanhazip.com or http://icanhazptr.com if you want your reverse DNS.

I want one.

Thats great! All the money I raise gets put back into the project and will go towards the development of future features from this list.