Disabling the built in bluetooth and use a USB adaptor instead (on Linux)

My laptop has an Intel wireless card for wifi, and an Intel Bluetooth adaptor too. Hardware wise I suspect there is just one hardware device, and this causes problems on Linux. Although I can scan for devices and pair with some of them, connections rarely work, and my Bluetooth headset does not connect and work most of the time. Apparently this is a known problem with the Intel driver.

I suspected the issue was entirely related to this combination device, so I bought a plug-in USB Bluetooth adaptor. This is where the next problem began. Although the underlying Bluetooth stack on Linux supports multiple controllers, none of the graphical configuration tools do. GNOME’s Bluetooth support, blueman, blueberry… they all are written to only work with one controller. Whatever I did, they only saw the Intel controller.

You can select the default Bluetooth controller via the bluetoothctl command line tool, but this still doesn’t affect the graphical tools. If you disable the Intel Bluetooth controller via the same tool – or via the graphical tools – then all of them insist Bluetooth is turned off whilst gleefully ignoring the second Bluetooth controller.

You also can’t just blacklist a kernel module, such as btintel because this seemingly disables the btusb module too, and the USB adaptor doesn’t work either. Trying to use rfkill won’t likely get you anywhere either.

There is however a solution! We can use good ‘ol udev rules to remove authorisation from a particular device. First of all we need to work out how to specify what USB device we want to disable. We need to find the vendor ID and the product ID. One method for this is lsusb -v. There is a lot of output from this, but you’ll want to find the built in device (in my case, Bus 001 Device 003: ID 8087:0a2b Intel Corp.). Underneath this it will have some information:

Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass          224 Wireless
  bDeviceSubClass         1 Radio Frequency
  bDeviceProtocol         1 Bluetooth
  bMaxPacketSize0        64
  idVendor           0x8087 Intel Corp.
  idProduct          0x0a2b 

The lines we’re interested in are idVendor and idProduct. In my case the values I took from this output were 8087 for the vendor (Intel) and 0a2b (the product ID). As an aside, I love how Intel’s vendor ID is a reference to the Intel 8087 co-processor! (Apparently the Intel PCI ID is 8086 too!).

Another approach to find the information is to use udevadm, like so:

udevadm info -a -p /sys/class/bluetooth/hci0

This also outputs a lot of information. You want to find lines like:


If there are multiple entries you want to choose the entries near the top as the udevadm command walks up the device tree showing information for all parent devices (which we don’t want).

Once you know the product ID and the vendor ID we can now create a rules file and disable the device. Create a new rules file at /etc/udev/rules.d/81-bluetooth-hci.rules and add the following:

SUBSYSTEM=="usb", ATTRS{idVendor}=="your-vendor-id-here", ATTRS{idProduct}=="your-product-id-here", ATTR{authorized}="0"

Here is mine for example:

SUBSYSTEM=="usb", ATTRS{idVendor}=="8087", ATTRS{idProduct}=="0a2b", ATTR{authorized}="0"

Now all you need to do is reboot! Once you do the built in Bluetooth controller should vanish and all of the graphical tools should work nicely with your Bluetooth USB adaptor instead. ūüėÄ

Why I now prefer Arch Linux

Yesterday I installed Linux onto a fresh laptop and there was no debate in my mind what Linux distribution I would use – Arch. A year ago that would not have been the case, I might even have considered installing RHEL instead. So why Arch?

When I was younger – and still today – I noticed that most people I spoke to in the IT profession who had reached their fourth or fifth decade of life had given up on new ideas or new technologies and had settled into what they knew best. Although I respected this, I didn’t enjoy the thought that maybe one day I’d become the same way, even though I felt that my openness to new ideas and technologies was one of my strengths, and something I deeply enjoyed.

Well, I’m in my third decade of life now, and I have definitely begun to notice how easy it can be to settle down for what is comfortable and stick with it. Thankfully I haven’t lost my passion for new ideas and trying new things. Last year at Southampton we hired an intern, this in of itself is a tribute to my manager for changing the norm and doing something new because nobody had ever done that before. It was an eye opening experience, not just because it was enjoyable to teach, but because the internship taught us far more than we realised it would. It challenged us. It made us think differently. The whole team improved. What I didn’t expect was to change my preferred desktop operating system – but, it did. He (the intern) had an established like for Arch linux, advocated it when asked why, and left it at that. He didn’t push me to try it – but it made me want to try it. And so I did.

For a typical end user, Arch Linux is next to impossible to install. It doesn’t have an installer at all. The bootable image you download starts Linux running and then drops you at a zsh prompt. From there its entirely up to you on what you do next. The installation documentation is clear, concise, but not detailed – you need to know what you’re doing to get anywhere – but this, for me, is perfect. It gets out of the way, gives me the tools to install, and then lets me get on with it. I loved it.

The initial experience made me feel like a young teenager installing Linux for the first time. It literally made me feel young again. I felt like I was learning Linux again. Although I had used all these tools before, I had become used to an installer doing it all for me, and I enjoyed learning things I didn’t know about tools I’d used for years.

Arch Linux is minimalist, it is simple. It gets out of your way. You have total control over what your system looks like. It doesn’t hide details from you so you learn why things are done the way they are. At each stage the documentation lets you decide how you want to build your system and gives you information on why you might want to pick any particular solution.

It is easy to build a desktop linux installation. If you want GNOME, on Wayland, with all the bells and whistles, its easy. If you want Xorg and fluxbox, this is very easy too. If you want to use GRUB as your bootloader, you can, but if you want to use systemd-boot (as I do), you can do that too! Each option is easy to achieve, and isn’t hidden away. On Debian I wanted to switch to systemd-boot, but it was far more difficult than expected, and I gave up and just left it with GRUB. On Arch there is support for many different options, and they’re all very easy. You don’t have to accept what the distribution wants you to use.

The other fantastic feature of Arch Linux though is that it seems to solve the classic divide of Linux distributions: new software vs stable software. If I were to pick RHEL as my distribution it would be stable, sure, but the software is old, and most modern apps can’t be easily installed. Just getting flatpak on RHEL is next to impossible. I could alternatively pick Fedora, but then I have to go through a huge upgrade every six months and often things break.

Arch is a hybrid of these approaches, in that it is a¬†rolling release distribution. I have access to all the latest stable versions of all the software I want, but there is no ‘big bang’ forced upgrade every six months. I update when I want to, and Arch updates individual packages as they reach a stable point based on the application itself, rather than an arbitrary time deadline. With RHEL software is considered stable just because it is of a certain age – with Arch, software is considered stable when the developers have declared it to be so based on their thorough understanding of their own releases. You might think that Arch would be somewhat like Debian unstable – frequent bugs and breakages from updates – but in fact Arch seems to hit the sweet spot of stability¬†and modernity.

Pretty much my only criticism of Arch is the lack of a standard AUR installation tool in the base platform. AUR refers to ‘arch user repository’ – add-on software packages that aren’t maintained by the core team. It is heavily used by end users though, and there are many front end tools to make it easy to install AUR packages. Without one of these tools users have to clone a git repository and build packages themselves. With a frontend, such as pacaur (which is what I used), its as simple as using the standard package manager.

So, if you like Linux, and use it on the desktop, and want a better experience: go give Arch a try. You might just like it ūüôā

bargate 1.6

bargate, an open source web interface to SMB file servers,¬†hasn’t had many new features recently. It is now very stable/reliable and there is little need for any user-visible features. I do however plan a number of changes for v1.6:

Add pysmb support for better SMB2 performance

bargate uses the pysmbc library to talk to SMB file servers. This is a very thin wrapper around Samba’s libsmbclient – the defacto standard open source SMB client. Sadly when using SMB2 or later performance is terrible. The trouble seems to be in the stat() call, here is a comparison:

When using SMB2, pysmbc/libsmbclient performance is so bad that it is unusable. Until now this hasn’t really been a problem – everything supported SMB1 still, and Samba still doesn’t enable SMB2 by default. Since the Windows wannacry vulnerability though many folks are just turning SMB1 off altogether (although there really isn’t a good reason for doing so).

I haven’t made any progress with the Samba team in identifying why using SMB2+ is so much slower, so I’ve decided I will make the ‘backend’ of bargate modular. You will be able to choose which library to use – either pysmbc (libsmbclient) or pysmb, which is an alternative pure-python SMB1/2 implementation. Unlike pysmbc, the pysmb library does not suffer performance issues when using SMB2.

Switch to Bootstrap 4

When Bootstrap 4 is released I’ll switch over to using that rather than the current Bootstrap 3, and take the opportunity to re-write much of the HTML to make the pages more efficient.

Add text editing in browser

The final feature I’m planning to add in bargate 1.6 is support for editing files ‘in-browser’ via the codemirror javascript-based text editor. This would allow users to edit text documents from within bargate, such as plain text, or HTML or programming code.

Attacking racism does not stop racism

This year – 2016 – has to be one of the most politically divisive years in recent history. In the UK voters opted to leave the European Union, many clearly hoping to end immigration, and in the USA voters opted to elect Trump, again many hoping to end immigration. In both cases polls before the elections proved to be largely incorrect, and whats more, the principal of the silent majority seems to be the cause.

I’m not going to spend long arguing this point, but it seems clear to me that a lot of people in both countries didn’t admit how they truly felt and so also didn’t admit what they were going to vote for. After the election seemingly legitimatises how they voted they pretty much always say they hate being labelled racists and they feel like their fears and arguments are shut down as being racist. They usually end up by saying this vote means they are the majority and they are in fact not racist at all.

It is clear a majority of these voters feel ignored by the ‘establishment’ and feel that they are not represented. They are correct. The establishment has ignored them. Mainstream political parties don’t know what to do with them.

In both elections the winning slogans have been succinct and¬†obvious: “Take back control” and “Make America great again”. They are anti-progressive rallying calls and the left-wing and centre-ground response is to call people who agree with them ‘a basket of deplorable’s’, ‘racists’, ‘sexists’, ‘homophobes’ and ‘transphobes’. They are correct. These people really are motivated by these emotions.

It is thus both true that these people are ignored and¬†that they hold racist views. The mistake that we make is to demonise these people, to attack them and to call them racist – even if it is in fact true. Unless we’re planning on disenfranchising these people because they are racist then elections will continue to be lost and the “new right” will continue to grow – condemning them makes them more angry and more likely to vote for people like Trump who legitimatise actions which they think will make them feel better.

Recently I fell out with a close friend because his reaction to racism in America was to attack people who happened to be white¬†– calling them racists (“all white people are racist”). Rather than engage with these people, hold civilised debate, he called white people “saltine crackers”, he accused them of having no culture, and he accused them of not having friends who weren’t white. Such a reaction is obviously completely bonkers because it does nothing to actually end racism and discrimination. It only angers people further, and emboldens them to spread their unacceptable views. Its like fighting a house fire by throwing bombs at it.

What we need to do is accept that racism is natural and very human. Instead what happens more often than not is that we assume that racism exists only because white people are ‘inherently’ racist, which is in of itself racist! The irony should be obvious. We should seek to understand racism, accept that its a perfectly normal thing for people to feel, and educate ourselves on how to not take racist actions.

It is not hard to see why we’re all capable of having racist views. Humans evolved to survive – just like every other creature on earth. As such when we interact with people who look or act significantly different to ourselves we react in fear – because that is what kept us alive for the past few million years. Its a perfectly logical response. I will call upon Star Trek to better illustrate my point:

QUARK: You never pulled a stunt like that. You’re smart enough to know that people don’t want to be reminded that you’re different. Who wants to see somebody turn into goo? I hope you don’t do that around Kira.

ODO: Why shouldn’t I?

QUARK: If she’s anything like me, she’d rather you didn’t. Don’t you get it, Odo? We humanoids are a product of millions of years of evolution. Our ancestors learned the hard way that what you don’t know might kill you. They wouldn’t have survived if they hadn’t have jumped back when they encountered a snake coiled in the muck. And now millions of years later, that instinct is still there. It’s genetic. Our tolerance to other lifeforms doesn’t extend beyond the two arm, two leg variety. I hate to break this to you, but when you’re in your natural state, you’re more than our poor old genes can handle.

ODO: So what are you saying, Quark? That the Klingons couldn’t help what they did because of their genes?

QUARK: I’m not trying to excuse what they did. I’m only telling you why it happened.

from Star Trek: Deep Space Nine Season 7 Episode 14 “Chimera”

Now, some people might think Quark and I are justifying racism. What we’re doing is justifying racist feelings not actions.¬†Feeling fear and disliking people that are different from you is¬†normal. What matters is the actions you take based upon those feelings¬†and logical thought. We don’t teach this though. We simply condemn racism, and in the process, confuse and anger most human beings because we’re not making it clear that is is¬†perfectly normal and acceptable to feel fear and dislike about people who look or act differently.¬†What is NOT acceptable is intentionally acting on those feelings.


This is the difference between childhood and adulthood – learning to accept emotions and not just acting directly on them. What we need to do is encourage people to express how they feel and take positive action rather than voting for a narcissistic sexual predator who has no problem saying we should act on our fears¬†because…¬†“we have no choice”.

Don’t be so dramatic! Part 1


So Britain voted to leave the European Union. The leave campaign largely based its campaign on two pillars of “there won’t be an economic impact!” and “immigration must be controlled!”. Essentially the older and less educated people voted Leave after being lied to by the leave campaign. In order to provide evidence to leave voters who even now refuse to admit their mistake I will document on this blog every so often the reality of Brexit.

We’ve had enough of experts

  • On credit ratings:¬†The UK’s credit rating has been changed to ‘Negative’ by Moodys (one of the ‘big three’ credit agencies). “Moody’s said the referendum result would have “negative implications for the country’s medium-term growth outlook”, and it lowered the UK’s long term issuer and debt ratings to “negative” from “stable”.”¬†Source
    Standard and Poor‚Äôs has also warned Britain‚Äôs top ‚ÄúAAA‚ÄĚ credit rating is now at risk. Source
  • On the value of our currency:¬†“Sterling also plunged, falling more than 8% against the dollar and 6% against the euro.” Source
  • On the UK stock market:¬†FTSE100 down 3.12%. “In London the FTSE 250, which mostly comprises companies that trade in the UK, shed 7.2% to close at 16,088 points.” This was the worst slide in history. “That was the biggest daily slide for the index, and equated to ¬£25bn being wiped off the value of its¬†companies,¬†according to the LSE.”¬†Source, Source
  • On the topic of why the FTSE recovered:¬†“”A significant number of FTSE 100 stocks ended the day in positive territory, predominantly those companies with lots of overseas earnings, which stand to benefit from a weaker pound”¬†Source
  • On the lack of housing:¬†“House builders were also the three biggest fallers on the FTSE 100, with Taylor Wimpey suffering a 29% slide.”¬†Source
  • On British banks: “Major UK¬†banks were also badly hit. Lloyds fell 21%, while¬†Barclays and RBS both slid¬†18%.¬†HSBC, which¬†has a large Asian business, fell just¬†1.4%.”¬†Source
  • On the European stock market: “European markets have been well and truly spanked, however, with the Dax in Frankfurt down 6.8% – its worst day since the financial crisis in 2008, the Cac in Paris shed¬†8%, Madrid fell 12%, while¬†Milan takes the wooden spoon with¬†a¬†12.5% plunge.”¬†Source
  • On the US stock market:¬†“Wall Street wobbled further¬†in the last hour of trading in New York, with the¬†Dow Jones ending¬†more than 600 points, or 3.4%, lower¬†at 17,400 points – the biggest one-day fall in almost five years.¬†The S&P 500 fell¬†3.6% – the biggest daily slide in 10 months – while the Nasdaq slumped 4.1%. That was the tech-focused index’s¬†worst day since 2011.” Source
  • On the price of fuek: Prices are likely to rise: ‘Retailers and the AA motoring organisation warned that petrol prices were likely to rise by 2p-3p a litre because of the pound’s fall against the dollar.’ Source
  • On growth: “BBC business correspondent Joe Lynam tells a special edition of Business Live that¬†UBS is predicting UK economic growth will swiftly fall to¬†zero this year.¬†The Swiss¬†bank forecasts that GDP¬†will remain at zero¬†for much of 2017, raising the strong likelihood of a recession, he says.¬†It won’t take much – economic growth slowed to¬†0.4%¬†in the first quarter of the year.” Source
  • On jobs:¬†“Sources within Morgan Stanley have told the BBC that the bank is stepping up a process that¬†could see up to 2,000 of its London-based investment banking staff being relocated to Dublin or Frankfurt.”¬†¬†Source¬†Airbus, which employs thousands in the UK, said: “Britain will suffer” and “Of course we will review our UK investment strategy, like everybody else will.”¬†¬†Source

Its alright though, because: “Andrea Leadsom, a Leave MP, says¬†there “just is not the evidence”¬†of a financial meltdown hitting the UK in the wake of the vote, as predicted by some Remain campaigners.”

Lies, damn lies, and leave campaign lies

  • On spending money on the NHS instead of the EU: Farage admits that the idea of spending the ‘¬£350 million a week’ figure which we ‘send to the EU’ (which was proven to be a lie before the vote) will not be spent on the NHS. Source.
  • On immigration and the free movement of people: “Meanwhile, Conservative MEP and Leave campaigner Daniel Hannan told BBC Newsnight he could envisage a situation where the UK had “free movement of labour” and “From earlier on the Big Decision,¬†Conservative MEP and Leave campaigner¬†Dan Hannan¬†said¬†there was no promise to reduce immigration by leaving the European Union.”¬†Source
  • On France moving Calais border checks back to the UK: During the campaign it was suggested by France that they would no longer honour a 2003 deal in Calais. The leave campaign said this was ‘fear mongering’. Not surprisingly within 24 hours of the Brexit vote France said they would indeed end the deal.”The British must take the consequences of their choice,” she said on Friday.¬†Source
  • On stable government: David Cameron claimed he would stay on as Prime Minister even if he lost the referendum, and we were told by Leave campaigners that he should stay on whatever the outcome. The reality: he resigned. Source
  • On the ‘United’ Kingdom:¬†The leave campaign assured us that there would be no second Scottish independence referendum. After Scotland voted to remain overwhelmingly the Scottish government has begun work to hold one. Source
  • On trade deals with the rest of the world:¬†“A White House spokesman said Mr Obama “stands by what he said” about the UK going to “the back of the queue” when it comes to trade deals with the US.” Source

This of course was just the first 24 hours. Uncertainty is the order of the day.

Dear canonical: we don’t want or need ZFS

It is the late 1990s and the computer server world is dominated by enterprise UNIX operating systems – all competing with¬†each other. Windows 2000 is not out yet and Windows NT 4 is essentially a toy that lesser mortals run on their Intel PCs which they laughingly call ‘servers’. Your company has a commercial UNIX and its called Solaris. Your UNIX is very popular and is a leading platform. Your UNIX however has some major deficiencies when it comes to storage.

IRIX – a competing proprietary UNIX – has the fantastic XFS file system which vastly out performs your own file system which is¬†still UFS (“Unix File System” – originally developed in the early 1980s) and doesn’t even have journalling – until Solaris 7 at least (in November 1998). IRIX had XFS baked into it from 1994. IRIX also had a great volume manager – where as Solaris’ ‘SVM’ was generally regarded as¬†terrible and was an add-on product that didn’t appear as part of Solaris itself until Solaris 8 in 2000.

It wasn’t just IRIX that was extraordinarily better in this area. AIX was ahead of both – JFS was released in 1990 and had file system features that were only just recently introduced by Microsoft with ReFS. JFS was a journaled file system – the first¬†ever¬†journalled file system included in an OS – as I mentioned above it took until November 1998 for Sun to catch up. AIX had a “Logical Volume Manager” (LVM) implementation as well, which again was much better than Sun’s SVM.

This disparity between Solaris and other commercial UNIX platforms did not however hold Solaris’s market share back as it perhaps should have. This was because customers using Solaris on big high-end servers would simply not use UFS, especially not between 1998 and 2005. Customers used VxFS instead – a third party file system, but one that was developed originally at AT&T’s Unix labs, one that was the core file system in another proprietary unix – HP-UX – and one that had journalling, was modern, and could actually compete against XFS and JFS. Of course customers had to buy this from Veritas, but this was a small price to pay for a decent file system and volume¬†system (yes, it came with an alternative volume manager too – Veritas Volume Manager).

So eventually Sun realised that storage was exploding in capacity and UFS just wasn’t up to the task. They also realise that VxFS wasn’t likely to be up to the task either, and with the growing threat of Linux and Windows Server a different solution was needed – a file system to fix all the problems with UFS and leap-frog the competition. As a young man I was fortunate to work at Sun Microsystems when this was happening and I got to meet the core ZFS developers and even work in the ZFS ‘development lab’ – I worked in the same building.

Sun had a problem though – they didn’t just need a new file system. Their RAID implementation (entirely in software, Sun servers never had hardware RAID), and volume management implementations also needed to be replaced. So ZFS sought to replace all three of these components at once. Sadly it would take until 2006 for ZFS to be released into production usage on Solaris, and by then the battle for the enterprise operating system was already over. Linux and Windows had won, the commercial UNIXes had lost. Intel had won the hardware war – the commercial UNIX vendors had lost. Perhaps file systems weren’t as important as Sun had thought.

ZFS is a child of the 1990s commercial UNIX systems. It is an incredibly complicated file system that manages the entire storage stack from the disk spindles all the way up to the file system exposed to application. It can manage¬†vast quantities of disk spindles and scale up to 16 exabytes of storage. It is however still very much a product of the 1990s Sun thinking – a file system for large, beefy all-in-one servers running hundreds of applications. The world had however moved on whilst Sun wasn’t watching.

By 2006 the dominant server platform was 1 or 2U Intel server running Linux or Windows 2003 – servers that almost universally shipped with hardware RAID controllers. High-end SAN storage arrays were king in the enterprise and ZFS wasn’t built for them at all – ZFS was designed to manage the disks directly, making it a great platform for building a SAN storage array itself! Except it wasn’t, because ZFS was still designed with a 1990s mindset. It has no clustering support, its a local file system designed for just Solaris to utilise.

The goal of ZFS to allow Solaris to compete and address vast swathes of storage that UFS and the other competing file systems could not. However by 2006 when ZFS was finally released the other file systems had caught up. They had evolved to scale to the available storage. For a short while everybody talked about how Linux needed ZFS, how Mac OS X needed ZFS, and how ZFS could even turn up in Windows. Ten years after ZFS was launched none of those things have turned out to be true.

Even more frustrating for ZFS fans is that today the dominant computing model is virtual machines and containers: lots of relatively small operating system instances utilising relatively small data sets working together. ZFS makes very little sense in this environment.

Proponents of ZFS on Linux and elsewhere said that ZFS was required because it was revolutionary and much better than what Linux had. In some cases this was true, but in the important cases¬†it was not. Linux was then, and still now, mostly run on hardware RAID systems, had fantastic simple and reliable software RAID, a performant and simple volume manager (LVM) and a range of file system choices that scaled to what was available then and now. Linux was gifted both XFS and JFS from Solaris’ rivals – and both of which continued to develop, XFS particularity so.

Linux did lack some features of ZFS – namely efficient snapshots and data checksumming – that were important. Ten years later we can clearly see that these issues did not prevent the adoption of Linux and ZFS did not in any way save Solaris – Solaris is dying, slowly in private, away from the eyes of the press. Linux won, despite not having ZFS (or Dtrace).

So what about today, does Linux needs ZFS? Canonical thinks it does, and thinks ZFS is exciting technology – more exciting than we’ve seen in “a long time”[1]. Except it really isn’t. These are the same arguments we heard 10 years ago and yet ZFS is even less relevant today than it was a decade ago. Canonical¬†tried to justify ZFS with a series of ‘killer’ features:

  • ‘snapshots’
    Linux already has copy on write snapshots via LVM thin provisioned pools, its in production and supported in RHEL. What’s more it supports most¬†Linux file systems – you and choose whichever you like. If you prefer you can dump LVM and use btrfs which supports snapshots in the same way. So no, sorry canonical, this is not a killer feature of ZFS.
  • ‘copy-on-write cloning’
    ZFS clones are just writeable snapshots, it snapshots ZFS and then copies this (via COW) to create a writable clone. Well, shucks, Linux’s LVM supports this as well and has done for years. It also is a COW based system. Oh and btrfs does this too. This isn’t a killer feature of ZFS either.
  • ‘continuous integrity checking against data corruption’
    XFS has metadata-only (non-data) integrity checking too. Btrfs has full data integrity checking against data corruption. So, no, ZFS can’t claim this is a killer feature that others don’t have. It doesn’t matter anyway – this continuous integrity checking means nothing if you’re using ZFS on a hardware RAID controller or against an enterprise (or non-enterprise) storage array. It only works and guarantees anything if you’re letting ZFS manage the spindles directly. This was a product of 1990s thinking about how storage would be attached or baked into Sun’s servers. Besides, when was the last time you got data corruption? What problem is this trying to solve? I’ve never felt that Linux needs this feature, have you? This isn’t a killer feature.
  • ‘automatic repair’
    Whilst it is true that ZFS does not have to run a horrible NTFS-style chkdsk process, or a horrible ext3-style fsck process either, other file systems have progressed in this regard too. XFS has a similar automatic repair function, doesn’t ever run fsck at boot (there is no XFS fsck!), and does have an xfs_repair tool that¬†nobody ever has to use. Its also worth pointing out that ZFS does have to have non-automatic repairs sometimes, in fact, I’ve had to do it a lot when running ZFS in production. ZFS scrub’s are…not fun, ridiculously slow and can lose files just like any file system does. I found this in production multiple times. Oh, and btrfs¬†supports ‘automatic repair’ too. This isn’t a killer feature.
  • ‘efficient data compression’
    I think this is the only feature that has any merit in Canonical’s list, but I cannot call it a¬†killer feature. Work is ongoing on adding compression into ext4, but nobody seems to care much about doing it. If you really want it its baked into btrfs on Linux. So no, canonical, this is not a ‘killer’ feature.

ZFS – and sadly btrfs – are both rooted in a 1990s monolithic model of servers and storage. btrfs hasn’t caught on in Linux for a variety of reasons, but most of all its because it simply isn’t needed. XFS runs rings around both in terms of performance, scales to massive volume sizes. LVM supports XFS by adding COW snapshots and clones, and even clustering if you so wanted. I believe the interesting direction in file systems is actually things like Gluster and Ceph – file systems designed with the future in mind, rather than for a server model we’re not running any more.

Canonical are targeting ZFS support for containers, saying that its the perfect fit for that. The irony is containers don’t need ZFS.¬†Red Hat¬†is using a LVM/devicemapper CoW based approach. CoreOS has switched away from btrfs (ZFS-style snapshots!) to overlayfs and ext4 – and apparently performance was much better. Docker can use OverlayFS as well and recommends against using ZFS.

Of course, Ubuntu suffers from NIH syndrome, and so isn’t using Docker/Atomic/rkt etc – it has created its own container technology – LXD. Perhaps it doesn’t matter then that you’re using OpenZFS – if you’re planning on using LXD (and thus ZFS) you’re ignoring the mature, established container technologies and picking a platform that is almost certainly going to be poorly supported going forward.

Life is not fair

I loathe the phrase. As a statement of fact it doesn’t really make that much sense. Life can’t be fair or unfair. Its just what it is. People rarely utter this phrase to support somebody and is not supportive anyway. More often people use this phrase as justification.¬†¬†“Well, life isn’t fair”. Its a common phrase for parents to utter to their children. “Life is awful so don’t complain” seems to be the implication.

If you were to disagree with my assertion above, then logically we use the phrase “life isn’t fair” to explain that the world is not a fair place and thus¬†logically we must expect in our lives to be treated unfairly, to be treated badly, to expect misfortune, bad luck, and for other people to not treat us¬†justly.

Are we really saying that though? We also get told we must follow the rule of law, we must do what our teachers say, we must do what our managers say, we should¬†treat others as we expect to be treated, we should be fair to others. Aren’t these two statements in conflict with each other? The phrase “life isn’t fair” as a justification only works if we thus teach or ‘allow’ people to be unfair to others.

I don’t believe in a “just world”. I don’t believe karma is real or that a divine entity will make adjustments to make life fair. I do believe that actions have consequences but not to a grand plan of fairness, actions and reactions are just what they are, and just happen, in many cases randomly.

Despite all of this I go about each day with a genuine, deeply felt sense that I have been treated badly, undeservedly. I do not deserve to be treated as I am. Given how much time I spend caring about everybody else, given how much effort I put into things that benefit others, its not fair, right?

Well, I believe this because I was taught that we must all be kind to each other, we must honour each other, be nice, and not be selfish. Quite frankly this is¬†utter bullshit. I have nothing but anger and contempt for the actions of people who taught me this tripe. Its not true. The world isn’t a fair place, as they blindly would tell me when I was mistreated, but would enforce their moral views on me anyway. I’m required to care about others, not be selfish, but when others do the same to me, all that is left is an empty statement of agreement and a useless retort –¬†life isn’t fair.

One of the most important lessons I’ve learnt through counselling is that our life is ours alone and we should not let others control us, instead, we must seek to achieve what we want and in most cases put our needs before others. All of this is in super stark contrast to what most of us are told when children, when we’re taught that we should put others first.

Life isn’t fair, so why should I act fairly all the time? Why exactly must I accept that life is shit, and yet, feel guilty if I do what I want rather than what others want? Why exactly am I expected to not do things others don’t like, or act the way others want me to act, when nobody ever does that for me?

What really bothers me is that in life other people act unfairly, and I have to accept this, because¬†life isn’t fair, and I’m not allowed to act unfairly myself, but on top of this, I’m not¬†even allowed to express my frustration about other peoples actions. No. I must not do this. I must be quiet, not cause a problem, and censor myself, because, after all,¬†life isn’t fair.

In researching this blog post I read a lot of articles about fairness and the reality of the world, one of my favourites was your broken idea of fairness. I don’t agree with all of it, and I don’t have a broken idea of fairness because, as you have just read, I know the world isn’t fair. What I think is important is his Rule number one.¬†Life is a competition. It isn’t meant to be fair, and if you believed all that crap growing up about sharing, fairness, etc, then you were gullible. Life is about getting what you want over others.

I think the most important part of the article is talking about how other peoples morality is forced onto us as children:

People like to invent moral authority. It’s why we have referees in sports games and judges in courtrooms: we have an innate sense of right and wrong, and we expect the world to comply. Our parents tell us this. Our teachers teach us this. Be a good boy, and have some candy.

But reality is indifferent. You studied hard, but you failed the exam. You worked hard, but you didn’t get promoted. You love her, but she won’t return your calls.

Life isn’t fair. But don’t let others tell you¬†to act fairly when simultaneously justifying the world by saying it isn’t fair. Do what you want. Be you. Say it like it is, and realise that others are competing with you. Its very unlikely they will place you before them, so don’t place them before you.

Bargate security overhaul

Bargate is a web application that lets users access their files on SMB/CIFS servers within the corporate network. It thus connects to SMB/CIFS servers on behalf of the user and authenticates on their behalf as well. To do this it needs their password each time the user loads the page and thus connects to the back end SMB server.

The existing design

The existing security design of Bargate is predicated on the belief that the server should not be trusted to store the user’s password. If it stores the user’s password then any break in to the server / web application could obtain the list of stored user’s passwords. Encrypting them, whilst making an attack slightly more difficult, doesn’t solve the underlying problem¬†since the application will need to have the decryption key stored on it in order to use them. An attacker could steal both in nearly all circumstances.

Bargate thus stores the passport in the user’s session which is client side (stored in cookies). It is encrypted first using AES 256-bit CFB, then put in the session, and the session is signed by¬†itsdangerous¬†before being put into a cookie for the user. The encryption/decryption key for the AES 256-bit step is stored in the Bargate configuration.

The danger in this design is:

  • The encrypted password is sent across the network on every request (even if it is over SSL)
  • The encrypted password is stored in the cookie and thus on a myriad of end user devices, for perhaps up to 30 days (depending on session lifetime)
  • If an attacker gains access to the ENCRYPT_KEY (stored on the server) it can decrypt any password stored on any end user device, and gets the user’s actual password

This design was chosen of course because storing the password on the server, with or without encryption, is even worse. It would also mean any flaws in¬†Bargate to allow attackers to steal a user’s session would work without without first having to first compromise the end user’s device as is the case today. Today if there are any flaws like that in the code they are innocuous as the attacker won’t have the encrypted password, and thus won’t be able to access any systems.

The new design

What we want to achieve is quite simple:

  • The bargate server, if attacked, can’t be used to steal user passwords (i.e. don’t store users passwords in plain text and don’t store them encrypted if the encryption key is known by the application)
  • The end user device, if attacked, can’t be used to get the user password or even the encrypted password
  • The user’s password or encrypted password should not be sent over the wire on every request, only at log on time

The password of course has to be stored somewhere, but it does not have to be stored in plain text, and the place where it is stored does not have to have the encryption key. That is how it works today – its stored on the client which doesn’t have the encryption key – but this has several downsides. Instead the new Bargate authentication system will store the password encrypted on the server, but encrypted with a key stored in the user’s session, thus reversing the design.

This means:

  • Passwords are no longer encrypted using the same encryption key for every user, each session has a unique encryption key.
  • The end user device does not store the password in any form, which allows the deploying company/group/user to focus on server security rather than end user device security (especially important in the age of BYOD).
  • Attacking the end user’s device gives the attacker no useful information. If you get access to the per-user/session encryption key stored on the client this key only decrypts an encrypted password the client never has and never will have.
  • The encrypted password is not sent over the network on each request
  • The decryption key sent over the network on each request is itself encrypted by a key known only by the server, so it is useless to an attacker eavesdropping on the connection (if they had broken TLS).

The new design in detail

  1. The end user logs into Bargate by sending their username and password over TLS
  2. Bargate checks the username and password via LDAP, Kerberos or “SMB”
  3. Bargate generates a 32-byte (256-bit) session encryption key for the user
  4. Bargate encrypts the user’s password using the session encryption key¬†and stores it on the server (most likely in Redis with an expiration)
  5. Bargate encrypts the session encryption key using ENCRYPT_KEY (a bargate config option) and stores it in the user’s session. Bargate does not store the session encryption key any longer.
  6. The user’s browser saves the encrypted decryption key in the browser’s cookie storage
  7. The user’s browser is redirected to view a file server
  8. The user’s browser presents the encrypted decryption key to the server as a cookie over TLS
  9. Bargate decrypts the decryption key using ENCRYPT_KEY
  10. Bargate uses the resulting decryption key to decrypt the password stored in Redis
  11. Bargate uses the decrypted password to authenticate to the SMB server on the user’s behalf

Remaining attack vectors

There are two remaining attack vectors.

  • Session hijacking
    • An attacker can still take session cookies off a client and then use them. This threat is reduced with TLS and http only cookies, but an attacker could still get to them. This is a generic problem with web applications however. Adding restrictions to lock sessions to an IP address is an option, but can be disruptive and is of limited benefit.
  • Attacker with access to both the¬†server and client
    • If the attacker has compromised both ends, well, you know, game’s over anyway.

A replacement for nss_updatedb: nis2db

In 2011 the glibc project released version 2.15 which dropped support for using Berkley DB based database files as a source of user/group information (well, any name service switch information). Instead the “db” backend of name service switch (nss) is now provided by a simple glibc-specific db file format.

This means the nss_updatedb tool, which I have used for years to provide network-free user/group information on Linux machines, no longer works on modern Linux systems. The tool generated BDB files that glibc’s nss db module¬†simply does not support anymore. All Linux systems using glibc 2.15 or later are affected by this change.

To restore the functionality I need, which is to pull user and group information out of NIS and place them in db files that glibc can read, I have written “nis2db” which is a really simple python script which reads from NIS and uses the “makedb” command shipped with glibc.

The tool is available now and is open source: https://github.com/divad/nis2db