Argh!

Wednesday, 29 November 2017

Once again I should to explain a lot of the context (and apologize for some of it) before I get to the point. But the short version is both the mysql and mysqli extensions seem to cast all mysql data as strings.

Normally this goes unnoticed when you're working with PHP - its dynamic type conversion handles all the bumps. But not this time.

I wrote an application for tracking data quality some time ago. This runs various rules against several large datasets and collects the exceptions in a single polymorphic table which looks something like:

Each row of output from each rule is stored in a record in this table. Periodically the historic data is deleted.

Yes, I confess I'm holding a structured opaque data blob in a relational database! But by synthesizing PK1, PK2 and PK3 from the rule data it means I can compare month on month outputs to get the age of the exception. And I have a reliable unique identifier for each record in the (very large) table.

All well and good. This is currently tracking 320 rules and the application maintains summary level data. It has been running for about 8 years. But users as *so* demanding! They wanted to see one of the rules (customer whose email addresses pass a regex check but fail MX lookup) decorated with information come a separate database. Not such a big deal. The application has the capability to mailmerge the output of one ruleset to create a query for another (clever, eh?). So all that was needed was a bit of configuration.

But now the story takes a twist. The number of records involved is.....lets just say a lot. This all runs on a very old version of MySQL (because that's how RHEL rolls its distro, and everyone hates systemd) which was originally configured to give great performance on low latency storage (i.e. MyISAM) but now runs on a SAN with very high latency (but massive bandwidth).

Because of these considerations, along with the fact that I rarely rebuild the indexes, joining the output of the two rules was HORRENDOUSLY slow. Its supposed to be available at the click of a button, but I got bored waiting after one hour. This version of MySQL does not do merge joins. So I wrote my own in PHP. This took around 5 seconds to retrieve the data. But huge chunks of the data were missing! I eventually tracked this down to the fact that the mysql PHP extension was converting the PK1 integer attribute to a string. So "110" was greater than "1000" and my code was discarding lots of rows when trying to join the two datasets.

I can understand that there would be a cost if the library had to call back to the DBMS to get the type of each attribute in a resultset, but looking at the C API, it looks like the client lib is returning a correctly typed attribute, but the PHP engine is casting it as a string!

But, of course, the MySQL extension has long been deprecated. So I decided to see of MySQLi was similarly afflicted. After a brief diversion when I discovered about SO_PEERCRED I found that MySQLi does the same thing.

Saturday, 22 July 2017

When training
inexperienced programmers, there are 2 mantras I always need to
repeat:

You don't
write code for computers to understand; you write it for humans to
understand.

validate
input : escape output

Since this the
latter has cropped up a few times recently, I have documented the
reasoning here.

Both validation and
escaping are about protecting your system against attack and ensuring
good data quality.

Validate input

Validating input
means checking that the input falls within the acceptable boundaries
for your application. Does it look like the right data type? Is it
within a sensible range? Does it contain malware?

The first 2 examples
are there to avoid processing data which is obviously wrong. People
make mistakes when entering data – and prompting them to fix the
error is massively cheaper than trying to find and fix the bad data
later. Checking for malware might be as simple as rejecting content
which looks like HTML markup, or you might test the content against a
complete AP14 gateway with multiple heuristic malware checkers. The
point is that your code must decide whether to accept the data and
process it or reject it .

Why not transform input?

If your underlying
platform is vulnerable to buffer overflows or similar attacks, then
by the time the thread of execution reaches your code your system is
already compromised.

Later in your
processing you may well transform the data, but not at the point
where it enters your code. You need to ensure that you know how the
data is represented anywhere you are processing it; the simplest way
to do that here is to leave it in as raw a state as possible.

Exceptions

Validating input

In some cases you
may choose to reject the data silently, or re-route it to a honeypot
– but that is a very esoteric edge case. And both are still forms
of validation.

Transforming input

Again, there are
some edge cases where it is necessary to modify the input in order to
validate it, for example:

your code
expects data encrypted with specific keys

the input
data has tamper resistance added which should be removed before
processing (such as anti-csrf tokens)

Escape output

Escaping output means transforming it to a form where:

the original data is recoverable in a suitable
state/representation for further processing

the content of the data does no interfere with the control
channel for the data

That second one is a bit tricky. When writing to an SQL database
or creating HTML/CSS/Javascript for a browser the data is sent over
the same channel as the control structures. The relevant languages
have syntax for keeping the data and control seperate. Sometimes this
syntax is abstracted by an API (such as PDOs data binding or the
json_encode() function).
Any time is leaving your code and going somewhere is it should be
rendered in an appropriate format for the receiving process.
Sometimes a single script may have multiple output vectors – a
local record in a database, a notification email to the user, html to
the browser. Each need different representations of the data.
Hence to ensure the right representation of the data for the
target, the transformation should be near to the point where it is
output – both in the sequence of execution and the in the structure
of the code.
While a few of the transformation functions have both an encode
and decode implementation in PHP, it doesn't make any sense to try to
reverse the encoding of the data for one output channel in order to
write it to another within the scope of a single script. All the programming languages I have used are similarly asymmetric.

Exceptions

The only exception to the "Escape output" rule is If the data channel is independent from the control channel - for example when writing to a local file (although even then, you need to be careful not to write en EOF character as data to a file opened for text).

Monday, 6 March 2017

My teenage son, despite repeated requests, frequently walks away from the computer leaving it running. And now that I've got a shiny new and very quiet computer it is not always obvious that it has been left running.

While formerly I would just changed the program the window manager invoked as a screensaver Linux systems have got a lot more convoluted. Whether this complexity is a good or necessary thing - I will reserve judgement.

Anyway, to cut a long story short, the solution for me was to use xautolock which, once started, monitors for a period of inactivity then runs a program. Setting it to start automatically on my Mint / mate box (no XClients / Xsession) was a matter of adding a .desktop entry to the XDG autostart directory:

Monday, 26 December 2016

Given the brexit vote, falling pound and a reasonable bank balance, the whole family got new computers this year for Christmas. A Lenovo M700 MT desktop for myself, a HP laptop for my son and a Lenovo Yoga 510 for my daughter.

(I was rather surprised doing my initial research to discover that desktop computers now cost more than a laptop of equivalent spec - even though the latter also includes a battery and a screen.)

While my son just wanted MSWindows on his, my daughter wanted Linux. We had an interesting conversation with the salesman at PC-World. When she asked him if it would run Linux, he said that installing Linux would void the warranty. "But its just software?" she replied. He insisted that installing anything other than what it came with would void the warranty. I interjected at this point and asked him to clarify "Installing any other software than MS-Windows will void the warranty?"

"Yes - I can't get my manager if you don't believe me".

So we bought her computer from John Lewis.

Unlike my desktop which came with MSWindows 7, shrinking the C drive in Windows 10 actually worked! Getting into the UEFI setup to change the boot order was a bit harder though - no messages at startup to advice what key press would open the setup or allow the boot order to be changed.F1, F2, escape and F12 during boot had no effect.

The only option was to select restart from a running MSWindows session while holding down the left shift button. This rather enforced the option of dual boot - but with a 1Tb disk, there was plenty of room to spare.

I was rather surprised to discover that although I could get into the UEFI setup, the keyboard was completely unresponsive. Plugging in an external keyboard solved that problem. Boot order changed and I was ready to go.

Since it's a touch screen device, I decided to go with a Gnome 3 desktop. And since it came with SecureBoot, I decided to go with Fedora (which has a signed boot loader). I'll maybe get a bit more adventurous in future, but for now, baby steps.

Running from the USB stick was a bit ropey, but these problems went away when I installed to the hard disk. Everything is working apart from the touch screen. That's quite a big omission, but as relatively new hardware running with a very new display system (Fedora 25 uses Wayland rather than Xorg) it may take a while to get it sorted - meanwhile I'll let my daughter play around.

Even without the touchscreen stuff, this wee device is a viable proposition as a laptop - there are a lot of devices available at this price point, but this is notable in feeling very sturdy.

Having blogged about keyboards twice before, you may have got the impression I'm slightly obsessive. I am.

While my old PC was merely getting a bit past its sell-by-date, the keyboard I was using was horrible - a really nasty HiPoint silicone bump device purchased as a temporary measure until I could do some proper research.

As before I had started with what I thought was a reasonable set of requirements:

UK layout

Compact (i.e. full-sized keys, but without a numeric keypad) to save space on my small desk, and avoid having to reach so far for my mouse.

Nice tactile action

High contrast between the lettering and background on the keys (or illumination)

But it proved remarkably difficult to find anything even coming close to this bill. From much trudging around shops, I was able to identify that Cherry MX red's seemed to provide the right feel (albeit with rather a lot of travel needed). Even just reducing the list down to Cherry MX red / compact / UK layout, I could not find anything online. It seemed the only option I had to meet these requirements was to get a keyboard built to order by a specialist at a cost in the region of £150+ and even then, finding a tenkeyless offering was proving difficult.

My HP Pavillion desktop (4Gb, dual Athlon 2200) was getting a bit long in the tooth, and I decided to treat myself to a new rig.

In the past I've been rather impressed with the robustness of high-end PCs - I had a dual Pentium II HP Kayak XM600 for many years before I passed it on my mother for an extended tour of duty. Apart from the RD-RAM which never came down in price while the cost of DDR tumbled. Add to that the fact that running the X Window system, I wasn't going to see a lot of benefit from a fancy graphics card, I really wanted to get the fastest CPU I could. I did consider getting a retired server - but they are typically horrendously noisy. Then I thought about a used HP Z400. But looking at the behaviour of my current system, while the CPU was frequently maxed out, it was usually with just one or 2 processes causing all the pain. Having 8-16 CPUs wasn't going to help unless they were a lot faster than what I already had.

Looking at the single-core benchmarks, the 3.7GHz i3 6100 seemed to hit a sweet spot for budget/performance profile. So I started shopping for such a device. Form factor was a consideration - I have a small desk which my computer sits underneath - so a tower case was important. I ended up choosing a Lenovo ThinkCentre M700 from eBuyer.

The device itself is not as substantial as the PCs I've owned before - and despite the tower case, there is limited scope for expansion. Adding a couple of hard disks is about the limit. There are 2 DDR4 slots (one occuppied with a 4GB module), 3 PCI Express connectors (2 very small, and one small - I'm not familiar with the technology). The blanking plate behid one of PCIe connectors is filled with an external SATA connector (which could be moved elsewhere to make the slot usable). But apart from an SSD and possibly more RAM later I'm not looking at add a lot more hardware to this machine.

The handle at the front/top is not particularly sturdy, but adequate for occassionally lugging the device about.

Powered up, it seems remarkably quiet compared both to the HP Pavillion its replacing and the HP i5 generic corporate device I use at work.

Accessing the BIOS (sorry UEFI setup) was easy - just F1 at boot. The default configuration came with Windows 7 and Secure boot disabled. As I'm doing a lot more MSWindows dev and admin stuff these days, I decided to keep it as dual boot. Shrinking the C drive in MSWindows proved to be a complete PITA - even after disabling swap, updates and rollbacks, it still wouldn't release more than around 10Gb due to "unmoveable" files. And having to use the horrible tools with MSWindows was just too painful - so I booted up with a Mint install disk and had no problem shrinking the drive down to 100Gb.

Installing Mint 18.1 from a DVD I had prepared earlier was a no-brainer. I created the same users on the new machine as existed on the old, rsynched /home and everything worked - but a lot faster!

The onboard ethernet works with the RTL8168g driver. Its a 500Gb Western Digital HD. This feels a little slower than the 2 x Seagate 250Gb disks in the old machine (configured as OS+Backup and home+var+data+swap rather than in any RAID setup). But the planned SSD upgrade should fix that.

If only the laptop had been this easy! (details coming in a later post).

Saturday, 5 November 2016

If you find yourself here then you probably have at least some vague idea about how security is enforced on Unix systems. Or maybe you just wanted to read my continuing woes with computer systems.

I did spend some time thinking about a suitable title for this post. There were so many to choose from:

SELinux considered harmful

The emperor's new clothes

I want to believe

...but SELinux sucks sums it up nicely.

TL&DR

SELinux is ridiculously expensive and is unlikely to improve the Security of your system. It may make it worse.

Introduction

For those who know nothing about SELinux.....don't be hard on yourself. As a lot of this post discusses there are no SELinux experts. But in case you really know nothing about SELinux then a bit of context may help.

Unix (and therefore Linux, BSD etc) has a very elegant permissions system. There are lots of descriptions of how it works on the internet. Its read/write/execute and owner/group/other primitives can be combined to implement complex authorization models, not to mention the setuid, setgid and sticky bits.

What is SELinux?

It's a set of rules which are compiled into a configuration loaded and implemented at runtime.

Operations on entities are mediated by this abstract set of rules based on the labels attached to those entities and the user trying to effect the change.

So apart from the compilation step, not that different from permissions?

Well, actually, yes – the configuration is a mystery black box. Most experienced Linux/Unix users can tell by looking at permissions exposed in 'ls -l' and be able to make an accurate prediction about the outcome of an operation – and how to resolve the problem when this is not as required. The permissions are presented as 10 characters, sometimes more if we're talking about the directory the file is in or facls. While 'ls -Z' displays the SELinux labels on files, it doesn't say much about the permissions these enable. For that you need to look at the policy.

The targeted SELinux policy from Fedora is currently distributed as 1271 files containing 118815 lines of configuration. The rpm contains no documentation. On the other hand, the standard installation of Apache 2.4 on the machine I'm sitting in front of, has 143 configuration files (an unusually high number due to Ubuntu distributing stub files for every available module) and 2589 lines of configuration. So, SELinux has 10 times as many files and 45 times as much config as a very complex software package. Do I need to spell out the problem here?

Indeed, the recommended practice is not to change these files, but rather add more configuration to change the behaviour of SELinux.

One consequence of the Gordian knot, is that upgrades to the configuration (which at least won't trash the extra config you have added) often need to change the labels on the filesystem; a simple upgrade can unexpectedly morph from a brief outage to hours or days of disk thrashing while every file on your disks is relabelled. And hopefully that didn't also break your security model. But...

It breaks existing security models

The targeted policy not only overrules the filesystem permissions, but also the access control mechanisms built into programs, for example 'at' is unable to read from /etc/at.allow running as a system_u user!

With the setuid bit set on an executable, you can run it as a different user, but retains the original SELinux context!

It is inconsistent by design

"By default, Linux users in the guest_t and xguest_t domains cannot execute applications in their home directories or the /tmp/ directory, preventing them from executing applications, which inherit users' permissions, in directories they have write access to. This helps prevent flawed or malicious applications from modifying users' files"
- https://access.redhat.com/

In other words, Linux users can't run compiled C programs but can run (malicious) Java, shell script, python, PDF, Flash.... where the logic is bootstrapped by an existing executable but does not require the executable bit to be set.

What about networking?

Of course SELinux can solve every security problem; it has the capability to restrict network access. This is not available in AppArmor, and you can't apply restrictions on a per-user or per binary application using iptables.

Debugging SELinux problems

Selinux has an option to only report, and not prevent actions. Great, that should simplify fixing things, right? However it is my experience that it does not log all exceptions that it subsequently enforces.

Under very controlled conditions, I was investigating a problem with a system_u user running 'at'. Suspecting that SELinux was the culprit, I setenforcing 0, tried running application - it worked, no log entries found. Maybe SELinux was not the problem? So I setenforcing 1, ran app - got message "cannot set euid: Operation not permitted", no log entries found.

WTF?

Again, I set enforcing 0, ran the app. Again it worked. Again, no log entries. Just to be sure I run some stuff which I knew would violate the policy – and that produced log entries. With no idea how to begin to fix the problem, I setforcing 1 again, ran the app, this time it worked!

Yeah! problem solved.

Then, 10 minutes later "cannot set euid: Operation not permitted", but now I was getting log entries.

Automated Baselining

You don't start reading through the kernel source every time something misbehaves on Linux, so maybe you should treat the default policy in the same way, as a black box. It sounds like a reasonable strategy. Just run your system in a learning mode then apply those lessons to the policy. Indeed severalcommentators advocate just such an approach.
(Trying to fix permissions in enforcing mode is a thankless task - each blocked operation is usually masking 3 or 4 further rules preventing your application from working).
So the first step to getting your application working is to switch off a critical security control? Really??!!!

Anyone who has worked on developing a complex system will tell you that getting full code coverage in test environments is a myth.

Yes, as Darwin proved, evolution works really well - but it takes a very long time.

sealert is painfully slow; in recent exercise I clocked it at around 20-25 log entries per second. Not great when you have a 100Mb log file to read. Amongst some of the oddities it identified:

SELinux is preventing /usr/libexec/mysqld from write access on the file /tmp/vmware-root/vmware145.

- you might think this means that mysqld was attempting to write to /tmp/vmware-root/vmware145, and you'd be wrong. This is part of vmware's memory management. But vmware also uses this as a general dumping ground. The odd thing is the directory is:

SELinux is preventing /sbin/telinit from using the setuid capability. SELinux is preventing /sbin/telinit from read access on the file /var/run/utmp. SELinux is preventing /sbin/telinit from write access on the file wtmp.

Clearly Redhat are not reading their audit logs, or maybe they just disable SELinux?

SELinux encourages dumb workarounds

One the problems we ran into when SELinux was enabled on a server I recently migrated was that email stopped working. The guys with root access started working on this (I made sure they had a test script to replicate the problem) while I started looking at other ways of solving the problem - it was having a significant impact on the service. Guess who came up with a solution first?

In about 2 hours I had a working drop in replacement for '/usr/sbin/sendmail -t -i' which PHP uses for sending emails.

I'm not criticizing the Unix guys. The people working on this are very capable and do have expertise in SELinux. The problem is SELinux.

But go back and re-read my previous sentence; in 2 hours I had written a MTA from scratch and bypassed the SELinux policy written by the experts at RedHat. WTF????? If I am some sort of uber-cracker then I really am not getting paid enough.

(spookily enough one of the reasons the server could not send email is shown in the screen shot at https://fedorahosted.org/setroubleshoot/ ! This might be why Redhat 7 now has a selinux bool httpd_can_sendmail)

Now, which do you think is more secure, the original Postfix installation using a standardized config which has been extensively tested in house or the MTA I knocked up in between other jobs?

Maybe its just me?

I've spent a very long time searching the internet for stories about how people have used SELinux to prevent and investigate attacks. While there are a huge number of articles proclaiming its benefits, I struggled to find many which demonstrated any real effectiveness.

Excluding the cases where a patch had been available for at least a week before the alleged incident, I was left with:

Just 3 cases. The first one is a very good article and I recommend reading it (although there are some gaps in the story).

Exaggerating for dramatic effect?

Am I? Yes, SELinux does have documentation. Indeed there's masses of it (after all, my point is that there *needs* to be!). The following extract comes from the httpd_t man page; this document is intended to equip administrators with enough information to manage the sandbox that the webserver runs within on a machine running RedHat's Type Enforcement policy:

If you want to allow HTTPD scripts and modules to connect to
databases over the network, you must turn on the
httpd_can_network_connect_db boolean.
setsebool -P httpd_can_network_connect_db 1

What this actually means is if you want your httpd to connect to entities labelled as type database_t across the network then you need to enable this. And, of course, create labels representing those resources as well. Was that obvious from the instructions? Of course, there are a whole lot of databases which are not lablled as databases in RedHat - ldap and memcache for instance.

It is interesting to note that despite specific booleans for Bugzilla, Cobbler and Zabbix (does that mean I
should create a new type for every application I might run on the box?) there's no boolean controlling integration of an application server such as mod_php, fastCGI, mod_jk, mod_perl....

It also seems SELinux doesn't like other security tools muscling in on its action:If you want to allow all daemons to use tcp wrappers, you must turn on the daemons_use_tcp_wrapper boolean. Disabled by default.

setsebool -P daemons_use_tcp_wrapper 1

I thought SELinux was supposed to be granular - but I can only use TCPWrappers with everything or nothing?

Should I care?

One of the problems developing secure web based applications is that everything in the site ends up running as the same user id. This doesn't mean you can't do privilege separation using sudo or daemons, but it does mean that you always have to implement security controls in your applications. A mandatory access system does not solve these problems, but it should simplify some some of them. Fortunately SELinux is not the only game in town; AppArmor, GRSecurity and Smack are all available, well tested and widely implemented on Linux systems.

Of course, if you are Google or Facebook, then you can afford to spend 100's of man years working out how to get SELinux working properly (and of course there are no security bugs in Android).

What is wrong with SELinux?

The people developing SELinux (or insisting on its use) have missed out on something I have drummed into every junior programmer I have trained:

We don't write code for computers to understand; we write it for humans to understand.

SELinux/Targeted policy is

- really bad for productivity,

- bad for availability,

- bad for functionality

It is quicker to bypass SELinux/Targeted policy restrictions than change them to allow a denied action.

What is the solution?

The time you would spend aligning the off-the-shelf SELinux policies with your application will be better spent addressing the security of your system in conventional ways. Switch off SELinux and fix your security.

About Me

Sometimes known as SyMcBean (IRC,NNTP and other places).
Long term computer geek who has missed out on the web bubble, millenium bug and every other IT related gravy train in the last 20 years. Recovering SysAdmin and developer (but still maintaining a
couple of open source projects).