Apache Struts vulnerability (CVE-2017-5638) and the importance of RFC standards compliance

The recent Apache Struts vulnerability (CVE-2017-5638) has highlighted how quickly a vulnerability disclosure can morph into an exploitation frenzy.

From the initial disclosure announcement on March 6th, signs of testing and exploitation attempts in the wild materialised in less than 24 hours as the gravity of the situation sunk in.

The issue, being a specially crafted “Content-Type:” MIME header, was found to permit remote code execution (RCE) through calling Java to create a new java.lang.ProcessBuilder() object, executing the required arbitrary command on the web server without authentication. Further details about the exploit in action can be found here.

The challenge for many organisations facing this issue, is to be able to react in a timely fashion, to mitigate the risks of their own systems being exploited at the cost of loss of service, loss of confidentiality, and loss of own or customer data.

Under a backdrop of service management procedures and development safeguards, for many companies, simply upgrading the affected component may take days or weeks to implement, and meanwhile, attackers are all over your services like a rash.

The exploit used a specially crafted “Content-Type:” header, such as detailed here and the same as or simplar to the following example:

curl -i -v -s -k  -X 'GET' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0' -H 'Content-Type:%{(#nike=\'multipart/form-data\').(#dm=@ognl.OgnlContext@DEFAULT_MEMBER_ACCESS).(#_memberAccess?(#_memberAccess=#dm):((#container=#context[\'com.opensymphony.xwork2.ActionContext.container\']).(#ognlUtil=#container.getInstance(@com.opensymphony.xwork2.ognl.OgnlUtil@class)).(#ognlUtil.getExcludedPackageNames().clear()).(#ognlUtil.getExcludedClasses().clear()).(#context.setMemberAccess(#dm)))).(#cmd=\'echo "add malicious commands here"\').(#iswin=(@java.lang.System@getProperty(\'os.name\').toLowerCase().contains(\'win\'))).(#cmds=(#iswin?{\'cmd.exe\',\'/c\',#cmd}:{\'/bin/bash\',\'-c\',#cmd})).(#p=new java.lang.ProcessBuilder(#cmds)).(#p.redirectErrorStream(true)).(#process=#p.start()).(#ros=(@org.apache.struts2.ServletActionContext@getResponse().getOutputStream())).(@org.apache.commons.io.IOUtils@copy(#process.getInputStream(),#ros)).(#ros.flush())}' \   'http://vulnerable.web.site.com/struts2-blank/example/HelloWorld.action'

Now while many people are focusing on the "#cmd=" string to identify this being exploited, this is just a variable within the exploit, the real dodgyness happens when “new java.lang.ProcessBuilder(#cmds)” happens. So the problem is being able to call the java method to arbitrarily execute a command.

But is that it? maybe not. You might think ah! i’ve cracked the problem – prevent java from being used, but the problem is actually far more simpler than that.

Now what does this exploit have to do with standards compliance?

Back in June 1992, two people, Nathaniel Borenstein and Ned Freed devised RFC1341 – entitled “MIME (Multipurpose Internet Mail Extensions): Mechanisms for Specifying and Describing the Format of Internet Message Bodies”.

Within this document, in section 4, it describes in Backus-Nour-Form the structure of a “Content-Type:” declaration (as also detailed by W3C here).

That extract is as follows:

Content-Type := type "/" subtype *[";" parameter]

type := "application" / "audio" / "image" / "message" / "multipart" / "text" / "video" / x-token

x-token :=

subtype := token

parameter := attribute "=" value

attribute := token

value := token / quoted-string

token := 1*

tspecials := "(" / ")" / "<" / ">" / "@" / "," / ";" / ":" / "\" / <"> / "/" / "[" / "]" / "?" / "." / "="

The above tspecials must be in quoted-string, to use within parameter values.
Note that the definition of "tspecials" is the same as the RFC 822 definition of "specials" with the addition of the three characters "/", "?", and "=".

You might be thinking…”so what?”…well, the interesting thing is that the exploit was permitted due to lack of validation.

The exploit opened up its “Content-Type” value with the characters “%{“. Note that neither of these characters are permitted within the BNF specification for what a tspecial within a “Content-Type:” declaration should look like in RFC1341.

As such – it is evident that the impact of the exploit was accelerated due to lack of input validation, partly because the exploit was materialised using characters outside of the permitted structure.

While this validation itself would not have resolved the vulnerability in Apache Struts from existing, such validation would have significantly mitigated the impact of its exploitability.

Standards such as the RFC’s provide some great insight into what should be performed in terms of input validation. I am sure there are many other vulnerabilities and examples of where well defined constructs can be abused by using inputs not specified within the documented construct. It is for this reason that input validation is so important.

For those developing web application firewall rules, simply detecting a curly brace within the “Content-Type:” declaration would be sufficient to have mitigated this exploit and probably many more, I am sure this kind of logic could be applied to a whole host of declarations to improve security.

Seeing this exploit proven in testing brought only a sense of awe of how easy it was using this exploit to effectively walk through an open door. I am sure this exploit will have many reprocussions in weeks to come. The show is not over by far.

BIND DNS query log shipping into a MySQL database

BIND DNS query log shipping into a MySQL database

Yay!, I’ve been wanting to do this for a while! Here it goes:-

Documented herein is a method for shipping BIND DNS query logs into a MySQL database and then reporting upon them!

Note: SSH keys are used for all password-less log-ons to avoid prompt issues

BIND logging configuration

BIND named.conf query logging directive should be set to simple logging:-


  # Your other log directives here

  channel query_log {
    file "/var/log/query.log";
    severity info;
    print-time yes;
    print-severity yes;
    print-category yes;

  category queries {

The reason why a simple log is needed is because the built-in BIND log rotation only allows rotation granularity of 1 day if based on time, hence an external log rotation method is required for granularity of under 24 hours.

BIND query log rotation

My external BIND log rotation script is scheduled from within cron and it looks like this:-


if [ -e $LOCK_FILE ]; then
  if [ ` ps -p $OLD_PID > /dev/null 2>&1 ` ]; then
    exit 0
echo $$ > $LOCK_FILE

cat $QLOG > $QLOG.`date '+%Y%m%d%H%M%S'`
if [ $? -eq 0 ]; then
  > $QLOG
service named reload

rm -f $LOCK_FILE

Place this in the crontab, working at between one and six hours, ensure it is not run on the hour or at the same time as other instances of this job on associated servers

make sure /var/named/chroot/var/log/old exists for file rotation, used in the data pump script later on.

From here, I create a MySQL table, called dnslogs with the following structure:-

create table dnslog (
  q_server   VARCHAR(255),
  q_date     VARCHAR(11),
  q_time     VARCHAR(8),
  q_client   VARCHAR(15),
  q_view     VARCHAR(64),
  q_text     VARCHAR(255),
  q_class    VARCHAR(8),
  q_type     VARCHAR(8),
  q_modifier VARCHAR(8)

You can either define a database user with a password and configure it such in the scripts, or you can configure a database user which can only connect and insert into the dnslogs table.

Then I use the following shell script to pump the rotated log data into the MySQL database:-

PATH=/path/to/specific/mysql/bin:$PATH export PATH
NAME_SERVERS="your name server list here"


if [ -e $LOCK_FILE ]; then
  if [ ` ps -p $OLD_PID > /dev/null 2>&1 ` ]; then
    exit 0
echo $$ > $LOCK_FILE
for host in $NAME_SERVERS; do
  REMOTE_LOGS="`ssh -l $SSH_USER $host find $LOG_DIR -maxdepth 1 -name $LOG_REGEX | sort -n`"
  test -n "$REMOTE_LOGS" && for f in $REMOTE_LOGS ; do
    ssh -C -l $SSH_USER $host "cat $f" | \
      sed 's/\./ /; s/#[0-9]*://; s/: / /g; s/\///g; s/'\''//g;' | \
        awk -v h=$host '{ printf("insert into '$DEST_TABLE' values ( 
STR_TO_DATE('\''%s %s.%06s'\'','\''%s'\''), 
$3 * 1000, 
"%d-%b-%Y %H:%i:%S.%f", 
); }' | mysql -A -S $DB_SOCK -u $DB_USER --password=$DB_PASS $DB_NAME 2> $ERROR_LOG
    if [ $RETVAL -ne 0 ]; then
      echo "Import of $f returned non-zero return code $RETVAL"
      test -s $ERROR_LOG && cat $ERROR_LOG
    ssh -l $SSH_USER $host mv $f ${f%/*}/old/

Put this script into a file and schedule from within crontab, running some time after the rotate job suffice to allow it to complete, but before the next rotate job.

Note that the last operation of the script is to move the processed log file into $LOG_DIR/old/.

This will take each file in /var/named/chroot/var/log/query.\* and ship it into the dnslogs table as frequently as is defined in the crontab.

From here, it is possible to report from the db with a simple query method such as:-

PATH=/path/to/specific/mysql/bin:$PATH export PATH


if [ -e $LOCK_FILE ]; then
  if [ ` ps -p $OLD_PID > /dev/null 2>&1 ` ]; then
    exit 0
echo $$ > $LOCK_FILE

echo "select * from dnslogs where q_text like '$SQL_REGEX';" | \
  mysql -A -S $DB_SOCK -u $DB_USER --password=$DB_PASS $DB_NAME

rm -f $LOCK_FILE

And there it is! SQL reporting from DNS query logs! You can turn this into whatever report you like.

From there, you may wish to script solutions to partition the database and age the data.

Database partitioning should be done upon the q_timestamp value, dividing the table into periods which align with the expectation of the depth for which reporting is expected. On a minimal basis, I would recommend keeping at least 4 days of data in partitions of between 24 hours and 1 hour, depending upon the reporting expectations. If reports are upon the previous day’s data only, then 1 partition per day will do, while reports which are only interested in the past hour or so will benefit from having partitions of an hour. in MySQL, sub-partitions are not worthwhile because they give you nothing more than partitions but adds a layer of complexity on what is otherwise a linear data set.
Once partitioning is established, it should be possible to fulfill reports by querying only the relevant partitions to cover the time span of interest.
Partitioning also has another benefit, which is data aging. Instead of deleting old records, it is possible to drop entire partitions which cover select periods of time without having to create a huge temporary table to hold the difference as would be required by a delete operation. This becomes an extremely useful feature if you have a disk with a table size which is greater than the amount of free space available.

Script updates for add and drop partition to follow….

AV Comparatives

Today, let me introduce you to AV Comparatives, a trusty AV testing lab which will open your eyes to how good your anti-virus is. I have used these guys for many years to consider my options on AV.

Disclaimer: Don’t be fooled by the sell of McAfee and Symantec – they are *NOT* the best AV products by a country mile.

The reports from AV Comparatives shows the difference between “out-of-the-box” and “configured-for-security” effectiveness. This provides an interesting and sometimes scary revelation.

Please pay special attention to the historical reviews for proactive tests. The teams that score best consistently on this test do better overall because if they are on-top for 0day threats then the historical virus detection is, as they say, “history”. You can see developer drain happen when a product slips from it’s ranking where a developer leaves or the company generally lags.

For Windows, I normally use Avira Free with secure-start and detection of all categories including jokes and games. Just taking another look, I guess I might reconsider…..maybe Avast?

I’d like to try QiHoo but I can’t read Chinese and I’m not sure i trust a ‘free’ product which is difficult to find on Google and intended for a single-country only market (you can’t even find it easily on Baidu!). Chinese users – please leave comment on this point and let me know what your experience with QiHoo AV is like!

Meanwhile, I’m on Linux, so ClamAV will do for now.

sendmail relaying nightmare!

While I’m hot on the topic – I’ve just spent a whole afternoon/evening trying to figure out why my sendmail installation keeps on becoming an open-relay every time i configure my desired domains! – which I have now figured out!

While listing my desired domains in the access file, or in the relay-domains file, it seemed to turn my sendmail host into an open-relay.

It turns out that access and relay-domains supports relay for all valid hosts and sub-domains within the DNS domains permitted for relay, hence all hosts with a valid DNS A record within the defined domains becomes a valid source of mail! As my testing point had a valid DNS record within the permitted domain (and I did check to see whether it was an open-relay), the host allowed relay based on membership to the permitted domains.
This effectively made my sendmail box an open-relay to all internal hosts with a DNS name.

This was fixed with a FEATURE:-


This sanitised my security from internal abuse! and made my access file work as intended, supporting explicitly listed hosts and domains only.


Update: I later realised that the domain names I was configuring also had ‘A’ records in DNS for the top-level domain. As these hosts were not valid mail sources for this relay, I had to explicitly configure a REJECT action within the access file for all of the IPs named in an ‘A’ record lookups on the given domain names within the access or relay-hosts file in order to deny an implicit behavior which is the consequence of permitting a given domain.


So….some things to remember for Sendmail:-


Any domain listed in the access file or relay-domains file will allow ‘open’ relay for all hosts :-


1) Within the visible DNS structure beneath the defined domain (unless you use “FEATURE(`relay_hosts_only’)dnl”)

2) Defined as an ‘A’ Record for the given domain name as returned by DNS.
Does your Sendmail MTA relay to the hosts you intend?


Why Ubuntu is both one of the the best and one of the worst linuxes ever

Well, after it being many years since I last used Linux as my main workstation OS, I have realised that I no longer need the Windows apps which caused me to stop using Linux at home as I had done previously for many years, and reverted back to what I know is better, only, i’m a little dissapointed, and I’ll tell you why…..

My last foray into Linux workstations led me (through laziness) to Ubuntu Studio – a far superior spin of Ubuntu, which offers, pre-packaged, all of the multimedia gems that you could want for a Linux based *multimedia* system, it was like SuSE but without all the bad vibes from their Novell and Microsoft dealings. I come from a Slackware background and have never impressed by fancy guis and a shit-heap of patches on top of every app compiled from source, so i’ve never been a fan of Red Hat Linux for many reasons outside of the scope of this post.

So, I thought ….. Hmm ….. Slackware …. or …… Ubuntu Studio, err, which boiled down to the act of thinking and doing (or desired lack thereof) …. configure Slackware the hard way ………… or ………… use auto-configuring Ubuntu Studio? Easy, I haven’t got time for the krypton factor, so Ubuntu Studio it is.

Ubuntu Studio installed fine, but I was massively dissapointed that it didn’t support LVM on install (i’m yet to transplant it onto a LVM setup), so I decided to go with a 2Gb boot, 2Gb swap, and rest-of-disk root setup, with the on-disk ordering of swap, data, boot through using fdisk from the command-line. I don’t know why but I’ve never trusted the installers to partition the disks properly, I see some abhorrences of set-ups used by which quite frankly is going to be the subject of another (maybe next?) post. Here’s my partition table:-

    Device   Boot              Start                End        Blocks      Id     System
/dev/sda1     *     1460954864  1465149167       2097152   83   Linux
/dev/sda2                        2048        4196351       2097512  82    Linux swap
/dev/sda3                  4196352  1469854863   728379256  83    Linux

With sda1 being /boot, sda2 being swap and sda3 being root (/)

The rationale for my partitioning was this:-

  • You shoudn’t need more than a few gig for a few kernels, and disk space is cheap
  • You shoudn’t ever need more than 2-4Gb of swap – if you’re using this much swap – you don’t have enough RAM – simple. The essence is this – if it’s imporant enough to want running – you don’t want it sitting in swap, and conversely, if it is sitting in swap then you have to ask yourself is it important enough to keep it running? sometimes yes, most often no
  • For the actual ‘UNIX’ filesystem you can partition and fragment your filesystem across your disks to your heart’s content and in a plethora of ways but in essence, with one spindle only, it can only do so much, and despite the number of partitions per spinde (set), will take the same amount of time per spindle (set) to recover hence I opt for a single filesystem on the grounds that with a single disk, it’s all-or-nothing anyway.
  • Logical partitioning should be in the order, boot, swap, root – why it makes sense and is easy to conceptualise
  • Physical partitioning should be in the order swap, root, boot (the 1024 cylinder limit doesn’t apply now!) – why this order? because swap is wanted on the fastest part of the disk, and we don’t care how slow boot is so long as it works, so boot gets the slowest slice and swap gets the fastest slice, and in-between, root or LVM gets the rest.

This was great, and the install, once partitioned to my satisfaction, was seamless, I had a working machine.

I then proceeded to install KDE through “apt-get install kubuntu-full”, and many will ask – why didn’t you just install kubuntu and then install the packages you need for everything else? well, the reason is that I wanted a multimedia system, and Ubuntu Studio offers the best multimedia foundation (IMHO), hence my starting point.

But… being a cradle-to-the-grave KDE user (i’ve never cared too much about the cathederal and the bazaar so long as everyone gets their expected cut), I just had to have KDE over XFCE and/or Gnome, I would use WindowMaker as my second choice, but I’ve just got to move with the times (although it’s an excellent light-weight window manager for recovery shells).

So… Now I have a working Ubuntu Studio + KDE system, and i’m rather happy.

I then go venture under-the-hood, crack open konsole, and install MySQL and Apache. “apt-get install xxx” works fine as expected, and then I go to bounce the services, to be notified with:-

  • “Rather than invoking init scripts through /etc/init.d, use service(8)
    utility, e.g. service mysql restart
  • Since the script you are attempting to invoke has been converted to anUpstart job, you may also use the stop(8) and then start(8) utility is also available.
    e.g. stop mysql ; start mysql. The restart(8) utility is also available.

WTF? what’s wrong with calling init scripts from /etc/init.d/xxx <start|stop|restart>?

I refer to the System V as setting the standard of using /etc/init.d for init scripts as a way of managing run-levels, even Solaris have migrated away from init as a runlevel manager, but my gripe is – if you’re going to use SysV instead of BSD init scripts then why change it? why whinge? and why try and re-invent the wheel? SysV init works fine as it is ffs? don’t change it! and don’t nag me about knowing about it either!

So, after explaining why Ubuntu Studio + KDE is (IMHO) one of the best Linuxes, I finish on a note of annoyance as the system tries to tell me my standards compliant ways are not appreciated here.

I guess this is where Linux fails in the Enterprise – it’s willingness to re-invent the wheel (and the lack in thereof which makes me now prefer AIX and FreeBSD), I’m not too old to move with the times, but have matured enough individually to want some stability in things that work and are tried and tested.

Linux has deviated far too much fromt the core of what a UNIX system is, into some hybrid that is absolutely great for the desktop but still leaves a lot lacking for the Enterprise server estate. Red Hat has become a safe choice, but not because it is superior but in fact is because it is a Linux which *everyone* and *anyone* who has heard of Linux knows about Red Hat!

The problem with Ubuntu Studio is in fact the “Ubuntu” bit, the distro itself is an absolute merit to what a multimedia platform should be like, but alas, beyond the savvy collection of apps, is a nannying, almost patronising Ubuntu systems, goading me to adopt non-standard ways. Grrr. Give Me “Slackware Studio” – now that’s an idea

Now I have a primary Linux system, my next distro foray is going to be with Slackware (possibly on ARM – thanks Stuart! I owe you a biscuit or three when the talking clock strikes 😉

Cached Passwords

Cached passwords are the holy grail of hackers and a principal target for information thieves, for this reason you should never save your password on your workstation and never use the same password across systems.

When prompted to save a persistent authentication token such password, you should always say no or never. The only place for a ‘stored’ password is an encrypted and password protected database (for which it should not have the password written down).

SRW Iron

Who does your browser consult when deciding which sites to go to? well, many browsers come with ‘smart’ technologies which ‘phone-home’ to various vendors and organisations of sorts, providing a potential data leakage of web request urls through services such as ‘safe-search’ and browser search plugins.

SRW Iron is Google Chrome with all the google-phone-home stuff taken out, and while yes, I know you can tweak Chrome to be as quiet if you want – Iron comes secured out of the box and has all functionality where data privacy is a concern is turned off permanently. Get it at:-