"Are you sure it's not something on the server?"
"Can't see how it could be. Maybe it's some Drupal security thing that's being triggered."
Lara's blog post wouldn't save, and no error was being thrown. When she tried to save it, it just reloaded the edit page.
You know how this goes. After insisting it couldn't be the server, and that nothing had changed, I remembered that a few days ago I had enabled Apache's mod_security module.
I love ModSec. It's a wonderful piece of work that not only helps keep your server secure, it can reduce load by rejecting obviously bogus requests, and log suspicious activity that you might not otherwise catch. But as with power tools, you have to be careful not to take your eye out.
Colonel Mustard, on the server, with the regex
It took a while to figure out what was going on. Lara had narrowed down what was triggering this odd behaviour. Two words and one character - "update", "set" and "=". They were in widely different parts of the text, but it was those three strings, and in that order.
If you're thinking
update set = looks a lot like SQL, you are correct. One thing ModSec can do is examine content submitted to the server. Some of those rules look for SQL injection attacks. One of these rules was a bit too general and was matching the content of the blog post. Once we had figured this out, it was easy enough to find which rule, and turn it off in specific circumstances. Blog post saved!
Read on for the exciting technical bits...
Choose your rules...wisely
If you suspect you are getting ModSec errors, which often manifest as 400 or 500 level HTTP codes, you can search the Apache error log. ModSec will record entries there, and they will include a string like "ModSecurity", along with the rule that was matched. This command will find these:
grep -i "modsec" /usr/local/apache/logs/error_log | sed "s/$/\\n/"
(That sed bit at the end just adds an extra newline between matches so it's easier to read.)
On a CentOS serve with WHM/cPanel installed, your error log will be at
/usr/local/apache/log/error_log. File locations and the preferred way of organizing ModSec rules may be different on other set ups, but the principles will be the same.
Running the above command generated a number of lines like this:
[Wed May 07 18:01:09 2014] [error] [client 127.0.0.1] ModSecurity: Access denied with code 500 (phase 2). Pattern match "((alter|create|drop)[[:space:]]+(column|database|procedure|table)|delete[[:space:]]+from|update.+set.+=)" at ARGS:body[und][value]. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "368"] [id "300015"] [rev "1"] [msg "Generic SQL injection protection"] [severity "CRITICAL"] [hostname "othermachines.com"] [uri "/node/30/edit"] [unique_id "U2qtJUBb5WEAACk@7hQAAAAF"]
There are two interesting parts in there. One is the "Pattern match" section. This shows you the rule that was actually triggered. If you're familiar with regular expressions, you can see that
update.+set.+= is going to match a lot of text that is perfectly innocuous, especially on a developer blog.
The second part that we're interested in is the rule ID:
[id "300015"]. We need the ID so we can find the ModSec rule and modify or disable it.
You can find your ModSec rules in
/usr/local/apache/conf/modsec2.user.conf, or in WHM under Packages > Mod Security. Do not modify this file as your changes may be overwritten.
Searching for the ID, we find the rule that was triggered:
#Generic SQL sigs SecRule ARGS "((alter|create|drop)[[:space:]]+(column|database|procedure|table)|delete[[:space:]]+from|update.+set.+=)" "id:300015,t:lowercase,rev:1,severity:2,msg:'Generic SQL injection protection'"
I happen to like this rule, even if it's a little broad. It does match SQL injection attacks, and in the majority of cases will not cause problems. I do want to be able to bypass it in areas where we might be editing site content. There is a file designated for whitelisting ModSec rules,
We wanted to configure things so that the rule would be ignored for specific URLs using Apache's
LocationMatch allows us to restrict how a directive is applied based on its URL. It's also handy because it will accept regular expressions. (You can use
Location if you don't need the pattern matching). In our case, this is URLs that are used for editing a Drupal node, and URLs in the /admin section, so we added this to the whitelist file:
# Disable generic SQL injection rules globally # for Drupal content admin <LocationMatch /node/[0-9]+/edit> <IfModule mod_security2.c> SecRuleRemoveById 300015 </IfModule> </LocationMatch> <LocationMatch /admin> <IfModule mod_security2.c> SecRuleRemoveById 300015 </IfModule> </LocationMatch>
As you may have guessed,
SecRuleRemoveById 300015 removes this rule from ModSec processing, but because we've limited where this is applied with
LocationMatch, it will still be applied to all other URLs on the site.
Once that was saved, and Apache restarted, we were able to continue blogging as normal.
More on disabling or modifying ModSec rules from Atomiccorp:
More on Apache's LocationMatch directive: