There has been much discussion lately about how to prevent spambots from submitting forms on web sites. Many solutions have been presented, many of which impact the usability and accessibility of the web page. CAPTCHA is a classic case where the user and accessibility is directly impacted.
Note
A Brazilian-Portuguese translation of this blog entry is available at http://www.maujor.com/tutorial/spam-em-formularios.php.
Over the last year or so I have compiled the following basic techniques for blocking spam submission in web forms. I’ve implemented just a couple of these and through logging have found that they have effectively reduced around 99% of spambot submissions while having no or very little impact on the usability or accessibility of the forms. Nearly all of these techniques are performed server-side using PHP and the relevant PHP code is shown below, however, the tests can be readily implemented in nearly any server-side scripting language.
Disclaimer 1: These spam prevention techniques may not work for enterprise level application where spammers may target forms specifically. They are intended for generic contact, comment, or registration forms where a spammer is less likely to take the time to try and bypass your specific spam prevention mechanisms.
Disclaimer 2: These techniques primarily stop bots and automated spam submission programs. They also can filter certain content. However, they likely will not prevent an actual dedicated human from posting spam to your web site.
The techniques are:
- Detect spam-like content within submitted form elements
- Detect content within a hidden form element
- Validate the submitted form values
- Search for the same content in multiple form elements
- Generate dynamic content to ensure the form is submitted within a specific time window or by the same user
- Create a multi-stage form or form verification page
- Ensure the form is posted from your server
Detect spam-like content within submitted form elements
This technique is likely the most powerful spam prevention technique. Most spam bots are in existence to either post URL’s of web sites in an effort to increase traffic or increase their search engine ranking or they are attempting to hijack your form to send spam messages to you or others. Detecting commonly used spam content or e-mail header injections will stop nearly all spam bots dead in their tracks.
The following PHP code, when placed on your form processing page (the place where the form is submitted to), will search all of the form elements for the most common header injections and other code that may trick your mail processor into sending carbon copy or blind carbon copy messages to others. It also detects any content that includes the string “[url” which is used by most forum software to specify links. If any are found, it sets the $spam variable to true.
if (preg_match( "/bcc:|cc:|multipart|\[url|Content-Type:/i", implode($_POST))) {
$spam=true;
}
NOTE: Internet Explorer 6 has a bug that will not allow proper overflow of preformatted text. If you are still using that browser, you will need to properly reflow the PHP code lines from this page.
You can also detect links and urls within the form elements. The following will set the $spam variable if more than 3 instances of “<a” or “http:” appear anywhere within the form.
if (preg_match_all("/<a|http:/i", implode($_POST), $out) > 3) {
$spam=true;
}
This will defeat most spambots as they primarily focus on posting links or hijacking your mail script. Beyond this, some very basic word filtering can often catch spam that finds its way through.
$spamwords = "/(list|of|naughty|spam|words|here)/i";
if (preg_match($spamwords, implode($_POST))) {
$spam=true;
}
You can also use external spam detection services with up-to-date patterns of spam content. My favorite is Akismet. Akismet is commonly used for filtering spam on blog comments (it has blocked nearly 14,000 spam comments to this blog in the last 9 months!), but it can be used successfully for nearly any web form.
Detect content within a hidden form element
Most spambots will find your form, determine what the form element names are, and find the URL where the form is posted to. The software will then post those form elements with modified, spam-filled values back to the form submission URL. Typically, the bot will populate every form element with some value so as to best ensure that it will succeed in being posted. So, if you insert a standard text input element into your form, but hide it visually from the user so the user cannot enter anything into this field, it is quite likely that the spambot will still post some value for this form element. If you detect that the form element is submitted with a value, then it’s almost certainly a spambot.
For instance, your form element may be inserted as
<span style="display:none;visibility:hidden;"> <label for="email"> Ignore this text box. It is used to detect spammers. If you enter anything into this text box, your message will not be sent. </label> <input type="text" name="email" size="1" value="" /> </span>
Notice that CSS is used to hide the text input and its label from view. This code will also hide this content from modern screen readers. However, if CSS is disabled, the input will still be displayed. For this reason, an explanatory label is provided that informs the user to not enter anything into the text box. I also gave the input a nice, juicy, tempting element name of “email” – that’s almost certain to get the spambots to enter a value.
You then simply detect if the form element is empty. If it is not, then it’s either a spambot or a user that has CSS disabled and did not follow the label instructions.
if(!empty($_POST['email'])) $spam=true;
This tactic, like all of those listed here, should still present a useful, informative error message in case the user somehow triggers your spam detection flag.
Validate the submitted form values
This one perhaps goes without mentioning, but if you want certain form elements to be required, ensure that you are using a server-side script to detect if information has been entered into those form fields. If you require form information is a particular format (such as requiring a valid e-mail address), then validate it. Many bots will simply submit empty information for fields they do not recognize or will submit random information for certain form fields. Your standard form validation mechanisms can stop many spambots.
// If the message is empty, throw an error
if(empty($_POST['message'])) $error=true;
// if e-mail is not formatted correctly, show error message
if(!eregi("^[_a-z0-9-]+(\.[_a-z0-9-]+)*@[a-z0-9-]+(\.[a-z0-9-]+)*(\.[a-z]{2,3})$", $_POST['email']))) {
echo "Please enter a valid email address.";
}
Search for the same content in multiple form elements
Some spambots will post the same text into all unrecognized form fields. If you have two form fields that should never contain the same information, you can detect if their values are indeed the same and if they are, you can flag an error. On our forum registration form, I found that simply throwing an error if the first and last names were the same cut down on bot registrations by around 80%. It’s not a perfect technique and you should ensure that the fields you analyze should always be unique (I guess there is still a chance that a person could have the same first and last name, huh?).
if($_POST['firstname'] == $_POST['lastname']) $spam=true;
Generate dynamic content to ensure the form is submitted within a specific time window or by the same user
By generating unique form elements or creating session variables, you can ensure that the person that visits your form page is the same one that submits the form. For instance, when a form is accessed, you could use server scripting to write the current time to a hidden form element. When the form is submitted, you can compare the hidden form value with the current time and ensure that no more than, say, an hour has elapsed. The likelihood of a spambot generating the correct value for the time form value is very unlikely. You can also set browser cookies or use other client sessioning systems to ensure that a user session is established and maintained between the form page and the submission page.
The following will write the current time in UNIX time format to a hidden form input.
<input type="text" name="formtime" value="<?php echo time(); ?>" />
When the form is submitted, you can measure the difference between the current time and the value stored within the form. If the time difference is more than a specified value, you can flag it as spam. In this example, if more than an hour (3600 seconds) has elapsed between the time the form was viewed and the time it was submitted, the spam variable is set. This code will also flag the message as spam if the formtime value has been changed to some other value, such as a URL or an e-mail address.
if($_POST['formtime'] < time()-3600) {
$spam=true;
}
Create a multi-stage form or form verification page
By creating a multiple stage form process, most spambots will be unable to find the actual script that processes the final form data. This can be as easy as having the user verify their input after submitting a form and then selecting a second button to actually submit the form elements for processing. This can be made even more foolproof if the original form and the verification page are processed at the same URL. If the form element data is stored server side before the final verification step (rather than in hidden form elements that can be submitted by the spambot), it becomes very difficult for an automated system to submit the form.
if($formsubmitted == true) {
// database the form elements and display the verification page.
// If the user verifies the form information, then process the databased data.
}
else {
// display the empty form
}
Ensure the form is posted from your server
Because most spambots post to your form script from a remote computer, by detecting if the form information has been submitted from your own web site, you can stop many spambots from submitting the form to your processing script. Most scripting programs can check the page referrer, or the page that was used to get to the current page. It’s important to note that it is quite easy for spambots to forge the referrer information to appear as if the form is coming from your web server. Also, some browsers and firewalls will not send the referrer header at all.
The following code will check to ensure that the page referrer (incorrectly spelled ‘referer’ in the HTTP spec and in PHP) exists, and if it does, that the referring page is on the same web site as the form processing script. For browsers or spambots that send no referrer information, the message is never flagged as spam.
if((isset($_SERVER['HTTP_REFERER']) && stristr($_SERVER['HTTP_REFERER'],$_SERVER['HTTP_HOST']))) {
$spam=true;
}
Conclusion
Preventing spam submissions to web forms is difficult work. However, when possible we should not place the burden of preventing spam on the end user through CAPTCHA or other turing tests. Any time it becomes the user’s responsibility to somehow manually prove that they are a human, accessibility will be decreased. These techniques offer several methods of filtering out most form spambots without placing any burden on the end user.
I’m sure these are not all of the possibilities and it’s likely that there are flaws in my techniques above. If you have comments or better techniques, please post them below.
Thanks Jared, thats a a useful collection of techniques and its good to see them complied in one place.
Josh
Just in case someone figures out that you’re timestamping the submission form (or use any other dynamic content generation technique), generate MD5 or SHA hash of (dynamic content + server-side-secret), store it into a hidden field and check it on form submission (similar to what’s done in IPSec or any other digital signature method).
Also make sure your dynamic content prevents replay attacks (resubmitting of the same content); timestamps are usually the best solution.
Fine list of tips, thanks.
I have found, however, that testing $_SERVER[‘HTTP_REFERER’]) makes it impossible for users with Norton firewall or similar software that blocks communication of the referer value, from filling out the form!
As long as the firewall either blocks the HTTP_REFERER altogether or sends the correct referer, then there shouldn’t be a problem with this approach. This code only flags it as spam if the referer is present AND is not the same as the web site. However, if the firewall send an incorrect header (which it should never do), then there might be problems. I need to do some more looking into this.
Well, as I understand it, that makes the HTTP_REFERER approach pretty useless – a pirate isn’t going to send that, so the result will be the same as a firewall blocking that info. It’s really a shame, because otherwise this seemed to be the most elegant approach to this headache!
A colleague pointed me to this article, and it is pretty useful – thanks for the tips, which are in essence simple but wouldn’t always occur to web admins. For instance, checking for “http://” in the submitted data, which is obvious in retrospect but it just hadn’t occurred to me before.
sooo … what does one then do with $spam=true; ??
at the risk of irritating those who know it all, i respectfully request a short sample of how this fits in with an overall script. we’re not all admins; some of us are self-taught n00bs who, at best, learn by copy/paste.
That’s a good question PHPdummy. I’m actually working on an entire form processing script that includes these code options as well as several accessibility features for error recovery and validation. I hope to have things packaged up and will make them available soon.
In short, if $spam is true, then you want to alert the user of the problem, identify the form fields that were problematic, provide easy access to those fields to allow the user to fix the problems, then allow them to re-submit the form. So, that’s not a super easy copy/paste type of thing. đ But my forthcoming script will have all of this.
A very basic solution would be to use PHP echo to display the error message on the same page as the form. Something like…
if ($spam == true) {
echo(“<p>Your message appears to be spam and was not processed. Please remove all links, code, and other spam-like content from your message and resubmit the form.</p>”);
}
Will you be publishing the mail script on this blog?
This is a really great article, I’m tempted to list it on Digg… would that be ok? (Don’t want the Digg effect to affect the site adversely!)
Woody-
I just added the article to Digg – http://digg.com/programming/Spam_free_accessible_forms Please Digg there or at the top of the article.
I have a quick question, on my form submission page, which for me is insert.php, I want to know where does those if statements go, do they go anywhere inside of the Or does it have to go in a certain spot of the code Like at the end or the very beginning. I am very new at this and just got my form spammed and it is very frustrating. Can I also use multiple if statements within the php code as well, like for example can I use the first two examples you gave within the same php code area?
Let me know,
Carl (thanks)
I’m attempting ascertain why the following code won’t set the $spam flag if the following occurs:
if (preg_match( “/bcc:|cc:|multipart|[url|Content-Type:/i”, implode($_POST))) {
$spam=true;
}
if (!preg_match( “/bcc:|cc:|multipart|[url|Content-Type:/i”, implode($_POST))) {
$spam=false;
}
As far as I can tell, even putting bcc: or cc: into the form on purpose doesn’t set the flag to true. The onl time it occurs is if I reduce the string match to just somethinng like /bcc\:/i which is not good enough in this case. Am I missing some logic here? Thanks for any help that can be provided.
Carl-
The PHP code can be placed anywhere within your page. In this case, it should probably go toward the top of the page and then based upon what happens, you can then display an error message to the page. Something like…
if ($spam==true;) echo(“Your message appears to be spam.”);
And yes, you can use as many if statements as you would like.
Sorry, the last statement should say “elseif”.
Jared,
For some reason, i am having troubles getting any of the if statements to work, I am not sure if i am putting it to the right form, so if its not to much trouble I was wondering if you might be willing to see what I am doing wrong. Here is the code I am using on my page. I have people send their information from a questionnaire page, then it gets put into the database by using insert.php page. I am assuming that was the page you mentioned to be the form processing page. Here is the code i have on that page, see if you can see what I have done wrong.
3) {
$spam=true;
}
if((isset($_SERVER['HTTP_REFERER']) && stristr($_SERVER['HTTP_REFERER'],$_SERVER['HTTP_HOST']))) {
$spam=true;
}
if(!empty($_POST['email2'])) $spam=true;
if (!mysql_query($sql,$con))
{
die('Error: ' . mysql_error());
}
echo "Thank you for your submission. Please view the Directory to see your information, and your classmates as well.";
mysql_close($con)
?>
If you could debug this in laymans terms I would appreciate it.
Thanks Carl
Sorry the first part of my code didnt make the last post.
Here it is
$con = mysql_connect("p41mysql3.secureserver.net",$username,$password);
if (!$con)
{
die('Could not connect: ' . mysql_error());
}
mysql_select_db($database, $con);
$sql="INSERT INTO student_information (LastNameSchool, LastNameCurrent, FirstName, SpouseName, StreetAddress, City, State, Zip, PhoneArea, PhonePrefix, PhoneSuffix, EmailAddress, FamilyInfo, PersonalInfo, Retrospect, Photo)
VALUES
('$_POST[lastnameschool]','$_POST[lastnamecurrent]','$_POST[firstname]',
'$_POST[spousename]','$_POST[streetaddress]','$_POST[city]','$_POST[state]',
'$_POST[zipcode]','$_POST[areacode]','$_POST[prefix]','$_POST[suffix]',
'$_POST[email]','$_POST[familyinfo]','$_POST[accomplishments]',
'$_POST[retrospect]','_$POST[photo]')";
$spamwords = "/(list|of|naughty|words|here)/i";
if (preg_match($spamwords, implode($_POST))) {
$spam=true;
}
if (preg_match_all("/ 3) {
$spam=true;
}
if((isset($_SERVER['HTTP_REFERER']) && stristr($_SERVER['HTTP_REFERER'],$_SERVER['HTTP_HOST']))) {
$spam=true;
}
if(!empty($_POST['email2'])) $spam=true;
if (!mysql_query($sql,$con))
{
die('Error: ' . mysql_error());
}
echo "Thank you for your submission. Please view the Directory to see your information, and your classmates as well.";
mysql_close($con)
?>
Carl, you are correct in that this code is to be included on the PHP page that is used to process the submitted form information, but look at your code carefully.
You have condition statements that set the $spam flag to be true if it encounters any of those conditions. On its own however, the $spam flag is not much use. You have to use it as part of another condition statement that uses the value to either process the rest of the form or to stop it in its tracks if it encounters spam related content. You will need something like this:
if(!spam)
{
}
else
{
echo “Sorry, but your message appears to be spam.”;
}
So basically, you want to check to see if the $spam variable is true and if it is, not to bother running the other database processing scripts.
Man… The if section is blank? Anyway, within the if statement you see there, you put all your form/database processing scripts in there.
Corrected:
if(!$spam)
{
$con = mysql_connect(“p41mysql3.secureserver.net”,$username,$password);
… etc…
}
else
{
echo âSorry, but your message appears to be spam.â;
}
I tried using the statements to set the $spam variable, but keep getting:
Warning: preg_match() [function.preg-match]: Compilation failed: range out of order in character class at offset 32 in /home/.demitri/golem_herald/heralds.westkingdom.org/ProcessAwardInfo.php on line 64Line 64 is:
if (preg_match( "/bcc:|cc:|multipart|[url|Content-Type:/i", implode($_POST)))What is weird is that sometimes it seems to work in my testing of the script and web page. The error that I want displayed displays, and the email is not sent (the one the form is building). Any suggestions? (I am checking with the administrator of the server/website to see if he can point me at anything that might help as well). The warning is a bit disconcerting …
Ken-
It appears your installation of PHP is interpreting the – in “Content-Type” to be a range identifier. You should (haven’t tested to be sure) be able to fix this by escaping the – (e.g., Content\-Type).
Hmm. Well, I tried adding the backslash in there, and it’s still giving the warning … however, it’s now a little different:
Warning: preg_match() [function.preg-match]: Compilation failed: missing terminating ] for character class at offset 38 in /home/.demitri/golem_herald/heralds.westkingdom.org/ProcessAwardInfo.php on line 64(I appreciate the assistance!)
A bit more experimentation, and changing the statement at line 64 to:
if (preg_match( "/bcc:|cc:|multipart|[url]|Content\-Type:/i", implode($_POST)))MAY have fixed this. I am not sure. I do not understand regular expressions, so cannot guarantee it. But it may be fixed now. (I added the second square bracket after “url” in the line above)
(I don’t want to appear to be spamming your blog here …) The above posting is not quite correct, as it is now finding *all* posts that I try as spam. I am outputting the value of the $spam variable, and it’s coming out True, whether or not there’s a URL in the data. I am wondering if perhaps removing the “[url]” would work … Nope. That didn’t do it either. I am completely stumped here. Sorry …
This worked for me…
if (preg_match(“/bcc:|cc:|multipart|\[url|Content\-Type:/i”, implode($_POST)).
i.e. since we want “[url” we slash the “[” the same way that “-” is slashed.
Kia — I will try that. Hmm. Weirder and weirder. Now I get no syntax errors in the preg_match function, but I am getting:
Parse error: syntax error, unexpected '{' in /home/.demitri/golem_herald/heralds.westkingdom.org/ProcessAwardInfo.php on line 65All that is on line 65 is the { which I could have sworn was necessary …
(Thanks!)
I’m not real familiar with all the ins and outs of PHP, but …
FWIW, I gave up on the preg_match() function. It may simply be that there is some problem with the PHP interpreter being used on the server for the site I’m having this problem with. I did find a workaround, in case anyone is interested — it’s more lines of code, but it seems to be working fine:
if ( strpos( strtoupper( $_POST['SCAName'] ), "CONTENT-TYPE:" ) False ||strpos( strtoupper( $_POST['SCAName'] ), "A HREF=" ) False ||
strpos( strtoupper( $_POST['SCAName'] ), "HTTP:" ) False )
{
$spam="True";
}
The strpos() function returns false if a string (second parameter) is not contained in the larger string (first parameter). To avoid any case issues, I am using the strtoupper() function which converts the string to upper case, so I can compare the two only once. The three strings that seem
to be the biggest problems and appear in most of these spams are: “Content-Type:”, the “a href=” tag, and “http:”. The difficulty is that any entry area that can accept input must be checked, so this IF statement is repeated several times for my web form.
However, testing by filling the fields as they might normally be filled in, and testing them by using values in one of the spams I was getting, shows this is working.
I hope someone finds this useful …
Thank you so much for posting this. We’ve been getting slammed with spam and
I’ve been looking high and low for an alternative to CAPTCHAs.
Thank you! Thank you! Thank you!
I am very pleased with this list.
I was thinking about implementing a very simple SPAM check (together with regexp solution you suggested).
Here is how it works:
– Set a Session variable with a random nummber in it.
– Check on the confirmation page where the form is checked and processed if this Session corresponds with the Session set earlier on.
– If it doesn’t…
This could work or am I missing something?
Marky-
Session variables will work great, except for in the few cases where the end user blocks cookies. I don’t know that I’d set a session variable only for a form – it seems like a lot of overhead to me. But if you’re using session variables for anything else on the site, this approach would work great.
I agree with you on form validation, but the regular expression you suggest to validate e-mail addresses is infuriating: many savvy users (myself included) use e-mail addresses with ‘+’ in the address-part. With many mail systems, that allows a user to cause all messages to that address be sent to the folder named either before or after the ‘+’. Sites that don’t let me use this frustrate me, and I will often just avoid commenting/buying/whatever.
The ‘=’ is another possible separator of this type.
So, while validation is helpful, it’s important to think it through so it doesn’t alienate legitimate users.
I cannot figure out how to figure out the spam code if it is true. Do you have your article out yet that includes some code to try out? I cannnot get my condition to work our right if the spam is true.
This is what I have:
{
if(!empty($_POST[’email1′])) $spam=true;
{echo “You must enter a value for email1, otherwise your entry is viewed as spam.”;
exit;}
Nice to see some inspired php code once in a while. I would like to note a few things, however.
1. ALLWAYS validate form input from the real world, and always SCRUB user input before feeding it back.
2. Your hidden form field test, while accessible in spirit, does not strictly comply with section 508 (when interpreted most strictly). Yet another reminder that strict compliance is often not the point.
3. Good thinking about blocking on mail headers. The idea had crossed my mind, but I hadn’t come across a good reason to use it yet (non-RFC compliant mail gets trashed by our systems at work). I’ll probably throw it in just to see what I catch.
4. The PHP developers recommend using “preg_match” over “eregi” for speed of execution reasons (and frankly it is often just easier to write PERL-style regexes anyway).
5. Your e-mail checker is likely to block certian classes of legitimate mail addresses and my not survive internationalization of e-mail addresses (when they get done fighting about how to do it). Unfortunately a lot of us are going to have to deal with that. I looked to RFC 3696 (an informational) for the inspiration behind my also mildly flawed e-mail address checker.
6. Asking users to take one last look at their input before “final submission” is always a good idea for things which will be difficult to change. Unfortunately it can take a while to convince non-tech people of this. A link back to the original form (to edit the pre-selected input) is often a good idea when the review page is not also an edit page.
@ann:
Try this:
if(!empty($_POST[âemail1ââŹÂ˛])){
echo âYou must enter a value for email1, otherwise your entry is viewed as spam.â;
exit;
}
(and then finish the code so that it writes a valid page)
This was very useful. Thanks for taking the time to put this stuff together!
Will you be publishing the mail script on this blog?
This spam-free-accessible-forms tutorials gave lot of information to me.. i really
wish to say Thanks..!
I am very pleased with this list.
I was thinking about implementing a very simple SPAM check (together with regexp solution you suggested).
This is a nice script, as I’ve commented before, and thanks for Kia’s intervention on the PHP interpreter errors which helped. I’ve now got this working ok on a form on the site above. One thing I’ve noticed in recent spam, though, is that the spambots aren’t putting “http://” in any more, but “ttp://”, presumably in the hope that whatever client receives the email will interpret this as a URL anyway. An example is:
message_subject: Garrison chocolate Providence RI Nursehentai
So instead of detecting “http” in the script, perhaps it would be better to look for “href”? Mind you, the search for
My previous comment got a bit truncated on account of including naughty proto-HTML tags in it. The example spam didn’t show up well, and the last line should have referred to the regex search for the anchor tag. Sorry đ
Ok, I am stuck dealing with a Frontpage-created form (I know, I know). I can put the CSS code in there that hides the email entry box — but where would I put the following line?
if(!empty($_POST[’email’])) $spam=true;
The page is in .html and has Javascript for validating if the form fields are empty. But how would I use this one as a standalone script?
RvnPhnx notes that the hidden field approach is not strcitly 508 compliant. Why not?
Rick-
I don’t see any Section 508 issues with the hidden form field approach.
There are two 508 provisions that could apply. One says the page must be readable with style sheets disabled. In this case, the form field would be displayed – but with an adequate label (“Don’t enter anything in this textbox.”), I don’t see how this would be an issue. Besides, this provision deals with readability only, not forms navigation.
The second simply requires that forms be accessible to those using AT. This certainly is.
Jared,
I agree. Just checking to see if there was something i was missing.
This code only flags it as spam if the referer is present AND is not the same as the web site. However, if the firewall send an incorrect header (which it should never do), then there might be problems.
A colleague pointed me to this article, and it is pretty useful – thanks for the tips, which are in essence simple but wouldnât always occur to web admins.
Great article – simple, yet effective steps to tackle Spam.
Yet another simple technique i use: Send an email to the person who filled the form asking him to click on a link to confirm if he indeed submitted it. Serves to validate the email as well.