There has been much discussion lately about how to prevent spambots from submitting forms on web sites. Many solutions have been presented, many of which impact the usability and accessibility of the web page. CAPTCHA is a classic case where the user and accessibility is directly impacted.
Note
A Brazilian-Portuguese translation of this blog entry is available at http://www.maujor.com/tutorial/spam-em-formularios.php.
Over the last year or so I have compiled the following basic techniques for blocking spam submission in web forms. I’ve implemented just a couple of these and through logging have found that they have effectively reduced around 99% of spambot submissions while having no or very little impact on the usability or accessibility of the forms. Nearly all of these techniques are performed server-side using PHP and the relevant PHP code is shown below, however, the tests can be readily implemented in nearly any server-side scripting language.
Disclaimer 1: These spam prevention techniques may not work for enterprise level application where spammers may target forms specifically. They are intended for generic contact, comment, or registration forms where a spammer is less likely to take the time to try and bypass your specific spam prevention mechanisms.
Disclaimer 2: These techniques primarily stop bots and automated spam submission programs. They also can filter certain content. However, they likely will not prevent an actual dedicated human from posting spam to your web site.
The techniques are:
- Detect spam-like content within submitted form elements
- Detect content within a hidden form element
- Validate the submitted form values
- Search for the same content in multiple form elements
- Generate dynamic content to ensure the form is submitted within a specific time window or by the same user
- Create a multi-stage form or form verification page
- Ensure the form is posted from your server
Detect spam-like content within submitted form elements
This technique is likely the most powerful spam prevention technique. Most spam bots are in existence to either post URL’s of web sites in an effort to increase traffic or increase their search engine ranking or they are attempting to hijack your form to send spam messages to you or others. Detecting commonly used spam content or e-mail header injections will stop nearly all spam bots dead in their tracks.
The following PHP code, when placed on your form processing page (the place where the form is submitted to), will search all of the form elements for the most common header injections and other code that may trick your mail processor into sending carbon copy or blind carbon copy messages to others. It also detects any content that includes the string “[url” which is used by most forum software to specify links. If any are found, it sets the $spam variable to true.
if (preg_match( "/bcc:|cc:|multipart|\[url|Content-Type:/i", implode($_POST))) {
$spam=true;
}
NOTE: Internet Explorer 6 has a bug that will not allow proper overflow of preformatted text. If you are still using that browser, you will need to properly reflow the PHP code lines from this page.
You can also detect links and urls within the form elements. The following will set the $spam variable if more than 3 instances of “<a” or “http:” appear anywhere within the form.
if (preg_match_all("/<a|http:/i", implode($_POST), $out) > 3) {
$spam=true;
}
This will defeat most spambots as they primarily focus on posting links or hijacking your mail script. Beyond this, some very basic word filtering can often catch spam that finds its way through.
$spamwords = "/(list|of|naughty|spam|words|here)/i";
if (preg_match($spamwords, implode($_POST))) {
$spam=true;
}
You can also use external spam detection services with up-to-date patterns of spam content. My favorite is Akismet. Akismet is commonly used for filtering spam on blog comments (it has blocked nearly 14,000 spam comments to this blog in the last 9 months!), but it can be used successfully for nearly any web form.
Detect content within a hidden form element
Most spambots will find your form, determine what the form element names are, and find the URL where the form is posted to. The software will then post those form elements with modified, spam-filled values back to the form submission URL. Typically, the bot will populate every form element with some value so as to best ensure that it will succeed in being posted. So, if you insert a standard text input element into your form, but hide it visually from the user so the user cannot enter anything into this field, it is quite likely that the spambot will still post some value for this form element. If you detect that the form element is submitted with a value, then it’s almost certainly a spambot.
For instance, your form element may be inserted as
<span style="display:none;visibility:hidden;"> <label for="email"> Ignore this text box. It is used to detect spammers. If you enter anything into this text box, your message will not be sent. </label> <input type="text" name="email" size="1" value="" /> </span>
Notice that CSS is used to hide the text input and its label from view. This code will also hide this content from modern screen readers. However, if CSS is disabled, the input will still be displayed. For this reason, an explanatory label is provided that informs the user to not enter anything into the text box. I also gave the input a nice, juicy, tempting element name of “email” – that’s almost certain to get the spambots to enter a value.
You then simply detect if the form element is empty. If it is not, then it’s either a spambot or a user that has CSS disabled and did not follow the label instructions.
if(!empty($_POST['email'])) $spam=true;
This tactic, like all of those listed here, should still present a useful, informative error message in case the user somehow triggers your spam detection flag.
Validate the submitted form values
This one perhaps goes without mentioning, but if you want certain form elements to be required, ensure that you are using a server-side script to detect if information has been entered into those form fields. If you require form information is a particular format (such as requiring a valid e-mail address), then validate it. Many bots will simply submit empty information for fields they do not recognize or will submit random information for certain form fields. Your standard form validation mechanisms can stop many spambots.
// If the message is empty, throw an error
if(empty($_POST['message'])) $error=true;
// if e-mail is not formatted correctly, show error message
if(!eregi("^[_a-z0-9-]+(\.[_a-z0-9-]+)*@[a-z0-9-]+(\.[a-z0-9-]+)*(\.[a-z]{2,3})$", $_POST['email']))) {
echo "Please enter a valid email address.";
}
Search for the same content in multiple form elements
Some spambots will post the same text into all unrecognized form fields. If you have two form fields that should never contain the same information, you can detect if their values are indeed the same and if they are, you can flag an error. On our forum registration form, I found that simply throwing an error if the first and last names were the same cut down on bot registrations by around 80%. It’s not a perfect technique and you should ensure that the fields you analyze should always be unique (I guess there is still a chance that a person could have the same first and last name, huh?).
if($_POST['firstname'] == $_POST['lastname']) $spam=true;
Generate dynamic content to ensure the form is submitted within a specific time window or by the same user
By generating unique form elements or creating session variables, you can ensure that the person that visits your form page is the same one that submits the form. For instance, when a form is accessed, you could use server scripting to write the current time to a hidden form element. When the form is submitted, you can compare the hidden form value with the current time and ensure that no more than, say, an hour has elapsed. The likelihood of a spambot generating the correct value for the time form value is very unlikely. You can also set browser cookies or use other client sessioning systems to ensure that a user session is established and maintained between the form page and the submission page.
The following will write the current time in UNIX time format to a hidden form input.
<input type="text" name="formtime" value="<?php echo time(); ?>" />
When the form is submitted, you can measure the difference between the current time and the value stored within the form. If the time difference is more than a specified value, you can flag it as spam. In this example, if more than an hour (3600 seconds) has elapsed between the time the form was viewed and the time it was submitted, the spam variable is set. This code will also flag the message as spam if the formtime value has been changed to some other value, such as a URL or an e-mail address.
if($_POST['formtime'] < time()-3600) {
$spam=true;
}
Create a multi-stage form or form verification page
By creating a multiple stage form process, most spambots will be unable to find the actual script that processes the final form data. This can be as easy as having the user verify their input after submitting a form and then selecting a second button to actually submit the form elements for processing. This can be made even more foolproof if the original form and the verification page are processed at the same URL. If the form element data is stored server side before the final verification step (rather than in hidden form elements that can be submitted by the spambot), it becomes very difficult for an automated system to submit the form.
if($formsubmitted == true) {
// database the form elements and display the verification page.
// If the user verifies the form information, then process the databased data.
}
else {
// display the empty form
}
Ensure the form is posted from your server
Because most spambots post to your form script from a remote computer, by detecting if the form information has been submitted from your own web site, you can stop many spambots from submitting the form to your processing script. Most scripting programs can check the page referrer, or the page that was used to get to the current page. It’s important to note that it is quite easy for spambots to forge the referrer information to appear as if the form is coming from your web server. Also, some browsers and firewalls will not send the referrer header at all.
The following code will check to ensure that the page referrer (incorrectly spelled ‘referer’ in the HTTP spec and in PHP) exists, and if it does, that the referring page is on the same web site as the form processing script. For browsers or spambots that send no referrer information, the message is never flagged as spam.
if((isset($_SERVER['HTTP_REFERER']) && stristr($_SERVER['HTTP_REFERER'],$_SERVER['HTTP_HOST']))) {
$spam=true;
}
Conclusion
Preventing spam submissions to web forms is difficult work. However, when possible we should not place the burden of preventing spam on the end user through CAPTCHA or other turing tests. Any time it becomes the user’s responsibility to somehow manually prove that they are a human, accessibility will be decreased. These techniques offer several methods of filtering out most form spambots without placing any burden on the end user.
I’m sure these are not all of the possibilities and it’s likely that there are flaws in my techniques above. If you have comments or better techniques, please post them below.
I cannot get any of these to work properly. I either get undefined variable
errors with the preg_match example or parse erros with the strpos example.
I am just so new at php, that I really need to see a full example in context
to make things work.. even if it’s a simple page. Does any one have that?
Well, as I understand it, that makes the HTTP_REFERER approach pretty useless – a pirate isn’t going to send that, so the result will be the same as a firewall blocking that info. It’s really a shame, because otherwise this seemed to be the most elegant approach to this headache!
If you are using frontpage forms, simply make one of the fields required. This will at least prevent blank forms.
Ken (and others),
I use this for catching the errant spam character.
foreach($_POST as $key => $val) {
if ($key != ‘message’) {
if (stristr($val,’
‘)) $spam++;
if (stristr($val,’
‘)) $spam++;
if (stristr($val,’%0A’)) $spam++;
if (stristr($val,’%0D’)) $spam++;
}
if (stristr($val,'<a’)) $spam++;
if (stristr($val,’content-type’)) $spam++;
if (stristr($val,’mime-version’)) $spam++;
if (stristr($val,’cc:’)) $spam++;
}
…or some variant thereof in most of my contact forms. I’ve also started to employ the use of the occasional hidden-by-css field (that is not always in the same place, in conjunction with random field names), and something like sha1(md5(“”.gmtime().mt_rand(9999,99999))). Call me paranoid if you wish, but I ain’t gettin’ no spam (of course, there is probably the occasional legit message not getting through)!
looky there. should have escaped the newline chars. sorry folks. those were \r and \n
Great script, simple and easy to integrate. Thanks a lot!
Quite good. I use some similar ideas. (If == yes then is spammy for the following examples.)
TIME CHECK
On form page:
$start_time = $_SESSION[‘start_time’] = time();
In processor script:
$timeSubmit = time();
if ($timeSubmit – $_SESSION[‘start_time’] < 5)
IP CHECK
On form page:
$ip1 = $_SESSION[‘ip1’] = $_SERVER[‘REMOTE_ADDR’];
On processor page:
$ip2=$_SERVER[‘REMOTE_ADDR’];
if ($_SESSION[‘ip1’] !== $ip2)
HTTP CHECK
On processor page:
if ( (preg_match(“/http/i”, $name)) || (preg_match(“/http/i”, $subject)) || (preg_match(“/http/i”, $message)) )
Using techniques like these along with the usual CSS-driven bogus captcha fields, ordinary validation, etc., it’s still possible to build an accessible form that is almost bulletproof.
There are many other techniques, as well. None are particularly difficult. They just require thinking about the goals of a spammer and the differences between human and robot behavior, and condensing that information into code.
One thing I heartily recommend is to land a spambot on the exact same success page that a successful submission would land on, to avoid triggering human review of the failure. Just send the bot to the success landing page, but kill the script before the mail is sent. For example:
if (is spammy)
{
print “”;
die;
}
Best,
Richard
Copy and pasted from above.
For instance, your form element may be inserted as
Ignore this text box. It is used to detect spammers.
If you enter anything into this text box, your message
will not be sent.
1. Can the about be put in a file?
———————————–
Copy and pasted from above.
You then simply detect if the form element is empty. If it is not, then it’s either a spambot or a user that has CSS disabled and did not follow the label instructions.
if(!empty($_POST[’email’])) $spam=true;
This tactic, like all of those listed here, should still present a useful, informative error message in case the user somehow triggers your spam detection flag.
2. Should this [if(!empty($_POST[’email’])) $spam=true;] be put in a javascript, a php script, or a perl script?
If it is true it is spam, then send it to a different action url such as <form name=”Spam” action=”http://www.url.com/cgi-bin/spam.cgi
or a redirect script
Hello all! Yes, the ‘spam’ topic is so annoying for me also – these letters coming in churns and all the things – it was horrible for my work.. I used Barracuda (too expensive), then SpamAssasin (not really that effective), then Postini but the same result and now I’m trying to use Gafana.com – having a trial period but it sounds like a good service – no spam at all, AT ALL!!!! No false positives either. Ok with the price. So, I’m inclined to have a long-term relationship with it. Ifanyone of you has any suggestions, please write them, would be really interesting!
I’m a spammer.. ahah!! and i’m posting here my spam message! XD
seriously.. really interesting post.. i think i’m gona use theese lines of code in my site..
First of all, thanks for helping stop the spam problems out there today. I am not a php programmer but do use phpformmail for my clients forms. So I pasted the following line (the first one you offer at the top of this page:
——
if (preg_match( “/bcc:|cc:|multipart|\[url|Content-Type:/i”, implode($_POST))) {
$spam=true;
}
——
Here’s my dumb question: This code flags the message as spam but what else do I have to do? How does formmail know what to do when $spam=true; is detected? Is there more code necessary to tell formmail what to do when $spam=true; Something like “if this is spam do this.”
OR…is this covered in the “implode ($POST))) statement?
Hope this makes sense and thanks again.
Fred
FredB-
This assumes that you will have some other logic later in the file that checks for the spam variable and displays an error message if it is true and sends the e-mail (or whatever) if it is not true.
Something like:
if ($spam==true) {
echo(“I’m sorry, but this message appears to be spam.”);
}
else {
// send the e-mail message and show a success message.
}
Thanks Jared, I’ll add that statement to my php file.
I just used the “hidden field” method to great success. Instead of a blog post form, this one actually sent an email to my client AND a harvested email address. After implementation, spam immediately ceased (after having 90,000 in a week, my client was very happy to stop receiving the).
I included a message for disabled CSS users stating not to fill in the field, or enter the word “human”.
(Since the site is for the Disabled American Veterans, it was extremely important for the fix to be accessible.)
A combination of Captcha and email activation is most effective, as it is too troublesome for the mass offenders, and even the minor offenders. Serious commenters will give real email addresses and will not mind, proud that their important contribution is recognized.
Hi, i was wondering if anyone can help me; I have tried putting the following on our form processing page to stop spammers leaving links in our posts;
if (preg_match(“/bcc:|cc:|multipart|\[url|Content\-Type:/i”, implode($_POST))) {
$spam=true;
}
if ($spam == true) {
echo(“Your message appears to be spam and was not processed. Please remove all links, code, and other spam-like content from your message and resubmit the form.”);
}
but we are still receiving tons of spam which just link to other websites, can anyone tell me how to stop people posting website links on our site. I am also using recaptcha but that doesnt seem to be stopping them either.
I am fairly new to php so maybe i have entered something wrong??? – the spam links are driving me insane, can someone please help.
Regarding the section:
Generate dynamic content to ensure the form is submitted within a specific time window or by the same user.
The dynamic content could be made secure using:
$token = hash_hmac(‘md5’, $_SERVER[‘REQUEST_TIME’], ‘secret_string’). $_SERVER[‘REQUEST_TIME’];
Where secret_string is a sufficiently random string. MD5 is still ok for HMACs.
When someone or something posts, you compare the hash and time value. There is no way you can fake the HMAC, which means the bot MUST fetch your page before posting. You can enforce minimum and maximum times since a human isn’t going to be able to post as fast as a bot.
Although a bot can still be made to download and parse your page, it is about as much as you can do without causing accessibility problems for humans.
Hi,
Thanks for the scripts.
I have a form in a contact page(contact.php) with validation code in every fields.
This form call another page (send_contact.php):
where I get the variable posted at the first one (ex. $HTTP_POST_VARS[“name”], $HTTP_POST_VARS[“email”], $HTTP_POST_VARS[“message”], etc…
I have tried putting your hint at send_contact.php:
if (preg_match( “/bcc:|cc:|multipart|\[url|Content-Type:/i”, implode($_POST))) {
$spam=true;
}
if ($spam==true) {
echo “Sorry, but your message will not be delievery.”;
}
else {
require “includes/send_contact.php” ;
}
But I still receiving messages with html code…
Please, Where is the error?
Regards,
Great article!
I had a contact form and a networking form on my web-site, but I had to take them down because they were swamped with spam attacks.
Captchas are not a solution for me because they place too much burden on the user.
So now I plan to implement all the ideas in this blog. I expect to see a dramatic reduction in spam…
You mention that you planned to release a full-blown script. Did you ever do this, and if so where can I download it?
Regards
RobD
Hey Jared,
Good article mate – it’s definately pick of the bunch….found it very practical and on the mark.
FYI, it inspired me to write a ASP version using VBScript Regular Expressions. Can be found here :
4 Tips on How to Stop the Blog Spam monster
To tigra and Luiz Araujo
The code you show in your posts only filters spam submissions that include e-mail headers. To filter HTTP links you need to add this code:
if (preg_match_all(“/<a> 3) {
$spam=true;
}
This code counts the total number of occurrences of “<a” and “http:” across all the fields in your form. You can decide what number you consider to be an indicator of spam. In the example above, the spam flag is set when more than 3 occurrences are detected.
Sometimes certain fields in your form are expected to contain “<a” or “http:” (for example, “your website”), while other fields should never contain these tags. In this case it probably makes sense to do the check on a field-by-field basis. Be aware that scanning “implode($_POST)” will scan all the form fields indiscriminately.
Problem posting code in my previous post.
If this does not show up correctly, look for the section in Jared’s article entitled “Detect spam-like content within submitted form elements” and add the second code sample which checks for occurrences of “<a” and “http:”
Trying to paste again here from page source:
if (preg_match_all(“/<a|http:/i”, implode($_POST), $out) > 3) {
$spam=true;
}
Great article! However I think I might have just found a slight floor with the ‘detecting input in hidden fields’ trick. If people are using an automatic form filler that populates a field entitled “email” with their address, and that input is one of our hidden spam trap ones, then they will unwittingly submit a form that will never validate as non spam. Even if it’s visually hidden using CSS, and written warnings are given not to enter stuff into these fields, an auto form filler isn’t going to know that! I love the idea though so I’d love to hear some slightly different takes on it.
I hadn’t considered the potential problems with automatic form fillers inserting content in the hidden form field. I think the solution is to simply name the field something other than email. Most bots will put something in all of the form fields regardless of what they are names. Naming it “email” just makes it much more tempting for them to do so.
We create websites and none of us are programmers (more of artists). So we don’t know php. We have been using a form2email.php script for a few days to have an easy contact form on our site.
I started with
td class=”shead”>Enter the number 777 to help us avoid spam:
and then
I almost immediately began getting spam.
After reading through this I have added a line to the form2email.php file (and renamed it, since the spammers already have that filename) as follows:
if($_POST[‘AvoidSpam’] != ‘777’) {$errors[] = “Spam catcher must equal 777 – Please hit Back and try again”;}
The form2email.php already has some error checking in it so this will also prevent it from getting sent, at least in my testing – we’ll see if the spam robots can use it!
I created my original form using a helpful tutorial from here http://dreamweaverspot.com/adobe-dreamweaver-tutorial-contact-forms/ which also has a helpful form2email.php to download – note that it is susceptible to spam though.
The above eliminated a lot of my html, I’m going to take out brackets and other stuff so you can still read it but it will not be as helpful:
I started with
td class=”shead”> -label- Enter the number 777 to help us avoid spam:
-td–label–input name=”AvoidSpam” type=”text” class=”redtable” id=”AvoidSpam” tabindex=”10″ size=”50″ maxlength=”3″-
-/label–/td-
and then
-input name=”SendRequest” type=”submit” id=”SendRequest” onClick=”MM_validateForm(‘Name’….,’AvoidSpam’,”,’RinRange776:778′);return document.MM_returnValue” value=”Send Request”-
Fingers crossed, lets try again…
I have managed to kill spam at almost 100% level combining a few techniques.
Have a look:
http://netwings.co.uk/blog/
thanks for the scripting, start with it right away, would be really nice to get this thing working.
regards
Another issue I have encountered periodically using the hidden form field method, on top of auto fill is tabbed entry. If I just tab to each form entry and put in data, and do it without paying attention, it will end up in the hidden filed and an entry will be made. tab-order won’t help ’cause the hidden filed is already at the bottom of the form. Doesn’t happen often, but it can …
Rick-
If the form field is hidden with display:none and/or visibility:hidden in CSS, then it will not be placed in the tab order in any major browser. There will be no way of tabbing to it. Of course this is only the case if CSS is enabled – thus the big explanatory label to not type anything into it.
Captchas are pretty bothersome, but they do work. I’m not crazy about the ones Google uses that are hard to decipher.
NO ROCKET SCIENCE REQUIRED. Look folks. All this hoopla over a problem with a very simple obfuscation solution on the CLIENT side. The only caveat is that javascript is required. Here’s how it works: in the tag, use a fake url for the action, say action=”http://google.com”. (Or http://irs.gov but send them away from your site) Then on the submit button say: Using this script will ensure the real form processor is used in a reasonably obfuscated way:
function revealRealURL(fld){
fld.form.setAttribute(“action”,”ht” + “tp” + “:”+”/”+”/yoursite” + “.” + “yourscript” + “.” + “pl”)
}
Bots that simply harvest and store your form info (that’s virtually all bots) will end up never even accessing your script, since it is not even in the form tag where they are looking. Now you save your server hundreds or thousands of hits to your script.
If you’re worried about the nearly non-existent users that fill out forms without javascript, then change the fake url to a real static page on your own site informing them that javascript is required.
(corrected) NO ROCKET SCIENCE REQUIRED. Look folks. All this hoopla over a problem with a very simple obfuscation solution on the CLIENT side. The only caveat is that javascript is required. Here’s how it works: in the form tag, use a fake url for the action, say action=”http://google.com”. (Or http://irs.gov but send them away from your site) Then on the submit button say: input type=”submit” value=”Bots not Allowed” onClick=”revealRealURL()” Using the following script will ensure the real form processor is used in a reasonably obfuscated way:
function revealRealURL(fld){
fld.form.setAttribute(“action”,”ht” + “tp” + “:”+”/”+”/yoursite” + “.” + “yourscript” + “.” + “pl”)
}
Bots that simply harvest and store your form info (that’s virtually all bots) will end up never even accessing your script, since it is not even in the form tag where they are looking. Now you save your server hundreds or thousands of hits to your script.
If you’re worried about the nearly non-existent users that fill out forms without javascript, then change the fake url to a real static page on your own site informing them that javascript is required.
The above method also requires deleting any pre-existing compromised scripts. Your obfuscated script name must be fresh and un-compromised – remembering that bots almost always submit using previously acquired field names and action urls. ANY method that uses client side changes to forms must also change the script url simultaneously, or it may appear that the changes are not effective.
Shouldn’t the test for HTTP_REFERER use !stristr since the goal is to trap and mark as spam those instances where HTTP_HOST is different to HTTP_REFERER?
if((isset($_SERVER[‘HTTP_REFERER’]) && !stristr($_SERVER[‘HTTP_REFERER’],$_SERVER[‘HTTP_HOST’])))
A great article. I plan to add some of your traps to help stop spam altogether. Along with using a reasonably good form (formmail.php) I use a hidden field to trap bots, and a couple of different methods to stop bots.
The best method I have found is to send forms to $_SERVER[‘PHP_SELF’] and have the form processor load to the form page only if the submit button has been clicked. No submit action, no formmail.php. The processor is in a random number directory with htaccess deny and a php trap that exits the form if somehow it is accessed directly. Almost zero spam received.
Big thank’s for the scripting, start with it right away, would be really nice to get this thing working.
Regards
James
I precisely wanted to thank you so much once more. I am not sure the things that I would’ve undertaken without the actual concepts provided by you concerning that problem. It became an absolute intimidating case for me personally, but considering the expert form you solved that took me to weep for gladness. Now i’m happier for the support and as well , trust you are aware of an amazing job that you are providing training people today by way of your site. I’m certain you’ve never encountered any of us.
Thanks you for this great hint to prevent comment spam. I will try it out. I search for a solution who sends me an email as soon as a spammer will fill out hidden formfields.