How To Deal With Bad Bots Coming To Your Website

How To Deal With Bad Bots Coming To Your Website

If #bots are inflating your site's bounce rate, there are several strategies you can implement to reduce their impact:

1. Block Known Bots via .htaccess or Firewall Rules

  • Identify common bot user agents (e.g., Qwantbot, SemrushBot, AhrefsBot) and block them in your .htaccess file or web server configuration.
  • Alternatively, configure a Web Application Firewall (WAF) like Cloudflare or AWS WAF to block or challenge bot traffic.

2. Use Robots.txt to Disallow Non-Essential Bots

  • Update your robots.txt file to discourage bots that don’t need to index your site. Here’s an example:
User-agent: Qwantbot
Disallow: /

  • Keep in mind, not all bots obey robots.txt directives, but legitimate search engines generally do.

3. Implement Bot-Detection Measures in Analytics

  • Many analytics platforms, including Google Analytics, have settings to filter known bots and spiders from your reports. This can prevent bot traffic from being included in your bounce rate calculations.
  • Enable "Exclude all hits from known bots and spiders" in Google Analytics (Admin > View Settings).

4. Rate-Limiting and CAPTCHAs

  • Apply rate-limiting to requests from specific IP addresses or use CAPTCHAs for certain actions (e.g., form submissions) to deter bots.
  • Some CAPTCHAs can be set to display only when traffic appears suspicious.

5. Log Analysis for Better Bot Identification

  • Regularly analyze server logs to identify high-frequency bot IPs or user agents, and consider blocking these at the server level.
  • Tools like AWStats or GoAccess can help you identify and filter out bot patterns.

6. Use Honeypots for Bot Detection

  • Implement honeypot fields in forms or pages (invisible to regular users) to trap bots. When bots interact with these hidden fields, you can flag and block them.

7. Adjust Bounce Rate Calculation Logic (if Customizable)

  • Depending on your platform, you might adjust how bounce rate is calculated. For example, trigger a small delay for engagement events (like scroll tracking) to more accurately capture real user engagement versus bot behavior.

By implementing these strategies, you should see a significant reduction in bot traffic, leading to a more accurate and potentially lower bounce rate.

A honeypot in a form is a hidden field that normal users won’t interact with because it’s invisible to them. Bots, however, tend to fill in every field they detect in a form, including hidden ones. When a bot submits a hidden field that a regular user wouldn’t, you can flag that request as suspicious or block it.

Here’s how you can implement a honeypot in PHP and MariaDB:

Step 1: Add a Hidden Field to Your Form

Add an invisible field to your form. Give it a name that sounds legitimate to a bot but is not critical for actual form submission.

<form action="submit.php" method="post">
    <!-- Normal form fields -->
    <input type="text" name="username" required>
    <input type="email" name="email" required>
    
    <!-- Honeypot field (hidden from normal users) -->
    <input type="text" name="phone" id="phone" style="display:none;">
    
    <button type="submit">Submit</button>
</form>

  • Field Styling: The honeypot field is hidden using CSS (style="display:none;") so users won’t see it, but bots that don’t account for CSS will still detect it.
  • Labeling: Avoid naming the honeypot field something obvious like “honeypot.” Use a generic name like phone, website, or nickname to make it blend with other form fields.

Step 2: Detect Honeypot Submission in PHP

In your PHP processing script (submit.php), check if this hidden field has been filled in. If it has, treat it as a bot submission.

<?php
// Connect to MariaDB
$mysqli = new mysqli("localhost", "username", "password", "database");

// Check connection
if ($mysqli->connect_error) {
    die("Connection failed: " . $mysqli->connect_error);
}

// Get form data
$username = $_POST['username'];
$email = $_POST['email'];
$honeypot = $_POST['phone']; // Honeypot field

// Check the honeypot
if (!empty($honeypot)) {
    // Likely a bot submission
    error_log("Bot detected - honeypot field filled.");
    die("Error: Submission blocked.");
}

// Continue with normal form processing
$stmt = $mysqli->prepare("INSERT INTO users (username, email) VALUES (?, ?)");
$stmt->bind_param("ss", $username, $email);
$stmt->execute();
$stmt->close();

echo "Form submitted successfully!";
?>

  • Logic: If $_POST['phone'] contains any data, it’s likely a bot. You can choose to log the attempt, block it outright, or redirect the user.
  • Database Logging (Optional): Log bot attempts by inserting them into a “blocked_requests” table if you want to monitor patterns or IP addresses associated with bot traffic.

Step 3: Add Additional Measures (Optional)

To strengthen this approach, consider:

  1. Timestamp Field: Add a hidden timestamp field to track how long the form takes to submit. Bots tend to fill out forms instantly, while users take a few seconds at least.

    • Add a hidden timestamp field in the form when it loads.
    • In the backend, check if the form was submitted within an unusually short time frame.

  2. JavaScript Validation (Basic Bots): Load the form field dynamically using JavaScript. Bots that don’t process JavaScript may ignore this field, reducing interaction with honeypot fields.

Step 4: Review Honeypot Submissions in Database (Optional)

Create a MariaDB table to store attempts by bots:

CREATE TABLE blocked_requests (
    id INT AUTO_INCREMENT PRIMARY KEY,
    ip_address VARCHAR(45),
    user_agent TEXT,
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);

Log bot submissions by capturing the IP address and user agent:

// Log bot details in database
if (!empty($honeypot)) {
    $ip_address = $_SERVER['REMOTE_ADDR'];
    $user_agent = $_SERVER['HTTP_USER_AGENT'];

    $stmt = $mysqli->prepare("INSERT INTO blocked_requests (ip_address, user_agent) VALUES (?, ?)");
    $stmt->bind_param("ss", $ip_address, $user_agent);
    $stmt->execute();
    $stmt->close();

    die("Error: Submission blocked.");
}

This approach will help you track bot activity and reduce bot-related interactions, which should lower the bounce rate on your site.



Sign Up To Comment