AI Crawlers Overloading Your Website? How to Stop Them Fast
AI crawlers overloading your server? Find the bot in your logs, block it at the server for instant relief, then use robots.txt and 429 responses long term.
Guides
Practical guides on websites, email, security and AI automation, picked for web developers.
AI crawlers overloading your server? Find the bot in your logs, block it at the server for instant relief, then use robots.txt and 429 responses long term.
How to get cited by Claude: allow Claude-SearchBot, write quotable answer-first pages, keep facts current, and track Claude referral traffic in GA4.
WordPress 7.1.4, released October 28, 2026, fixes 22 vulnerabilities including an update race condition and many stored XSS bugs. What to check and update now.
Bing AI training and grounding preferences replace NoArchive and NoCache from February 1, 2027. The new tags, the March 1 deadline and what to do now.
Google and Microsoft said on October 30, 2026 they will honor a shared AI training robots.txt signal, __usage_ai-training. What it does and how to prepare.
A developer's guide to using a site audit API: queue scans, poll or use webhooks, verify HMAC signatures, handle rate limits and run crawls and batches safely.
What a white label SEO report should include, how agencies use white label website audits to win and keep clients, and the honesty rules that keep them credible.
How to get automated website reports by email: scheduled audits, Search Console summaries and uptime alerts, what each should contain and how often to send.
JSON vs YAML compared: syntax, comments, data types, where each is used, the YAML 1.1 yes/no trap, indentation errors and how to convert between them safely.
PX vs REM vs EM explained: how each CSS unit is calculated, why rem respects users' font settings, when px is still right, and a quick conversion table.
Cron expressions explained: the five fields, ranges, lists and steps, ready-made examples, time zone and daylight saving traps, and how GitHub Actions differs.
Let's Encrypt scheduled 64-day staging certificates from October 14, 2026, before they become the default on February 10, 2027. How to check your renewals.
WordPress 7.2 Beta 1 arrived on October 20, 2026, ahead of a December 8 release. What changes for editors, admin speed and developers, and how to test safely.
Drupal fixed a critical Field UI remote code execution flaw on October 14, 2026, rated 19/25. Which versions are safe, the stopgap and a patch checklist.
A website launch QA process for agencies: staging freeze, automated checks, triage, client sign-off, go-live window and post-launch rechecks, with clear gates.
Cut image weight without losing quality: when to use WebP or AVIF, how to write srcset and sizes, when to lazy load, and how to compress sensibly.
The HTTP security headers every site should send, what each one does, and ready-to-paste configs for Nginx, Apache and Cloudflare Transform Rules.
A practical website launch checklist covering DNS, HTTPS, redirects, SEO, email authentication, security headers and post-launch checks, with copy-paste config.
Eight eMudhra intermediate CA keys were found to be weak in October 2026 and browsers pulled trust. How to check whether your SSL certificate is affected.
PHP fixed two critical remote code execution flaws on 21 October 2026. Which versions are safe, what the NCSC advises and a patch checklist for UK websites.
JavaScript regex cheat sheet: character classes, quantifiers, groups, lookarounds, flags, UK postcode, US ZIP and phone patterns, and avoiding backtracking.
A Unix timestamp counts seconds since 1 January 1970 UTC. Learn seconds vs milliseconds, the 2038 problem, handling BST and time zones, and ISO 8601 dates.
How to format JSON so it's readable, validate it, and fix the most common JSON errors: trailing commas, single quotes, comments, unescaped characters and more.
How to run website audits automatically on every deploy and on a schedule with Lighthouse CI, the PageSpeed Insights API, audit APIs and signed webhooks.
Learn Content Security Policy step by step: start in report-only mode, use nonces or hashes, and fix the inline scripts and widgets that usually break.
301 vs 302 vs 307 vs 308 redirects explained: how Google treats each, when to use them, and how to set them up in .htaccess, Nginx, Cloudflare and Shopify.
Free checklist (PDF)
31 practical checks for web developers: website speed and mobile, Google Business Profile and reviews, email security, booking and enquiry handling, and AI search visibility. Each one says why it matters, and many link to a free tool that checks it.
Agencies spend their days making other businesses look and work better online. Their own projects deserve the same care: content that arrives, feedback that makes sense and live sites that stay quietly healthy long after launch day.
Usually a reply within one business day. No sales sequence, no obligation.
Make It Flow assistant
Tools, guides and AI automation
Hi! I can find the right free tool, explain an audit result, or tell you how Make It Flow can automate any part of your business. What do you need?
Book a free 15-minute call
Pick a time that suits you. Times are shown in your time zone ().
A Make It Flow agent will confirm the call by email within 24 hours.
AI answers can be wrong. Chats are stored so the team can follow up. Privacy