networkshard

FlashCrawler v2.0 — The Hacker’s Browser-Powered, JavaScript-Crunching Web Crawler

·5 min read
toolscrawlerjavascript

This write-up has been prepared under the guidance of Amish Patel, Lay Patel at Hacker4Help as part of our learning initiative on cybersecurity awareness.

Built for hackers, bug bounty hunters, OSINT wizards, and anyone who’s ever screamed at a JavaScript-heavy webpage.

So there I was, manually copying URLs from websites like some kind of digital caveman, when my friend Kaif casually drops this bombshell: “Oh, I built a web crawler that can do that in seconds.”

Enter Flash_Crawler — a tool that made me realize I’ve been living my entire developer life wrong.

What Even Is This Thing?

Flash_Crawler is basically that friend who remembers everyone’s name at a party, except instead of people, it remembers every single URL on a website. And instead of awkward small talk, it uses JavaScript-aware crawling with Playwright to actually understand modern websites.

You know how most web scrapers are like that person who shows up to a JavaScript conference and asks “What’s React?” Flash_Crawler actually gets it. It runs a real browser (Chromium), executes JavaScript, waits for pages to load properly, and then methodically discovers every nook and cranny of your target website.

So basically FlashCrawler v2.0 is a Python + Playwright-based crawler that:

  • Acts like a real browser (yes, it runs JavaScript)
  • Grabs endpoints from deep within script tags
  • Detects and deduplicates parameter-based URLs
  • Supports random user-agents (because websites judge bots too harshly)
  • Looks good doing it (thanks, Rich library 👑)

🐛 Bug Hunters Be Like:

“Okay, I found a target domain… now where’s the juicy stuff?”

Modern websites are script-driven nightmares. HTML-only crawlers? They die in silence.

FlashCrawler, on the other hand, shows up with Playwright, loads JavaScript, and finds links normal crawlers can’t even see.

🎯 Why You’ll Love FlashCrawler

⚙️ Quick Setup

  1. Clone the repo:
git clone https://github.com/SKaif009/Flash_Crawlercd Flash_Crawlerpip install -r requirements.txt

2. Run it on a target:

python FlashCrawler.py -u https://example.com --save --random-agent

Pro tip: Add -d 50 -t 2 for deeper scans with polite delays.

Output You Can Actually Use

When --save is enabled, FlashCrawler organizes your recon loot:

results/├── found_urls.txt             ← All discovered URLs├── found_parameters.txt       ← URLs with query params└── deduplicate_params.txt     ← Clean param key-based signatures

No messy CSVs. Just plain, hacker-friendly text.

😂 Sites When They See FlashCrawler Coming:

But seriously, it behaves nicely — if you use polite settings (-t 2 adds 2s delay).

🚀 Try It Now

Grab it from GitHub:

👉 https://github.com/SKaif009/Flash_Crawler

Star it. Fork it. Break it. Improve it.