MCP CheckClaude skills for web scraping
The 50 strongest of 368 web scraping skills on GitHub. Skills built for web scraping come first; no repository gets more than 3 spots. Each links to its folder; copy it into ~/.claude/skills/ to use it.
firecrawl search
Web search and page scraping via Firecrawl, returning full-page markdown. Use when user says "firecrawl search", "firecrawl scrape" or "/firecrawl-search".
wanshuiyin/Auto-claude-code-research-in-sleep★ 18kMITFolder:
firecrawl-searchgoogle maps scraper
Find businesses, leads, emails, reviews, ratings, and contact details from Google Maps. Use for requests such as "find dentists in Berlin", "scrape Google Maps", "get local business leads", or "collect Google Maps reviews". Runs the open-source scraper…
gosom/google-maps-scraper★ 7kMITFolder:
google-maps-scraperweb search scraper api skill
This skill helps users automatically extract complete Markdown content from any website via the BrowserAct Web Search Scraper API. The Agent should proactively apply this skill when users express needs like extract complete markdown from a specific…
browser-act/skills★ 6.3kMITFolder:
web-search-scraper-api-skillwebcrawler deep crawl
Deep-crawl any website from start URLs, return per-page LLM-ready text/markdown/HTML plus metadata (title, description, author, language, canonical URL, OG) and in-scope outbound links. Use when user mentions deep crawl website, recursive crawl, crawl a…
browser-act/skills★ 6.3kMITFolder:
webcrawler-deep-crawlfacebook groups scrape posts
Scrapes posts from a Facebook group given a group URL, sort order, and desired count — returns structured post metadata including post_id, permalink, author, timestamp, body text, images/videos, reaction counts, reaction type breakdown, comment count, and…
browser-act/skills★ 6.3kMITFolder:
facebook-groups-scrape-postsscrapecreators api
Scrape and extract public data from 27+ social media platforms using the ScrapeCreators REST API. Covers TikTok, Instagram, YouTube, LinkedIn, Facebook, Twitter/X, Reddit, Threads, Bluesky, Pinterest, Snapchat, Twitch, Kick, Truth Social, TikTok Shop,…
ScrapeCreators/social-media-research-skills★ 5.1kMITFolder:
scrapecreators-apifirecrawl scrape
Scrape web pages and extract content via Firecrawl MCP
parcadei/Continuous-Claude-v3★ 3.9kMITFolder:
firecrawl-scrapellm crawler access check
Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot. Use when someone asks whether AI bots are blocked, whether to allow or block GPTBot, why a site never…
davepoon/buildwithclaude★ 3.7kMITFolder:
llm-crawler-access-checkapify ultimate scraper
Universal AI-powered web scraper for any platform. Scrape data from Instagram, Facebook, TikTok, YouTube, LinkedIn, X/Twitter, Google Maps, Google Search, Google Trends, Reddit, Airbnb, Yelp, and 15+ more platforms. Use for lead generation, brand…
apify/agent-skills★ 2.5kNo licence statedFolder:
apify-ultimate-scraperboss zhipin scraper
Scrape BOSS直聘 (job listing site) via Chrome CDP. Searches jobs by keyword/city/filters, fetches JD details, outputs structured JSON/CSV with plaintext salary, and can summarize scraped results into a job-market prompt. Optional --company locally filters…
eatmoreduck/boss-zhipin-scraper★ 1.7kMITFolder:
boss-zhipin-scraperfirecrawl
Scrape and crawl websites with Firecrawl: scrape a page, start a crawl, check crawl status, map a site, search the web. Trigger phrases: firecrawl, scrape website.
Anil-matcha/awesome-muse-connectors★ 1.4kMITFolder:
firecrawlhacker news scraper
Search Hacker News stories and comments using the free Algolia API. No Apify token needed. Use when you need to find HN discussions, track mentions, discover Show HN launches, or monitor tech community sentiment.
gooseworks-ai/goose-skills★ 1.3kMITFolder:
hacker-news-scraperlinkedin profile post scraper
Scrape recent posts from LinkedIn profiles using Apify. Use when you need to monitor what specific people are posting on LinkedIn, track founder/exec activity, or gather LinkedIn content for competitive intelligence.
gooseworks-ai/goose-skills★ 1.3kMITFolder:
linkedin-profile-post-scrapersite crawlability
When the user wants to improve crawlability, fix orphan pages, or optimize site structure for search engines. Also use when the user mentions "crawlability," "crawl budget," "orphan pages," "internal links," "site structure," "site crawlability," "infinite…
kostja94/marketing-skills★ 1.1kMITFolder:
crawlabilityweb scraper api
Production-grade web scraping with automatic anti-bot bypass, structured JSON parsing for 40+ targets, and geo-targeting. Use when the user needs to scrape web pages, extract product data, get search results, or collect structured data from supported…
oxylabs/agent-skills★ 877MITFolder:
web-scraper-apixquik x tweet scraper
Run Xquik's Apify Actor for X searches, posts, timelines, conversations, lists, articles, and engagement research.
Varnan-Tech/opendirectory★ 686MITFolder:
xquik-x-tweet-scraperxquik x follower scraper
Run Xquik's Apify Actor for X followers, following, verified audiences, lists, communities, and overlap research.
Varnan-Tech/opendirectory★ 686MITFolder:
xquik-x-follower-scraperyc jobs scraper
Scrape daily job listings from YCombinator's Workatastartup platform without duplicates. Use this skill when asked to scrape YC jobs, update the YC companies list, or retrieve the latest startup jobs. It handles authentication, extracts company slugs via…
Varnan-Tech/opendirectory★ 686MITFolder:
yc-jobs-scraperblock ai scrapers
Block AI coding agents (Claude Code, Cursor, OpenCode, Grok, Codex, ChatGPT, Perplexity), WebFetch tools, CLI scrapers, and headless browsers from scraping your website or API while keeping verified Googlebot SEO indexing intact and redirecting every…
Illyism/seo-skills★ 679No licence statedFolder:
block-ai-scrapersfirecrawl map
Discover and list a site's URLs, with search filtering. Use for "map the site" or "find the URL for" requests — when the user knows the site but not the exact page, or wants site structure.
firecrawl/cli★ 674No licence statedFolder:
firecrawl-mapfirecrawl developer index
Searches current documentation, release notes, API references and public repository issues for libraries, frameworks, SDKs, APIs and CLI tools with `firecrawl developer`. Use before answering when the user names a version, is upgrading, pastes an error…
firecrawl/cli★ 674No licence statedFolder:
firecrawl-developer-indexfirecrawl agent
Autonomously navigate websites and extract structured data across pages. Use when the task requires navigation or no suitable ready-made workflow or data provider covers it.
firecrawl/cli★ 674No licence statedFolder:
firecrawl-agentharvest deep crawl
Multi-page deep crawling - documentation sites, wikis, knowledge bases
vibeeval/vibecosystem★ 541MITFolder:
harvest-deep-crawldouyin works crawler
抖音作品爬取工具，输入抖音名称或抖音ID，通过红狐API广域库获取账号基础信息和近期作品内容列表（支持分页，最多50条/页）。当用户提到"爬取抖音作品"、"抖音作品列表"、"查看抖音视频"、"抖音内容采集"、"抓取抖音作品"时使用。
redfox-data/redfox-community★ 450No licence statedFolder:
douyin-works-crawlergzh search crawler
公众号搜索爬虫 — 通过关键词搜索微信公众号文章，终端表格展示（标题/作者/阅读/点赞/分享/收藏/发布时间/文章链接），自动导出 CSV 并生成交互式 HTML 报告。当用户需要搜索、爬取或浏览公众号文章时使用；也适用于行业热点追踪、竞品内容分析、素材灵感搜集等场景。
redfox-data/redfox-community★ 450No licence statedFolder:
gzh-searchwechat channels crawler
视频号作品查询工具。根据用户输入的关键词搜索视频号热门作品，支持按最新、最多点赞、最多收藏和综合排序，结果以结构化表格展示。当用户想要查找视频号热门内容、搜索视频号爆款视频、查询视频号作品数据、了解某类内容在视频号的表现时使用。触发词包括：视频号搜索、视频号作品查询、视频号爆款、视频号热门、视频号内容搜索、搜视频号、微信视频号搜索。当接口返回积分不足（code=3210）时，自动唤起支付宝完成充值后继续查询。
redfox-data/redfox-community★ 450No licence statedFolder:
wechat-channels-crawlertwscrape
Guide for using twscrape to collect public Twitter/X data through its GraphQL interface, asynchronous queries, and account session management.
aiskillstore/marketplace★ 446No licence statedFolder:
twscrapeweb scrape
Intelligent web scraper with content extraction, multiple output formats, and error handling
aiskillstore/marketplace★ 446No licence statedFolder:
web-scrapeweb scraping
Authorized web scraping with fallback cascades and access-failure handling. Use for social media, yt-dlp, CAPTCHA or 403 blocks.
jamditis/claude-skills-journalism★ 439MITFolder:
web-scrapingfirecrawl api
This skill should be used for scripted Firecrawl API access from a dependency-light Python script (search, scrape, extract, map) and for Firecrawl Alexandria structured-data tools (discover, inspect contract, run with a spend ceiling), including a local…
glebis/claude-skills★ 402MITFolder:
firecrawl-apifirecrawl research
This skill should be used when the user requests to research topics using FireCrawl, enrich notes with web sources, search and scrape information, or write scientific/academic papers. It extracts research topics from markdown files, creates research…
glebis/claude-skills★ 402MITFolder:
firecrawl-researchdata scraper
Use when data lives on a website with no usable API — listings, prices, public records — and the scrape must stay legal and not get blocked: legal gate, extraction path, durable selectors, pacing, resilience. NOT parsing bytes you already hold into fields…
ericrisco/rsc-harness★ 388MITFolder:
data-scraperdata scraper intent
A toolkit for web data extraction and search intent analysis. Includes patterns for handling lazy-loaded content and LLM prompts for classifying search keywords.
kennyzir/7deer_skills★ 331MITFolder:
data-scraper-intentroblox game data scraper
从 Roblox 游戏的公开 Trello 看板采集卡片，并提取物品与兑换码结构化数据。适用于无需登录的公开看板资料整理和 JSON 数据准备。
kennyzir/7deer_skills★ 331MITFolder:
roblox-game-data-scraperapify ashby jobs scraper
Scrape Ashby jobs or discover companies using Ashby with the Apify Ashby Job Board API Actor (johnvc/ashby-job-board-scraper). Job mode returns live job rows with title, company, location, remote and employment fields, publish date, apply URL,…
apify/awesome-skills★ 285Apache-2.0Folder:
apify-ashby-jobs-scraperapify blocked scrape triage
Diagnose and recover a scrape that is blocked, throttled, or served empty or partial content, in cost order, before paying for a heavier Actor. Use when a run returns 0 items, when a site answers 403, 429 or a challenge page, when the HTML comes back empty…
apify/awesome-skills★ 285Apache-2.0Folder:
apify-blocked-scrape-triageapify reddit scraper
Scrape Reddit with the harshmaur/reddit-scraper Apify Actor: keyword search across all of Reddit or inside one subreddit, full subreddit listings, post permalinks with complete comment threads, user profiles and community info, with date-range filters and…
apify/awesome-skills★ 285Apache-2.0Folder:
apify-reddit-scraperfirecrawl deep research
Investigate questions needing multi-source synthesis: deep dives, comparisons, or find-all lists. Prefer focused workflows for enrichment, prospecting, or docs; not quick lookups or shopping.
firecrawl/firecrawl-claude-plugin★ 250MITFolder:
firecrawl-deep-researchgallery scraper
Bulk download images from login-protected gallery websites using an attached browser session. Use when asked to scrape, download, or save images from authenticated gallery pages, extract full-size images from thumbnails, or batch download from multi-page…
jdrhyne/agent-skills★ 240MITFolder:
gallery-scraperfirecrawl scraper
Firecrawl v2.5 API for web scraping/crawling to LLM-ready markdown. Use for site extraction, dynamic content, or encountering JavaScript rendering, bot detection, content loading errors.
secondsky/claude-skills★ 240MITFolder:
firecrawl-scraperx twitter scraper
Use Xquik to fetch X (Twitter) data or act through a connected account. It covers search, profiles, followers, replies, threads, timelines, media downloads, bulk exports, trends, monitors, signed webhooks, draws, posts, likes, follows, DMs, and AI analysis…
Xquik-dev/x-twitter-scraper★ 225MITFolder:
x-twitter-scraperweb scraper
Browse websites and extract structured information. Use when the user needs data from a web page.
skrun-dev/skrun★ 210MITFolder:
web-scraperseo firecrawl
Ad-hoc web scraping, site mapping, and full-site crawling via Firecrawl MCP. Returns raw HTML, parsed metadata (og:*, twitter:*, JSON-LD, canonical, robots), JS-rendered DOM, and screenshots that WebFetch cannot. Distinct from the SE Ranking skills (which…
seranking/seo-skills★ 189MITFolder:
seo-firecrawlfirecrawl competitive intel
Monitor competitor pricing, features, changelogs, dashboards, and product changes with Firecrawl. Use for recurring competitive intelligence, pricing tier extraction, feature change tracking, or structured competitor alerts.
firecrawl/skills★ 140ISCFolder:
firecrawl-competitive-intelfirecrawl company directories
Extract structured company lists with Firecrawl from any page or site that lists companies: accelerator batches, VC portfolios, member and sponsor lists, marketplaces and app directories, YC, Crunchbase, Product Hunt or G2. Use when the user wants those…
firecrawl/skills★ 140ISCFolder:
firecrawl-company-directoriesfirecrawl business contacts
Find the published business contacts of companies with Firecrawl, whether the user names them or describes them by type and place: press and media, sales, partnerships and support inboxes, shared business phone numbers and official contact forms, taken…
firecrawl/skills★ 140ISCFolder:
firecrawl-business-contactswebsite scraping
Generic playbook for extracting structured data from any website — hotel prices, flight fares, e-commerce listings, real estate, jobs, competitor product catalogues, anything where the goal is to turn one or more URLs into clean records on disk. Use this…
moonlight-lupin/agent-skills★ 113MITFolder:
website-scrapingseo ai crawlers
Audit AI crawler access and Google AI-feature eligibility for a page — check the snippet controls that actually gate AI Overviews and AI Mode (noindex, nosnippet, max-snippet, data-nosnippet), resolve per-bot effective access from robots.txt for the…
Hainrixz/claude-seo-ai★ 80MITFolder:
seo-ai-crawlersseo crawlability
Audit and generate robots.txt and general crawl access for a page — verify robots.txt reachability and syntax, detect Disallow rules that block CSS/JS or important content, sanity-check crawl-delay, confirm a Sitemap directive, and assert overall crawl…
Hainrixz/claude-seo-ai★ 80MITFolder:
seo-crawlabilityseo crawl render
Acquire a page or site into the shared PageSnapshot every audit module reads — real HTTP status chain, headers, robots directives, raw HTML on disk, parsed document, optional headless-Chrome render with a raw-vs-rendered delta, and once-per-run site…
Hainrixz/claude-seo-ai★ 80MITFolder:
seo-crawl-render
Claude skills for web scraping: common questions
Which Claude skill should I use for web scraping?
There are 368 of them, spread over 146 repositories on GitHub. Named for it and most starred: firecrawl search (wanshuiyin/Auto-claude-code-research-in-sleep, ★ 18k), google maps scraper (gosom/google-maps-scraper, ★ 7k) and web search scraper api skill (browser-act/skills, ★ 6.3k). Stars rate the repository a skill sits in, not the skill.
How do I install a Claude skill for web scraping?
For web scraping, take the firecrawl-search folder from wanshuiyin/Auto-claude-code-research-in-sleep (the folder that holds SKILL.md) into ~/.claude/skills/ to use it in every Claude Code project, or into .claude/skills/ for one project. In the Claude apps, zip that folder and upload it as a custom skill in settings.
What should I check before installing a web scraping skill?
A web scraping skill is instructions Claude follows, and it can bundle scripts Claude runs, so read SKILL.md and every script first: for firecrawl search, that is everything in the firecrawl-search folder. Red flags: piping a download into a shell, sending files to a URL, and instructions to hide what it does from you.
Are these web scraping skills maintained?
215 of the 368 web scraping skills sit in repositories that changed in the 30 days before 31 Oct 2026.
Are these web scraping skills official?
No. None of the 368 web scraping skills comes from Anthropic's own skills repository, so all of them are community-made across 146 repositories.
Can I use these web scraping skills at work?
Of the 50 web scraping skills listed here, 40 sit in repositories under an open licence such as MIT or Apache 2.0, and 10, such as apify ultimate scraper and block ai scrapers, state no licence at all, which legally means you may not reuse them. Check the licence file before you copy one into company work.
More topics: every “Claude skills for…” page.