# The Ultimate Developer Guide to Scrapall: Unified Social Intelligence API & MCP Server for 36+ Platforms > **Target Audience:** AI Engineers, Growth Hackers, Data Scientists, and Autonomous Agent Developers. > **Keywords:** Unified Social Media API, Model Context Protocol (MCP), Social Data Scraping, TikTok Scraper API, YouTube InnerTube API, Twitter Syndication API, Reddit Shreddit API, Zero-Credit Caching, AI Agent Tooling. --- ## Executive Summary Extracting public social data at scale has historically forced developers into an unsustainable dilemma: either pay exorbitant enterprise fees for official developer tiers (e.g., $5,000/month for Twitter/X Pro API) or endure the crippling latency, proxy expenses, and memory overhead of headless browser automation (Puppeteer, Playwright, Selenium). **Scrapall** fundamentally solves this problem through **Protocol-Level Reverse Engineering**. By interfacing directly with upstream mobile endpoints, syndication feeds, and CDN microservices, Scrapall delivers structured JSON payloads in **sub-second latencies (300ms – 1,200ms)** at a fraction of legacy costs. Furthermore, Scrapall integrates native **Model Context Protocol (MCP)** support, transforming these endpoints into autonomous tools that LLMs (Claude 3.5/3.7, GPT-4o, DeepSeek) can call directly to inspect the live web. Below is the definitive technical breakdown of all 10 supported platforms, their underlying reverse-engineered mechanics, and production request specifications. --- ## Architecture & Core Innovations ``` ┌────────────────────────────────────────────────────────────────────────┐ │ SCRAPALL GATEWAY ENGINE │ ├────────────────────────────────┬───────────────────────────────────────┤ │ 1. ZERO-CREDIT SMART CACHE │ SHA-256 parameter hashing. Cached │ │ │ responses refund credits (0 Cost). │ ├────────────────────────────────┼───────────────────────────────────────┤ │ 2. REVERSE-ENGINEERED PROTOCOL │ Direct HTTP/2 & TLS Chrome 120 │ │ │ fingerprinting via curl_cffi. │ ├────────────────────────────────┼───────────────────────────────────────┤ │ 3. NATIVE MCP SERVER (STDIO) │ 15 curated tools for Cursor, Claude │ │ │ Desktop, and autonomous AI agents. │ ├────────────────────────────────┼───────────────────────────────────────┤ │ 4. ON-CHAIN CRYPTO BILLING │ Non-custodial USDT (TRC20) auto- │ │ │ provisioning via TronGrid node scans. │ └────────────────────────────────┴───────────────────────────────────────┘ ``` --- ## 1. YouTube Data Engine ### Underlying Protocol Unlike the official Google Data API v3 which imposes strict quota limits (10,000 quota units/day), Scrapall leverages **Google InnerTube Client API v1** (`WEB` client identity). Requests bypass OAuth tokens and return complete video metadata and search query results directly from YouTube's internal edge servers. ### Endpoint A: Video Metadata * **Route:** `GET /v1/youtube/video` * **Query Parameters:** `url_or_id` *(String, required)* — Accepts either an 11-character video ID (`dQw4w9WgXcQ`) or full YouTube URLs (`youtu.be`, `watch?v=`). * **Credit Cost:** 1 Credit (0 Credits on Cache Hit). * **Payload Fields:** Title, channel name, view count, video length (seconds), thumbnail resolutions, description, tags, category, and publish timestamp. ```bash curl -X GET "http://localhost:8000/v1/youtube/video?url_or_id=dQw4w9WgXcQ" \ -H "X-Scapi-Key: your_api_key" ``` ### Endpoint B: Video Search * **Route:** `GET /v1/youtube/search` * **Query Parameters:** `q` *(String, required)* — Search query string. * **Payload Fields:** List of matching video objects containing title, video ID, channel author, view count, and duration. ### Endpoint C: Video Comments Extractor * **Route:** `GET /v1/youtube/comments` * **Query Parameters:** `url_or_id` *(String, required)*, `limit` *(Integer, 1 to 50, default 20)*. * **Underlying Protocol:** Interfaces directly with InnerTube `/next` continuation token pipeline and parses `frameworkUpdates.entityBatchUpdate.mutations` for real-time `commentEntityPayload` extraction. * **Payload Fields:** `comment_id`, `author`, `channel_id`, `avatar`, `published_time`, `text`, `likes`, `replies`, and `author_is_channel_owner`. ```bash curl -X GET "http://localhost:8000/v1/youtube/comments?url_or_id=Llw9Q6akRo4&limit=10" \ -H "X-Scapi-Key: your_api_key" ``` --- ## 2. TikTok Data Engine ### Underlying Protocol Scrapall queries TikTok's mobile-optimized web endpoints (`/api/user/detail/` and public catalog endpoints). It automatically parses mobile device signatures, extracting follower counts, total hearts, video uploads, and verified badge statuses without triggering sliding captcha verifications. ### Endpoint A: Creator Profile * **Route:** `GET /v1/tiktok/user` * **Query Parameters:** `username` *(String, required)* — Target creator handle (e.g., `tiktok`, `khaby.lame`). * **Payload Fields:** `unique_id`, `nickname`, `follower_count`, `heart_count`, `video_count`, `verified`, `signature` (bio), and avatar URLs. ```bash curl -X GET "http://localhost:8000/v1/tiktok/user?username=khaby.lame" \ -H "X-Scapi-Key: your_api_key" ``` ### Endpoint B: Public Video Search * **Route:** `GET /v1/tiktok/search_videos` * **Query Parameters:** `q` *(String, required)*, `count` *(Integer, default 10)*. * **Payload Fields:** Array of public video metadata including author, description, sound title, play count, share count, and direct video web links. --- ## 3. Instagram Profile Engine ### Underlying Protocol Direct web scraping of Instagram typically triggers immediate login walls. Scrapall interfaces with Instagram's web consumer API using client header `X-IG-App-ID: 936619743392459`. This enables real-time extraction of public account profiles without needing personal cookies or user credentials. ### Endpoint: User Profile * **Route:** `GET /v1/instagram/user` * **Query Parameters:** `username` *(String, required)* — Instagram account handle (e.g., `nike`). * **Payload Fields:** `id`, `username`, `full_name`, `biography`, `follower_count`, `following_count`, `media_count`, `is_verified`, and `profile_pic_url`. ```bash curl -X GET "http://localhost:8000/v1/instagram/user?username=nike" \ -H "X-Scapi-Key: your_api_key" ``` --- ## 4. Twitter / X Intelligence Engine ### Underlying Protocol Following Twitter's transition to X and the deprecation of free API access, public data scraping became notoriously difficult. Scrapall reverse-engineers the **Twitter Syndication CDN** and guest token endpoints (`cdn.syndication.twimg.com`). This provides zero-authentication access to public tweets, creator follower counts, and media attachments. ### Endpoint A: User Profile * **Route:** `GET /v1/twitter/user` * **Query Parameters:** `username` *(String, required)* — Twitter screen name (e.g., `elonmusk`). * **Payload Fields:** `id`, `name`, `screen_name`, `followers_count`, `friends_count`, `statuses_count`, `description`, `verified`, and profile banners. ### Endpoint B: Single Tweet Details * **Route:** `GET /v1/twitter/tweet` * **Query Parameters:** `id_or_url` *(String, required)* — Status ID or complete tweet URL. * **Payload Fields:** Tweet text, author info, retweet count, favorite count, reply count, creation date, and embedded media (photos/videos). ```bash curl -X GET "http://localhost:8000/v1/twitter/tweet?id_or_url=20" \ -H "X-Scapi-Key: your_api_key" ``` --- ## 5. Reddit Discussion Engine ### Underlying Protocol In late 2024–2026, Reddit deployed an advanced Proof-of-Work (PoW) JavaScript challenge (`js_challenge=1`, `jsc_token`) and Snooserv WAF to block generic web crawlers. Scrapall incorporates `curl_cffi` with **Chrome 120 TLS fingerprinting**, dynamically solves the arithmetic seed challenge, and queries Reddit's internal **Shreddit microapp API** (`/svc/shreddit/community-more-posts/`) with Atom feed fallbacks. ### Endpoint A: Subreddit Posts * **Route:** `GET /v1/reddit/subreddit` * **Query Parameters:** `r` *(Subreddit name)*, `sort` *(hot | new | top)*, `limit` *(1 to 50)*. * **Payload Fields:** Post ID, title, author, score (upvotes), comment count, permalink, and discussion thread URL. ```bash curl -X GET "http://localhost:8000/v1/reddit/subreddit?r=python&sort=hot&limit=5" \ -H "X-Scapi-Key: your_api_key" ``` ### Endpoint B: Discussion Search * **Route:** `GET /v1/reddit/search` * **Query Parameters:** `q` *(Search query)*, `limit` *(Integer)*. * **Payload Fields:** Search hits across global Reddit discussions, including topic titles, author tags, and post links. --- ## 6. Telegram Public Channel Engine ### Underlying Protocol Scrapall queries Telegram's official public web preview nodes (`t.me/s/{channel}`). This bypasses the need for MTProto bot tokens or telethon client sessions, allowing instant monitoring of public broadcast channels, crypto announcement channels, and news feeds. ### Endpoint A: Channel Overview * **Route:** `GET /v1/telegram/channel` * **Query Parameters:** `channel` *(String, required)* — Public username (e.g., `durov`). * **Payload Fields:** Channel name, title, description, subscriber count, avatar URL, and recent message snippets with message IDs. ```bash curl -X GET "http://localhost:8000/v1/telegram/channel?channel=durov" \ -H "X-Scapi-Key: your_api_key" ``` ### Endpoint B: Paginated Channel Posts * **Route:** `GET /v1/telegram/channel_posts` * **Query Parameters:** `channel` *(String, required)*, `limit` *(Integer, default 20)*, `before` *(Integer, optional)*, `after` *(Integer, optional)*. * **Payload Fields:** Array of full posts with `post_id`, `url`, `text`, `views`, `date`, `media_url`, and `reactions`. ### Endpoint C: Single Post Details * **Route:** `GET /v1/telegram/post` * **Query Parameters:** `channel` *(String, required)*, `post_id` *(Integer, required)*. * **Payload Fields:** Full post text, author, view count, published date, and media assets. ```bash curl -X GET "http://localhost:8000/v1/telegram/post?channel=durov&post_id=200" \ -H "X-Scapi-Key: your_api_key" ``` --- ## 7. Facebook Public Page & Group Engine ### Underlying Protocol Public Facebook scraping without developer accounts has been solved by Scrapall through **Meta Public Plugin Protocol** and mobile Graph endpoints. It extracts follower metrics, verified badges, cover images, and group discussions with zero cookie requirements and zero browser overhead. ### Endpoint A: Public Page Intelligence * **Route:** `GET /v1/facebook/page` * **Query Parameters:** `page` *(String, required)* — Page handle or URL (e.g., `meta`, `nike`). * **Payload Fields:** `page`, `title`, `page_id`, `followers_count`, `is_verified`, `avatar_url`, and `url`. ```bash curl -X GET "http://localhost:8000/v1/facebook/page?page=meta" \ -H "X-Scapi-Key: your_api_key" ``` ### Endpoint B: Public Group Intelligence * **Route:** `GET /v1/facebook/group` * **Query Parameters:** `group` *(String, required)* — Group vanity handle or numeric ID (e.g., `pythoncommunity`). * **Payload Fields:** `group`, `title`, `group_id`, `description`, `cover_image`, `privacy`, and `url`. ```bash curl -X GET "http://localhost:8000/v1/facebook/group?group=pythoncommunity" \ -H "X-Scapi-Key: your_api_key" ``` --- ## 7. E-Commerce Intelligence (TikTok Shop, Amazon, Shopify & Multi-Marketplaces) ### Endpoint A: TikTok Shop Search * **Route:** `GET /v1/ecommerce/tiktok_shop` * **Query Parameters:** `q` *(Search query, e.g. "wireless earbuds")*, `count` *(Integer, 1 to 30)*. * **Capabilities:** Scrapes live physical products, clean product URLs, product IDs, seller stores, review counts, and active pricing across global TikTok Shop marketplaces bypassing WAF. ### Endpoint B: Amazon Live Product Search & ASIN Extraction * **Route:** `GET /v1/ecommerce/amazon_products` * **Query Parameters:** `q` *(Search query)*, `count` *(Integer, 1 to 30)*. * **Capabilities:** High-speed Amazon product discovery extracting clean canonical ASINs, titles, ratings, estimated prices, and direct product URLs. ### Endpoint C: Amazon Consumer Search Suggestions * **Route:** `GET /v1/ecommerce/amazon_suggestions` * **Query Parameters:** `q` *(Keyword prefix)*. * **Capabilities:** Connects directly to Amazon's Completion Suggestion API (`completion.amazon.com`), delivering real-time consumer purchase intent and autocomplete queries with sub-50ms latency. ### Endpoint D: Amazon Creator Storefront & ASIN Extractor * **Route:** `GET /v1/ecommerce/amazon_storefront` * **Query Parameters:** `url_or_handle` *(String, required)* — Creator storefront vanity URL or handle (e.g., `mkbhd`). * **Capabilities:** Parses influencer storefront pages (`amazon.com/shop/{handle}`) to extract featured product titles, image assets, and standard 10-character **Amazon Standard Identification Numbers (ASINs)** for affiliate and catalog analysis. ### Endpoint E: Shopify Storefront Catalog Inspector * **Route:** `GET /v1/ecommerce/shopify` * **Query Parameters:** `domain` *(Shopify domain or URL, e.g. "kith.com", "bulletproof.com")*, `limit` *(Integer, 1 to 50)*. * **Capabilities:** Reverse-engineers standard Shopify catalog architecture via `/products.json`. Returns product titles, variants, SKUs, inventory availability, pricing signals, handles, and CDN image assets. Includes automated Cloudflare Bot-Fight detection with residential proxy failover signaling. ### Endpoint F: Multi-Marketplace Edge Search Protocol (eBay, Etsy, Shopee, FB Marketplace) * **Route:** `GET /v1/ecommerce/marketplace` * **Query Parameters:** `platform` *(ebay | etsy | shopee | facebook_marketplace)*, `q` *(Product search query)*, `count` *(Integer, 1 to 20)*. * **Capabilities:** Unifies marketplace search across global second-hand, artisan, and Southeast Asian retail giants. Employs Base64 edge URL unpacking to extract clean product listing URLs, prices, and title snippets without triggering marketplace bot blocks. ```bash # Query Shopify catalog curl -X GET "http://localhost:8000/v1/ecommerce/shopify?domain=kith.com&limit=10" \ -H "X-Scapi-Key: your_api_key" # Query eBay or Etsy listings curl -X GET "http://localhost:8000/v1/ecommerce/marketplace?platform=ebay&q=vintage+leather+jacket" \ -H "X-Scapi-Key: your_api_key" ``` --- ## 8. Creator Business Contact Engine (Email Finder) ### Functionality & Mechanics * **Route:** `GET /v1/creator/email_finder` * **Query Parameters:** `handle` *(Creator handle)*, `platform` *(instagram | tiktok | twitter)*. * **Mechanics:** Performs deep pattern scanning on creator profile bios, Linktree trees, and bio link landing pages. Automatically filters false positives (image extensions, domain masks) and returns verified business contact emails for influencer marketing campaigns. --- ## 9. Competitor Ad Intelligence Engine (Meta & Google) ### Endpoint A: Meta Ad Library Spy * **Route:** `GET /v1/ads/meta` * **Query Parameters:** `q` *(Brand name or keyword)*, `country` *(ISO Country code, default `ALL`)*. * **Capabilities:** Asynchronously queries the public **Meta Ad Library** API. Returns active commercial campaigns running across Facebook, Instagram, Messenger, and Audience Network, including ad creative copy, publisher platform flags, and display URLs. ```bash curl -X GET "http://localhost:8000/v1/ads/meta?q=shopify&country=US" \ -H "X-Scapi-Key: your_api_key" ``` ### Endpoint B: Google Ad Transparency Center * **Route:** `GET /v1/ads/google` * **Query Parameters:** `query` *(Brand name, company, or domain)*. * **Capabilities:** Reverse-engineers Google's internal RPC suggestions protocol (`adstransparency.google.com/anji/_/rpc/SearchService/SearchSuggestions`). Delivers verified advertiser IDs, active ad count estimates, registered country origin, and direct links to transparency dossiers without triggering captchas. ```bash curl -X GET "http://localhost:8000/v1/ads/google?query=nike" \ -H "X-Scapi-Key: your_api_key" ``` --- ## 10. Google Intelligence & SERP Engine ### Endpoint A: Google Autocomplete & Keyword Intent * **Route:** `GET /v1/google/keywords` * **Query Parameters:** `q` *(Seed keyword)*, `lang` *(Language code, default `en`)*. * **Capabilities:** Taps directly into Google's global suggest CDN (`suggestqueries.google.com`). Sub-50ms latency with 0 rate limit. Extracts user search intent, trending query variations, and long-tail keywords for programmatic SEO and marketing research. ```bash curl -X GET "http://localhost:8000/v1/google/keywords?q=python+ai+agent&lang=en" \ -H "X-Scapi-Key: your_api_key" ``` ### Endpoint B: Organic Web Search (SERP) * **Route:** `GET /v1/google/search` * **Query Parameters:** `q` *(Search query)*, `count` *(1 to 20, default 10)*, `lang` *(Language code)*. * **Capabilities:** High-speed organic SERP extraction delivering structured rank, clean canonical URLs, titles, and text snippets. ```bash curl -X GET "http://localhost:8000/v1/google/search?q=fastapi+production+setup&count=10" \ -H "X-Scapi-Key: your_api_key" ``` --- ## 11. Facebook Public Intelligence Engine ### Endpoint A: Public Page & Follower Metrics * **Route:** `GET /v1/facebook/page` * **Query Parameters:** `page` *(Page handle or full URL)*. * **Capabilities:** Scrapes public Meta Page Plugin protocols to extract real-time follower counts, verification status, and page ID without requiring user cookies or app review tokens. ### Endpoint B: Public Group Details * **Route:** `GET /v1/facebook/group` * **Query Parameters:** `group` *(Group name or numeric ID)*. * **Capabilities:** Extracts public group names, member count indicators, and group metadata. --- ## 12. Model Context Protocol (MCP) Server for AI Scrapall ships with a native **Model Context Protocol (MCP)** server built under the official `mcp>=1.2.0` specification. This allows developers to hook all 13 scrapers into local AI agent environments without writing API wrappers. ### Integration with Claude Desktop & Cursor Add the following snippet to your `claude_desktop_config.json`: ```json { "mcpServers": { "scrapall": { "command": "python", "args": ["-m", "app.mcp_server.server"], "env": { "BRAND_API_BASE": "http://127.0.0.1:8000", "BRAND_API_KEY": "scapi_test_key_free" } } } } ``` ### Exposed Curated Tools 1. `youtube_video_details`: Fetch video view metrics, tags, and descriptions. 2. `youtube_search`: Keyword search for public YouTube videos. 3. `youtube_comments`: Extract comments via InnerTube continuation token. 4. `tiktok_profile`: Extract follower counts and bio for TikTok creators. 5. `tiktok_search_videos`: Topic search for TikTok videos. 6. `instagram_profile`: Parse Instagram follower statistics. 7. `twitter_profile`: Extract X/Twitter follower metrics. 8. `twitter_tweet`: Fetch tweet engagement metrics. 9. `telegram_channel`: Read Telegram broadcast channel stats. 10. `telegram_channel_posts`: Paginated Telegram channel posts with views & reactions. 11. `telegram_post`: Single Telegram post metrics. 12. `facebook_page`: Real-time Facebook page follower counts & verified badge. 13. `facebook_group`: Public Facebook group metadata. 14. `google_keywords`: Google Autocomplete suggestions and search intent. 15. `google_search`: Organic SERP search results. 16. `google_ads_advertisers`: Google Ad Transparency competitor ads analysis. 17. `meta_ads_search`: Spy on active competitor Facebook/Instagram ad campaigns. 18. `reddit_subreddit` & `reddit_search`: Reddit discussions and trends. 19. `tiktok_shop_products` & `amazon_storefront`: E-commerce catalog data. 20. `creator_email_finder`: Public creator business contact emails. 21. `list_endpoints` & `call_endpoint`: Dynamic meta-tools for programmatic discovery. --- ## 13. Vertical Intelligence Tasks & Historical Snapshots (Tầng 2 & 3) Instead of stitching together multiple individual scrapers, Scrapall provides high-margin, synthesized multi-engine intelligence tasks orchestrated concurrently via `asyncio.gather`. ### A. Catalog of Vertical Synthesis Tasks * **Route:** `GET /v1/tasks/catalog` * **Access:** Free (Cost: 0 credits). * **Description:** Lists all available vertical tasks, their required credit cost, and cache TTL. ### B. Ad Spy Dossier (`ad_spy`) * **Route:** `GET /v1/tasks/ad_spy?target={brand}` * **Credit Cost:** 3 Credits (Cached: 0 Credits). * **Synthesis:** Concurrently queries Meta Ad Library, Google Ad Transparency Center, and Google Search Intent Autocomplete. Aggregates multi-network active ad counts, creative snippets, advertiser verification status, and consumer demand keywords into a unified intelligence report. ### C. Cross-Marketplace Arbitrage & Price Watch (`price_tracker`) * **Route:** `GET /v1/tasks/price_tracker?target={product_query}` * **Credit Cost:** 3 Credits (Cached: 0 Credits). * **Synthesis:** Concurrently searches Amazon Product Search, Shopify Catalogs, eBay, and TikTok Shop. Consolidates listings into a unified price table with direct item links and price comparison metrics. ### D. Influencer Due Diligence & Vetting (`influencer_vetting`) * **Route:** `GET /v1/tasks/influencer_vetting?target={creator_handle}` * **Credit Cost:** 3 Credits (Cached: 0 Credits). * **Synthesis:** Concurrently verifies Instagram profile metrics, TikTok fan counts, YouTube presence, and scans bio linktrees for verified business contact emails. ### E. Brand 360° Footprint (`brand_360`) * **Route:** `GET /v1/tasks/brand_360?target={brand}` * **Credit Cost:** 3 Credits (Cached: 0 Credits). * **Synthesis:** Synthesizes Facebook followers, Reddit community discussion volume, active Meta ad campaigns, and top organic Google SERP rankings into a complete 360-degree brand report. ### F. Append-Only Historical Snapshots (Tầng 3 Moat) * **Route:** `GET /v1/tasks/history/{task_type}?target={brand}&days=30` * **Access:** Free (Cost: 0 credits). * **Description:** Retrieves persistent time-series snapshots stored in WAL-mode SQLite (`data/scrapall.db`). Enables users to track how competitor ad volumes, product prices, and influencer followings evolve over time. ### G. Proactive Change Watcher & Webhooks * **Route:** `POST /v1/tasks/watch/create` * **Payload:** `{"task": "price_tracker", "target": "nike", "interval_sec": 86400, "webhook_url": "https://your-domain.com/hook"}` * **Description:** Schedules automated background polling. Detects key-level payload diffs against previous snapshots and fires HMAC-SHA256 signed webhook alerts upon change detection. ### H. Account Telemetry & Savings Analytics * **Route:** `GET /v1/account/analytics?days=30` * **Description:** Returns total API calls, credits spent, cache hit ratio, and estimated credits saved over 30 days. --- ## 14. Pricing, Zero-Credit Cache & Crypto Billing ### The Zero-Credit Smart Cache Every incoming request computes a SHA-256 fingerprint: $$\text{Hash} = \text{SHA256}(\text{Method} + \text{Endpoint} + \text{Sorted Query Params})$$ If a fresh record exists in cache: * Upstream latency drops to **0ms – 4ms**. * **The gateway automatically refunds the deducted credit.** The request costs exactly **0 credits**. ### Transparent Plans & USDT Billing | Plan | Price (USDT) | Credits | Effective Rate | Validity | |:---|:---:|:---:|:---:|:---:| | **Starter** | $15 | 8,000 | $1.87 / 1k | 30 Days | | **Pro Business** | $45 | 27,000 | $1.66 / 1k | 60 Days | | **Enterprise** | $150+ | 100,000+ | Custom SLA | 1 Year | Orders are settled via **Tether (USDT) on the TRON network (TRC20)**. When a transaction is submitted, Scrapall inspects the TronGrid node in real-time, verifying on-chain token balance transfers and auto-provisioning the customer's API Key in under 10 seconds. --- ## Getting Started 1. **Clone & Launch:** ```bash git clone cd API start.bat ``` 2. **Access Web Portal:** Open [http://localhost:8000/](http://localhost:8000/) for the interactive sandbox. 3. **Interactive Swagger Docs:** Explore schema definitions at [http://localhost:8000/docs](http://localhost:8000/docs). 4. **Machine Manifest:** Inspect AI crawler descriptors at [http://localhost:8000/llms.txt](http://localhost:8000/llms.txt).