https://youtu.be/5OJ5t5Nk0wQ Agency clients often ask a question legacy crawlers cannot answer: "We rank on Google, so why does ChatGPT never mention us?" Often, the blocker is invisible: robots.txt allows AI crawlers, but a CDN or firewall silently blocks them. Traditional crawlers only inspect the file directives; they do not test real-time AI user-agent responses. I built Scrawly to bridge that gap: a free, open-source (MIT License, v1.0.0) desktop crawler for macOS, Windows, and Linux. It audits the technical signals governing Google rankings alongside visibility in AI engines like ChatGPT, Claude, Perplexity, and Google AI Overviews. There are no subscriptions, account walls, or URL caps—all audit data remains stored in a local SQLite database on your machine. Scrawly is a local-first, open-source technical SEO crawler and audit platform engineered without artificial URL limits or licensing paywalls. Built from the ground up as a robust free screaming frog alternative software, Scrawly provides search professionals and technical architects with enterprise-grade crawl diagnostics, JavaScript rendering evaluation, and site architecture mapping right on their local machine. Beyond replacing traditional desktop crawlers, it functions as a modern, free Sitebulb alternative, pairing comprehensive technical issue categorization with AI-readiness auditing and native Model Context Protocol (MCP) integration. For teams seeking a transparent, community-driven screaming frog alternative, Scrawly delivers full crawl data sovereignty without the enterprise subscription tag. 250 Technical Checks Across 23 Categories Scrawly evaluates 250 distinct parameters using Playwright Chromium rendering: - Crawlability & Indexability (15): Robots blocks, noindex tags/headers, orphan pages, click depth > 4. - Status Codes & Redirects (14): 4xx/5xx responses, soft 404s, broken links, redirect loops/chains, 302 vs 301. - Canonicals (11): Missing, relative, conflicting, or dynamic post-render canonical shifts. - Titles, Meta & Headings (21): Pixel-width truncation, thin descriptions, duplicate H1s, Open Graph. - Content Quality (12): Exact/near duplicates, keyword cannibalization, placeholder copy, missing dates. - Internal Architecture (12): Link equity distribution, single-inlink pages, generic anchor text. - External Links (6): Broken outbound links, insecure HTTP, missing rel="sponsored" or ugc. - Images & Media (11): Missing alt text, files > 100 KB, missing dimensions, WebP/AVIF adoption. - Hreflang (8): Missing return tags, invalid language codes, x-default, header/sitemap conflicts. - Structured Data (15): JSON-LD validation, @id conflicts, Organization, Article, Product, and FAQ schema. - XML Sitemaps (10): Format errors, non-200 URLs listed, file size limits, stale lastmod. - Robots Directives (7): Site-wide disallows, blocked render assets, wildcard errors, high crawl delays. - Security (10): Mixed content, HSTS configuration, expiring SSL certs, server version leaks. - Core Web Vitals & Speed (14): LCP (>2.5s), INP (>200ms), CLS (>0.1), TTFB, DOM node size. - Mobile (7): Viewport setup, tap targets, horizontal scrolling, desktop/mobile parity. - JavaScript Rendering (6): DOM drift, client-only links, post-render metadata alterations. - Accessibility (9): WCAG elements impacting SEO and AI agents (ARIA, landmarks, color contrast). - AI Search & GEO (13): Silent bot blocking, client-side content barriers, entity markup, answer structure. - Agentic Web & llms.txt (13): /llms.txt quality (<800 links; 20–50 ideal), agent navigation discovery. - URL Hygiene (9): Uppercase characters, unsafe strings, tracking parameters, URLs > 115 chars. - Pagination (5): Page 2+ noindex/canonical traps, infinite scroll URL accessibility. - WordPress & HTML (17): Thin tag archives, staging leaks, dual Yoast/Rank Math conflicts, DOCTYPE. - Analytics & Config (5): GA4/GTM tagging, Search Console verification, favicon, 404 templates.

Hello. My name is Morimura you can simply call me Mori and I have 10 years of experience in full-stack and AI project development. Technically, I am proficient in full-stack development (including the MERN stack) as well as mobile app development using frameworks like React Native. Mainly use Python/Django, React/Nextjs, C#/.NET, Gofast and serveral Typescript and Python libs for developing cutting-edge AI production such as Voice AI, AI assistants, Automation and LLM projects. Let me briefly describe some of the latest projects I have built. I have handled numerous complex projects, such as developing an SEO content generation platform by building a language model backend (using Python/Django) integrated with an AI model and a frontend interface. I am currently developing a multi-device integration system for customer management in collaboration with a U.S. client, and we have already filed for and received a patent. As a senior-level full-stack and AI engineer, I look forward to the possibility of a long-term work with you.

Full recording of this week's State Change Mastermind livestream with Ray Deck and Dimitris Goudis. We talked about MCP connectors as the new interface for applications, turning experts' videos into knowledge-graph MCPs, and controlling a Mac from the Claude app. 00:00 Intro 01:11 Dimitris: MCP connectors are the new user interface (ClickUp to Airtable in one prompt) 02:52 Expert knowledge bases as public MCP connectors (Caleb Ralston, Rory Sutherland) 05:23 Live demo: adding the Rory Sutherland connector in Claude and asking it how to promote the livestream 16:33 How it was built: 3-4 hours, the MCP shaped around the database 19:34 Robert: notes from the MCP Dev Summit in Toronto (auth, governance, stateless and async) 22:35 Live demo: controlling my Mac from the Claude app by voice through the Snappy OS connector 28:33 How notifications get back to you (MCP tasks, the Claude app) 32:27 Does it only work on a Mac? 34:55 Experts and machines as connectors, and bespoke vs off-the-shelf MCPs 38:11 Getting things done vs gold-plating auth 41:00 Dimitris: lessons from designing the ontology for a custom knowledge graph 46:46 Wrap-up