My Fav Web browsers & their automation

Tl;DR

Playing again with scrappers: scrapegraph, crawl4ai, firecrawl.

Job Trends Repo

Source Code of the job trends + sqlite + plots

Scrapping Repo

Source Code of with tweaks to scrapegraph, firecrawl and crawl4ai

Intro

Browser automation is a thing.

Specially if we combine it with LLMs:

• https://console.groq.com/docs/browser-automation

• https://github.com/browser-use/browser-use?tab=readme-ov-file

MIT | 🌐 Make websites accessible for AI agents. Automate tasks online with ease.

{
  "mcpServers": {
    "browser-use": {
      "command": "uvx",
      "args": ["browser-use[cli]", "--mcp"],
      "env": {
        "OPENAI_API_KEY": "sk-..."
      }
    }
  }
}

Browsers Ive tried

Even a CLI browser called Lynx:

https://jalcocert.github.io/JAlcocerT/scrapping-with-llms/#faq

sudo apt install lynx
lynx duckduckgo.com

Lately I tried Zen browser.

Which we could say is a modern UI version of firefox.

See how some people make money around KASM:

https://www.youtube.com/watch?v=799uhYUxtvA

• https://browser.networkchuck.com/

Automating your Browser

Playwright is an open-source browser automation library developed by Microsoft. It is primarily used for fast, reliable, and cross-browser end-to-end testing of web applications.

Playwright allows developers to programmatically control browsers such as Chromium, Firefox, and WebKit through a unified API, supporting automation of user interactions, UI testing, and even web scraping.

Yes, Playwright is open source, available under the MIT license, and supports multiple programming languages including JavaScript, Python, C#, and Java. It relates to web scraping because it can automate browser actions including navigation, interaction with page elements, and data extraction, thus enabling scraping of dynamically rendered web pages that simpler HTTP request libraries cannot handle effectively.

In summary:

• Playwright is a web automation tool focused on end-to-end testing.
• It is open source.
• It can be used for web scraping as well as testing, especially for sites with complex, dynamic, or JavaScript-rendered content.[2][3][4][7][10]

1 8 10

Playwright vs Pupeteer

Puppeteer and Playwright are both web automation tools that allow programmatic control of web browsers to automate tasks like testing, scraping, and interaction.

Puppeteer

• Puppeteer is a Node.js library developed by Google to control Chrome and Firefox via DevTools Protocol.
• It enables high-level browser automation: navigation, clicking, typing, taking screenshots, generating PDFs.
• Mainly focused on Chromium-based browsers with growing Firefox support.
• Pros: Simple API, great for Chrome automation, headless or headed modes, widely used for testing and scraping.
• Limitations: Primarily Chromium-focused (though Firefox is supported), limited cross-browser support, JavaScript/Node.js only.

Long back I saw https://github.com/unconv/gpt4v-browsing

Web Scraping with GPT-4 Vision API and Puppeteer

Playwright

• Playwright is an open-source automation framework by Microsoft supporting Chromium, Firefox, and WebKit (Safari) with one API.
• Supports multiple languages: JavaScript, Python, C#, and Java.
• Advanced features include multi-browser, multiple contexts, network interception, automatic waiting, mobile/device emulation, and visual testing.
• Pros: True cross-browser support, multi-language, built-in test runner (@playwright/test), robust automation for modern web apps.
• Limitations: Slightly more complex API, heavier dependency compared to Puppeteer but offers much broader capabilities.

Other notable web automation tools

Summary

• Puppeteer: Great for Chrome-focused automation and quick JavaScript-based tasks.
• Playwright: More comprehensive, cross-browser, multi-language automation framework suitable for complex, cross-platform testing.
• Selenium: Most established with broad language and browser support but heavier setup.
• Others like Cypress, TestCafe offer specialized approaches with their own trade-offs.

These tools differ in browser support, language support, ease of setup, and test stability, so the choice depends on project needs and tech stack.

1 6 7 8

Scrapping Recap

Not long ago I got to know about ApiFy

• Apify uses a subscription plus consumption (pay-as-you-go) model, with plans ranging from free to $999/month. It charges for compute units, proxy rental, storage, and support. The free plan includes $5 in monthly credits. For large-scale tasks, compute units cost between $0.20 and $0.30 each. Apify is ideal for scalable projects with varied needs and enterprise support.

• Firecrawl has a simple and predictable credit-based model: 1 page scraped equals 1 credit. The free plan allows for up to 500 pages, and paid plans range from $16 for 3,000 pages to $333 for 500,000 pages. This includes infrastructure, proxies, and bandwidth. It is a more economical choice for simple scraping tasks and moderate volumes.

• ScrapeGraphAI offers annual plans with credits for requests and speed limits, ranging from a free plan with 50 credits to customized enterprise plans. It focuses on AI-powered scraping, with costs based on API calls or pages scraped. It provides support and scalable features, but exact prices vary depending on usage.

• Crawl4AI is free software since it’s open source, but the real cost comes from the infrastructure and external LLM APIs used for extraction and processing. This means the costs are variable and depend on the scale and external services you connect.

In short, Apify is flexible and powerful but more expensive for simple tasks. Firecrawl has a very clear and affordable cost for low to medium volumes. ScrapeGraphAI and Crawl4AI are more specialized and depend on additional infrastructure usage.

If you are new to coding and scrapping, just ask while vibe coding about: python uv environment setup + use any of these frameworks below for scrapping + create a container around your final setup for easy deployment.

ScrapeGrapAI

#lynx https://it.pracuj.pl/praca
curl -L https://it.pracuj.pl > page.html

Thanks to this, I discovered that it ships a CSR.

Scrapecraph allows to ask in NL, but it WONT take some hidden goodies that you might be interested.

Crawl4AI was behaving somehow better than ScrapeGraph…

But with FireCrawlAI, I got even the most complex data.

Crawl4AI

With C4AI, you can also forget about HTML and bs4.

Just provide the link, the openAI key and what exactly you want to extract from a page.

You are done after that.

FireCrawlAI

Just in case that you dont remember, we played with this here:

• https://github.com/firecrawl/firecrawl
• https://www.firecrawl.dev/signin

FireCrawl MCP

And now, even Firecrawl has its MCP server:

• https://docs.firecrawl.dev/use-cases/developers-mcp
• https://github.com/firecrawl/firecrawl-mcp-server

MIT | 🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.

See: https://docs.firecrawl.dev/mcp-server and https://apify.com/mcp-servers/firecrawl-mcp-server#running-on-windsurf

{
  "mcpServers": {
    "mcp-server-firecrawl": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": {
        "FIRECRAWL_API_KEY": "YOUR_API_KEY"
      }
    }
  }
}

Conclusions

Last time I was scrapping with the job-trends on this post I got limitation on how to capture the amount of offers automatically based on bs4.

Puppeteer and Playwright can help extract data from websites where BeautifulSoup (bs4) is not sufficient.

So my ranking goes like this:

1. HTML with bs4 + openAI to make easy to find the place of the data you want - for simple non changing sites, like this as per script

2. ScrapeCrapgh AI + OpenAI/Ollama -> To forget about html and ask in natural language what you want to pull and how. See how with this script.

I place on this second tier to Crawl4AI + OpenAI as well

It got me partial information of that very hard to extract info

3. FireCrawlAI with their API -> The one that has worked best for me, also with NL. You have a free tier and a playground: https://www.firecrawl.dev/app/playground

Firecrawl offers a free plan that allows you to scrape up to 500 pages per month without requiring a credit card.

This free tier provides:

• 500 credits (each credit generally equals one page scrape)
• Rate limits of 10 scrape requests per minute
• Up to 2 concurrent browsers (parallel scraping sessions)
• 1 crawl request per minute

This free plan is designed for lightweight testing and small-scale scraping projects. For higher volumes, paid plans start at $16/month for 3,000 credits and go up to enterprise plans offering unlimited credits and concurrency.

More FireCrawl: They have been adding features.

And getting funded

Now its not just scrape, crawl, map.

But search, lead, seo and….MCP

MCP x PlayWright

https://jalcocert.github.io/JAlcocerT/playwright-101/

• https://github.com/microsoft/playwright-mcp

{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": [
        "@playwright/mcp@latest"
      ]
    }
  }
}

Self-hostable browsers

Several self-hostable browsers running via containers are available, including:

• Neko: A self-hosted virtual browser running in Docker containers using WebRTC technology for streaming. It supports multiple browsers like Firefox, Chromium, and Brave and focuses on privacy, multi-user collaboration, and lightweight browser isolation.

sudo chown -R 1000:1000 /home/docker/neko
#docker compose up -d

Remember to do ctrl+c to copy from neko to your main desktop

BTW, How cool is to use perplexity and have a reference to your own website:

• Kasm Workspaces: A container streaming platform that can spin up various browsers or full desktops in ephemeral Docker containers accessible via the web browser. It’s designed for enterprise use, secure browser isolation, and remote workspace management.[5][6][7]

• Browserless: An open-source Docker container allowing deployment of headless Chrome browsers for remote web automation, scraping, and testing. It supports Puppeteer and Playwright libraries and offers robust API access for automation tasks.[8][9][10]

• linuxserver/firefox: A Firefox browser Docker image maintained by LinuxServer which can be run as a container, offering a self-hosted Firefox instance though without the streaming or remote interactive features of Kasm or Neko.[11]

These options differ in focus from lightweight isolated self-hosted browsers (Neko), full containerized workspaces/desktops with browser support (Kasm), to headless browser automation and scraping (Browserless), and a simple containerized Firefox for local or network access.[2][1][8][11]

2 7 8

Conclusions

This puts the typical desktop Firefox browser in context with containerized and virtualized environments, showing it lacks isolation compared to container or VM solutions but provides instant local access and persistent user state on the host OS.[1][2][3][4][5][6][7][8][9]

1 2 3 4 5 6 7 8 9

Scrappers life is hard

The biggest challenges that make life hard for web scrapers include CAPTCHA systems and content rendering methods such as Server-Side Rendering (SSR) and Client-Side Rendering (CSR).

CAPTCHAs

• CAPTCHAs are designed specifically to block automated bots, forcing scrapers to either halt or solve CAPTCHA challenges to continue scraping.
• They increase operational costs and reduce scraping efficiency because solving CAPTCHAs often requires human-like interaction or dedicated CAPTCHA-solving services.
• Frequent CAPTCHAs disrupt continuous data extraction, especially for large-scale projects, and can affect data completeness and quality.
• Bypassing CAPTCHAs ethically needs careful balance to respect terms of service and data privacy.[1][2][3][4]

Server-Side Rendering (SSR) vs Client-Side Rendering (CSR)

• SSR sends fully rendered HTML pages from the server, which are easier and faster for scrapers and crawlers to extract data from since the content is immediately available.
• CSR pages load only a minimal HTML shell, then dynamically render content using JavaScript in the browser, making scraping more complex as scrapers must execute JavaScript typically via headless browsers (e.g., Playwright, Puppeteer).
• CSR delays data availability and adds significant overhead during scraping, requiring scrapers to wait for content to load after JavaScript execution.
• SSR is generally friendlier to scrapers and search engine crawlers, while CSR adds complexity and scraping time.[5][6][7]

Other Challenges

• Dynamic content that changes with interactions or time complicates scraping.
• Frequent website structure changes break scrapers and require constant maintenance.
• IP blocking and rate limiting from websites also restrict scraper access, necessitating IP rotation and proxy use.[8][9][5]

Summary

CAPTCHAs present a direct technical and ethical hurdle to automation, while SSR vs CSR impacts the complexity and resource requirements for scraping. Both significantly affect scraper stability and efficiency, with CAPTCHAs acting as active gatekeepers and CSR increasing technical overhead. SSR content is easier to scrape compared to CSR, which demands JavaScript execution and increases scraper complexity.[3][6][1]

FAQ

What it is Puppeteer?

• Puppeteer is a Node.js library developed by Google, used for automating and interacting with web pages using a headless Chrome or Chromium browser.
• Key functionalities include:
  • Headless Browsing: Operates Chrome or Chromium without a graphical user interface, ideal for background automation tasks.
  • Web Scraping: Enables data extraction from websites through navigation and interaction with page elements.
  • Automated Testing: Useful for testing web applications by simulating user actions like clicks and form submissions.
  • Screenshots and PDF Generation: Captures screenshots and generates PDFs from web content, beneficial for reports or documentation.
  • Performance Monitoring: Measures web page performance metrics such as load times and resource usage.
  • Accessibility Testing: Incorporates tools to ensure web pages are accessible to users with disabilities.
• Popularity: Favored for web scraping and automated testing due to its ease of use and Chrome integration.
• Getting Started: Install via npm or yarn, then write scripts to automate browser interactions and perform tasks on web pages.

What is KASM?

Kasm is a container streaming platform that delivers browser-based access to secure, isolated workspaces such as desktops, applications, and web browsers via Docker containers. It essentially provides containerized virtual desktops or browser sessions accessible through a web browser, designed for security, scalability, and ease of access without heavy traditional VM overhead. Kasm workspaces are ephemeral, isolated, and can run full desktops or just apps within containers, allowing quick spin-up and destruction with robust access control and security features.

How Kasm compares with a Virtual Machine (VM)

• Kasm uses lightweight Docker containers instead of full virtual machines, resulting in much lower resource overhead, faster startup times, and easier scalability.
• VMs emulate complete hardware environments running full OS installations, which provide thorough isolation but require more CPU, memory, and storage.
• Kasm workspaces are immutable and disposable like lightweight virtual machines but lack some deep system-level isolation VMs provide.
• Kasm is designed for web-browser access and rapid ephemeral usage rather than persistent state typical for VMs.

How Kasm compares with a Webtop via Docker containers

• Webtop is also a Docker-based Linux desktop inside a container accessible via a browser or web interface.
• Kasm offers a full platform around container orchestration, access control, workspace management, session isolation, and security with features like chat, collaboration, and automated container lifecycle.
• Webtop focuses on providing a Linux desktop container image and environment, which can integrate with Kasm, but lacks Kasm’s broader workspace management and multi-user features.
• Kasm is more of an enterprise-grade container desktop delivery platform, whereas Webtop is more minimalist for running individual GUI desktops inside containers.

How Kasm compares with Neko (a browser running in a container)

• Neko is a self-hosted virtual browser in a container, primarily focused on secure, containerized browser sessions streamed via WebRTC.
• Kasm can run full desktops, multiple applications, or browsers inside containers, providing more versatility beyond just browsers.
• Both aim to isolate browsing or apps in containers for security, but Neko focuses on browser isolation and features like session broadcasting, multi-user control, and automation.
• Kasm offers a broader platform for workspace provisioning, workspace types, user management, and collaboration, whereas Neko targets secure browser use cases specifically.

In summary, Kasm can be seen as a comprehensive container desktop/workspace platform optimized for secure, scalable browser access to virtualized desktops and apps. It is more lightweight and scalable than traditional VMs, more fully featured than minimal container desktop solutions like Webtop, and more versatile than single-browser container solutions like Neko.[1][2][3][4][5][6][7][8][9]

1

12

15