Web Content Extraction Service (Single-Machine Docker Edition)

Engineering & Development

A web crawling and content extraction service you run on your own machine with Docker Compose: open the playground in a browser and turn pages into text for language models over an API, bound to 127.0.0.1. Basic crawling needs no key; extraction with a language model needs your own model-provider key. Derived from a frozen snapshot of an Apache-2.0-licensed open-source project.

Subscribe & download See how to start

SubscriptionUS$49 per month

Last run2026-11-14

In practice

Readiness3 / 5

What you need

Few changes needed3 / 5

Who it is for

Developers and small teams who need web pages turned into clean text or structured data for their own language models or pipelines, and want the crawler on their own machine.

How to start

$ sh scripts/quickstart/init-env.sh
$ docker compose up -d

Browse the files

Files in the product repository at the pinned commit. A few house files are left out of the zip.

File overview966 paths

▸crawl4ai86

__init__.py
__version__.py
adaptive_crawler.py
antibot_detector.py
async_configs.py
async_crawler_strategy.py
async_database.py
async_dispatcher.py
async_logger.py
async_url_seeder.py
…

▸deploy78

docker/

▸docs509

RELEASE_NOTES_v0.8.0.md
apps/
assets/
blog/
codebase/
deprecated/
examples/
md_v2/
migration/
releases_review/
…

▸prompts1

prompt_net_requests.md

▸sbom2

README.md
sbom.cdx.json

▸scripts3

gen-sbom.sh
quickstart/
update_stats.py

▸tests248

WEBHOOK_TEST_README.md
__init__.py
adaptive/
adversarial/
async/
async_assistant/
browser/
cache_validation/
check_dependencies.py
cli/
…

▫ATTRIBUTION.md

▫CHANGELOG.md

▫CODE_OF_CONDUCT.md

▫CONTRIBUTING.md

▫CONTRIBUTORS.md

▫Dockerfile

▫LICENSE

▫MANIFEST.in

▫MISSION.md

▫PRODUCT.md

▫PROGRESSIVE_CRAWLING.md

▫QUICKSTART.md

▫README.md

▫ROADMAP.md

▫SECURITY-CREDITS.md

▫SECURITY.md

▫SECURITY.upstream.md

▫SPONSORS.md

▫cliff.toml

▫delivery-manifest.txt

▫docker-compose.yml

▫mkdocs.yml

▫pyproject.toml

▫requirements.txt

▫setup.cfg

▫setup.py

▫uv.lock

▫Config and hidden files 12

Latest passing run

Clone → Checkout → Toolchain → Run server

Pre-purchase Disclosure

What you are buying

A web crawling and content extraction service that runs on your own computer: brought up with Docker Compose, opened in a browser playground, and used over an API to turn web pages into text suited to language models.

Derived from an open-source project under the Apache-2.0 license, and not the original project itself.

Delivery is a frozen snapshot: one zip, taken from the product repository at revision 8f08544.

Known limitations

The service is bound to port 11235 on the local machine (127.0.0.1) only and is not open to other machines
Language-model extraction needs your own model-provider key; without a key, extraction requests for that provider’s models do not work, although basic crawling still does
The first start downloads the container image from an image registry, and the compose file uses the latest image tag
The dependencies and the browser inside the image were not scanned for licenses one by one
Pinned to the delivered snapshot; it does not follow upstream.

The liability cap, the scope of what is promised and the statutory floor are governed by the Terms of Service.

Vetting record

Everything below was checked against the pinned commit cae74b2. Every verdict carries its full reasoning, an evidence link and the checker; failures are published, not hidden — do not take our word for it, verify it yourself.

1 PASS4 WARN0 FAIL

Source ReviewWARN

Read: docker-compose.yml, .env.example, QUICKSTART.md, scripts/quickstart/init-env.sh, LICENSE, PRODUCT.md and the first lines of the Dockerfile. The Python application source was not read file by file.

Script SafetyWARN

The compose file and the env-generating script were read: loopback-only port, all capabilities dropped, no-new-privileges, read-only root filesystem, memory and process limits; the script only writes .env from .env.example with a random token. The rest of the Dockerfile and the image contents were not read.

RecencyWARN

This is a frozen snapshot that does not follow upstream. The date of the pinned upstream commit and recent upstream activity were not checked.

LicensePASS

SPDX:Apache-2.0

The LICENSE file in the product repository starts with the Apache License text, and PRODUCT.md records the upstream license as Apache-2.0. The upstream copy was not compared against it.

Dependency LicensesWARN

Dependencies and the browser are installed inside the image or fetched at build or pull time. They were not scanned for licenses.