man crawlipede: the crawlipede manual

NAME

crawlipede: your browser tab grows a segmented crawler that crowd-crawls the crypto web into an open dataset, and the colony crawls fresh solana launches for an AI judge. the crawler reads, the judge decides.

SYNOPSIS

crawlipede [start] → crawl → clean → hash/dedupe → spot-check → dataset → brain v0

DESCRIPTION

crawlipede turns a browser tab into a small crawler. Many tabs together read public crypto documentation into one shared, downloadable dataset. Every number on the site comes from that dataset. Nothing is simulated.

start: Press ▶ start crawling on crawlpad.garden. A web worker starts inside your tab. Nothing is installed, and closing the tab stops it. You can give it a nickname or a solana address as its name, or leave it anonymous.
crawl: The worker asks the colony for a job, then reads that page through our read-only proxy (/api/fetch), roughly one page every couple of seconds. Only public pages on a fixed allowlist of crypto sites are reachable: docs, improvement proposals, governance forums and github readmes. Private and internal addresses are blocked.
clean, verify: The tab strips menus, scripts and boilerplate down to plain text, hashes it (sha256) and sends it back. The server spot-checks random samples of that text against its own copy of the page and rejects mismatches. Each content hash is stored once, so a duplicate page adds nothing and earns nothing.
crawlipede: Each crawler is drawn as a unique crawlipede generated from its name: 10 to 16 body segments with a leg pair each, its own acid colourway, antennae and glow. The same name always gives the same crawlipede. The creatures are always on. Rarity (common 70% · rare 20% · epic 8% · legendary 2%) is cosmetic only. It never changes what you earn.
dataset: Every kept page becomes one JSON line: url, domain, title, hash, tokens, chars, crawler, verified, ts, text. Download it all or in parts of 200 from /api/dataset. Token counts are approximate (chars ÷ 4).
brain: A community AI that is meant to be trained on this dataset. It is brain v0, training pending: no training run has started, so the site shows no loss curves, parameter counts or benchmark scores.
rounds: Every 4 UTC hours (00, 04, 08, 12, 16, 20) is a battle round: crawlers (spiders) vs crawlipedes. See BATTLE below. When a round closes, it becomes a public payout ledger at /api/rounds/<id> (CSV: ?format=csv).
boost: A multiplier from your live $CrawlPad balance, read on-chain (cached 5 minutes): under 1M = 1x, 1M+ = 1.5x, 5M+ = 2x. It is snapshotted when the round closes. You never need $CrawlPad to crawl.
verify: To earn, prove you own your crawler's address by signing a short message in your wallet. It is free and it is never a transaction: the message only names your address and crawler, and it approves nothing.
vote: $CrawlPad holders vote on what the brain reads next by signing a message. A vote counts as much as the $CrawlPad that wallet holds. Anyone can suggest an allowlisted site. The leading pick jumps the queue: it gets about 60% of new crawl jobs.

BATTLE

Pick a side: crawlers (the spiders, team web) or crawlipede (team burrow). Picking is free: no deposit, no buy-in, no stake, no purchase necessary. Your pick locks for the current 4h round (on your pick or your first verified page) and you can queue a switch for the next round.

work points: Each new, verified page (server spot-checks it against the live page; duplicates and unverifiable pages score 0) earns points for your side: crawler (spider) = 1, crawlipede = 2.
pace: Crawlers (spiders) read 2 pages per 2.4s tick in parallel and keep 12 links per page (wide + shallow). Crawlipedes read 1 page per tick and keep up to 40 links (deep). The server enforces this with a per-crawler token bucket, whatever client you use.
the math: max points per hour: spider 3600/2.4 × 2 pages × 1 pt = 3000; crawlipede 3600/2.4 × 1 page × 2 pts = 3000. Same cost (free), same expected points, different style. Neither side earns more by design; which side wins a round comes down to real luck: who is online and what pages they hit.
winner: The side with more verified work points when the round closes wins. A tie splits the pot across both sides. Winners get bragging rights: a win count, a streak and a called it stamp on their crawler.
fee share: The round's pump.fun creator-fee pot goes to the winning side only. Inside it, your share = your own verified points on that side × your holder boost ÷ the same total for every qualifier on that side. To qualify: a verified wallet and at least 50 points in the round. Anti-sybil: minimum points, per-crawler rate limits, content-hash dedupe and live spot-checks; one wallet = one share across its tabs.
payouts: Manual. Nothing signs or sends SOL automatically. Each closed round produces a ledger (wallet, side, points, share %, amount) and a CSV export that is reviewed and sent by hand from Phantom, with tx links added to the round.
status: ca coming soon. The pot reads pending until $CrawlPad exists and the creator-fee vault is connected. No pot numbers are shown before that, and fees may be zero.

Not financial advice. No purchase necessary. This is a free work reward for crawling, not a wager: nothing is staked and nothing can be lost by picking a side.

SIGNALS

The colony also crawls brand-new solana launches. Every few minutes (when someone has the site open) the server reads the newest pools from GeckoTerminal and judges the most active ones it hasn't seen. Built on @savipww's idea: the crawler reads, the judge decides.

the crawl crew: Four specialist crawlipedes each read one slice, timed, and report to the brain: chart crawler (price, liquidity, volume, buys vs sells from the live pool), mint crawler (mint and freeze authority, supply, token-2022 extensions), holder crawler (walks every holder, splits out pool and curve supply) and dev crawler (finds the dev wallet and weighs its bag; past launches are not tracked yet and show pending). There is no social crawler because there is no real social data source. The live burrow panels on the home page replay exactly what each crawler fetched on the latest crawl. Every piece of evidence on a specimen card is tagged with the crawler that found it.
read: Market: price, liquidity, fdv, 1h buys/sells and unique buyers/sellers (GeckoTerminal). On-chain via Helius RPC (server-side key): mint and freeze authority, risky token-2022 extensions, every holder with pool/curve/program-owned supply split out, near-identical bags, and the dev wallet (pump.fun bonding-curve creator, or the first mint transaction's fee payer) with its bag.
decide: An AI judge (Vercel AI Gateway) returns typed verdicts: crowd real / farmed / unclear, dev clean / loaded / unknown, concentration low / medium / high / unknown, plus p(up in 1h) and the evidence it used. Unknown inputs stay null and show pending.
paper call: Every verdict is a shadow call: up if p(up) ≥ 50%, else down, at the real pool price when judged. About 60 minutes later the real price (GeckoTerminal 1-minute candle) scores it win or loss. Win rate = right calls ÷ scored calls, shown live at /api/signals.
my burrow: Pin any specimen to your burrow: a watchlist kept only in this browser's localStorage. Nothing is pinned without your click and nothing is sent to a server.
never: No trades, no wallet, no orders, no pnl. Paper calls are a public scorecard for the judge. Not financial advice. New coins are extremely risky and most go to zero.

ECONOMICS

Crawling is free and always will be. Each 4h round's pot is the pump.fun creator fees $CrawlPad earned in that round, and it goes to the round's winning side (see BATTLE). It is measured from the creator-fee vault once the creator wallet is connected. Until then, every pot reads pending.

Nothing is paid out automatically. Each closed round produces a payout plan. Payouts are sent by hand after approval, and each one is recorded with its solana tx link on the round. Fees can be small or zero, so there may be nothing to pay.

status: pot pending · payouts pending

Not financial advice. Nothing here promises a return.

EXAMPLES

# start crawling: go to https://crawlpad.garden and press ▶ start crawling
# (runs in your tab: no install, no wallet needed, close the tab to stop)

# what the colony has read so far
$ curl -s https://crawlpad.garden/api/stats | jq '{pages, tokens, verified, domains, crawlersAwake}'

# dataset size and fields, then the first record of part 0
$ curl -s 'https://crawlpad.garden/api/dataset?meta=1'
$ curl -s 'https://crawlpad.garden/api/dataset?part=0' | head -n 1 | jq '{url, title, tokens, verified}'

# download the whole dataset (JSONL)
$ curl -sL -o crawlipede.jsonl https://crawlpad.garden/api/dataset

# this 4h battle round and the last few closed ones
$ curl -s https://crawlpad.garden/api/rounds | jq '{round: .current.id, teams: .current.teams, pot: .current.pot.status, closed: [.history[:3][].id]}'

# one closed round's payout plan (id = round start, YYYY-MM-DDTHH); add ?format=csv for the export
$ curl -s https://crawlpad.garden/api/rounds/2026-10-04T12 | jq '.round | {id, winner, teams, qualifying, payoutStatus}'

# the signals board: verdicts, evidence and the paper-call win rate
$ curl -s https://crawlpad.garden/api/signals | jq '{stats, latest: [.coins[:3][] | {symbol, crowd: .verdict.crowd.verdict, dev: .verdict.dev.verdict, p: .verdict.probUp}]}'

# is voting open, and what leads?
$ curl -s https://crawlpad.garden/api/vote | jq '{open, status, voters, top: [.proposals[:3][].title]}'

FILES

/api/signals: fresh-launch verdicts (crowd, dev, concentration, p(up 1h)) with evidence, paper calls and the live win rate
/api/stats: live totals, leaderboard, swarm, recent crawl log, queue and vote summary
/api/dataset: the dataset as JSONL. ?meta=1 gives size and fields; ?part=N gives one 200-page part
/api/rounds: the current 4h battle round (team scores, leader, ledger), plus the last 24 rounds
/api/side: POST {c, side}: pick spider or crawlipede for the current round (free); locked picks queue for the next round
/api/rounds/<id>: one round (start, YYYY-MM-DDTHH): winner, team scores, ledger, boosts, pot and payout status; ?format=csv = payout export
/api/vote: GET: proposals, tally and allowlisted hosts. POST: suggest a site or cast a signed vote
/api/job: the next page for a crawler to read (?c=<crawler id>)
/api/fetch: read-only proxy for allowlisted pages (?url=)
/api/submit: POST: cleaned page text and hash, for spot-checking and storage
/api/heartbeat: POST: keeps a crawler marked as awake
/api/wallet: POST: a wallet signature that verifies a crawler's address (no transaction)