A little like that j-thing, only in Go.
Go 13.8k 916
Polite, slim and concurrent web crawler.
Go 2k 197
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
Go 781 95
tiny Go library to normalize URLs
Go 468 61
Package rehttp implements a Go HTTP transport that handles retries.
Go 214 17
Proper URL escaping as per RFC3986
Go 27 15
Loading…