Search results
65 packages found
Parser for robots.txt
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
A specification compliant robots.txt parser with wildcard (*) matching support.
A selector-based html snapshot tool using Puppeteer or PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
Generate a robots.txt for Astro
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
A TypeScript-based project for easy serving of static files with support for live reloading, compression, and typed requests.
- TypeScript
- static file serving
- live reload
- compression
- typed requests
- HTTP server
- SSL
- cors
- express middleware
- proxy
- sitemap
- feeds
- robots.txt
- compression (gzip, deflate, brotli)
Vite plugin for generating robots.txt for production and development builds
Sitemap generator for TypeScript projects
A proper robots.txt parser and combinator that works with eulalie
The grunt task for html-snapshots
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
## About
An Eleventy plugin to generate a robots.txt file for your static site
Check user agents against robots.txt files
Awesome generator robots.txt
A webpack plugin to output a robots.txt file