Tap4 AI Crawler

This is an open-source AI web scraping project by Tap4 AI Tools Directory. Our goal is to provide an automated web scraping capability based on the tools directory. This project is based on Python, very lightweight, easy to maintain, suitable for individual developers interested in AI tools directories, and also for learners interested in Python. We welcome everyone to fork and star.

Follow Us

Follow us on Twitter: https://x.com/tap4ai

Features

• Support for fetching titles, descriptions, and introductions of specified websites
• Support for generating screenshots of specified websites
• Support for using llama3 tool (Groq) to process website introductions and generate Markdown descriptions
• Quick configuration
• Fast deployment

Quick start

Configuring environment variables in Zeabur. The environment variables are as follows:

• GROQ_API_KEY: Key for Groq, apply for it Here
• S3_ENDPOINT_URL: Endpoint for S3(such as Cloudflare R2), apply for R2
• S3_BUCKET_NAME: Bucket name for S3(such as Cloudflare R2)
• S3_ACCESS_KEY_ID: Access key ID for S3(such as Cloudflare R2)
• S3_SECRET_ACCESS_KEY: Secret access key for S3(such as Cloudflare R2)
• S3_CUSTOM_DOMAIN: Custom domain for S3(such as Cloudflare R2), if you have a custom domain, fill it in; otherwise, it can be left blank.

Links to our products

TAP4-AI-Directory

The Collection for the AI tools all over the world. | Collect free ChatGPT mirrors, alternatives, prompts, other AI tools, etc. For more, please visit: Tap4 AI

The Tattoo AI Generator and Design

Tattoo AI Design is a tattoo AI generator and design tool for tattoo fans. If you are interested, visit Tattoo AI Design

Sponsor List

Anime Girl Studio -- AI Anime Girl Chat & Generator

Anime Girl Studio is the AI anime girl generator and chat product. You can generate what you like and chat with the AI anime girl, please visit Anime Girl Studio

Best AI Girl Friend -- Best AI Girl Chat & Generator

Best AI Girl Friend is the AI girl generator and chat product. You can generate what you like and chat with the AI anime girl, please visit Best AI Girl Friend