hello-dd-code
38e7c284e8
- Updated `.gitignore` to streamline ignored files and added logging for common sites. - Expanded `config.py` with new configurations for Weixin and Redis, and improved database connection settings. - Refined `README.md` to clarify project structure and usage instructions. - Enhanced `requirements.txt` with additional dependencies for MongoDB and Redis support. - Refactored multiple spider scripts to utilize a session-based approach for HTTP requests, improving error handling and proxy management. - Updated `export_lawyers_excel.py` to include a default timestamp for data exports.
19 lines
287 B
Plaintext
19 lines
287 B
Plaintext
# 数据库驱动
|
|
pymysql>=1.0.2
|
|
pymongo>=4.0.0
|
|
|
|
# 调度器
|
|
schedule>=1.2.0
|
|
|
|
# 其他可能需要的依赖
|
|
requests>=2.28.0
|
|
beautifulsoup4>=4.11.0
|
|
lxml>=4.9.0
|
|
redis>=4.0.0
|
|
pyppeteer>=1.0.2
|
|
# 可选：提升反检测能力
|
|
pyppeteer-stealth>=2.7.4
|
|
|
|
# 日志相关
|
|
python-dateutil>=2.8.2
|