feat: enhance project configuration and improve data export functionality
- Updated `.gitignore` to streamline ignored files and added logging for common sites. - Expanded `config.py` with new configurations for Weixin and Redis, and improved database connection settings. - Refined `README.md` to clarify project structure and usage instructions. - Enhanced `requirements.txt` with additional dependencies for MongoDB and Redis support. - Refactored multiple spider scripts to utilize a session-based approach for HTTP requests, improving error handling and proxy management. - Updated `export_lawyers_excel.py` to include a default timestamp for data exports.
This commit is contained in:
+14
-2
@@ -1,6 +1,18 @@
|
||||
# 数据库驱动
|
||||
pymysql>=1.0.2
|
||||
pymongo>=4.0.0
|
||||
|
||||
# 调度器
|
||||
schedule>=1.2.0
|
||||
|
||||
# 其他可能需要的依赖
|
||||
requests>=2.28.0
|
||||
beautifulsoup4>=4.11.0
|
||||
urllib3>=1.26.0
|
||||
lxml>=4.9.0
|
||||
openpyxl>=3.1.0
|
||||
redis>=4.0.0
|
||||
pyppeteer>=1.0.2
|
||||
# 可选:提升反检测能力
|
||||
pyppeteer-stealth>=2.7.4
|
||||
|
||||
# 日志相关
|
||||
python-dateutil>=2.8.2
|
||||
|
||||
Reference in New Issue
Block a user