iNocturnis/webcrawler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Added some trap detection for really bad links

Browse Source
This commit is contained in:
Hieuhuy Pham
2022-04-25 15:54:57 -07:00
parent c1b7a50460
commit 8d5a669d9e
3 changed files with 65392 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
spacetime-crawler4py-master/Logs/FRONTIER.log
View File

@@ -140,3 +140,8 @@
2022-04-23 17:25:43,095 - FRONTIER - INFO - Found save file frontier.shelve, deleting it.
2022-04-23 17:58:37,549 - FRONTIER - INFO - Found save file frontier.shelve, deleting it.
2022-04-23 17:58:48,116 - FRONTIER - INFO - Found save file frontier.shelve, deleting it.
2022-04-23 20:19:11,395 - FRONTIER - INFO - Found save file frontier.shelve, deleting it.
2022-04-23 20:33:31,301 - FRONTIER - INFO - Found save file frontier.shelve, deleting it.
2022-04-23 22:13:18,206 - FRONTIER - INFO - Found save file frontier.shelve, deleting it.
2022-04-24 10:06:50,608 - FRONTIER - INFO - Found save file frontier.shelve, deleting it.
2022-04-24 14:10:29,409 - FRONTIER - INFO - Found save file frontier.shelve, deleting it.

65349
spacetime-crawler4py-master/Logs/Worker.log
View File

File diff suppressed because it is too large Load Diff