Python/ImageScrapper
Archived
2
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Update 'scrape.py'

Browse Source 
Fixed dedupe
This commit is contained in:
justine 2021-08-12 00:54:29 +00:00
parent b2e7deb058
commit ca78d58458
1 changed files with 2 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
scrape.py
View File

@ -84,6 +84,8 @@ def item_dl(sources, dlfolder):
for source in sources: for source in sources:
fullsource = "http://" + source fullsource = "http://" + source
if dlfolder[-1] == "/":
dlfolder = dlfolder[:-1]
if not deduplicate(fullsource, dlfolder.split("/")[-1]): if not deduplicate(fullsource, dlfolder.split("/")[-1]):
name = wget.download(fullsource, out=dlfolder) name = wget.download(fullsource, out=dlfolder)
print(f"{name} downloaded") print(f"{name} downloaded")