-
-
Save troy/184547 to your computer and use it in GitHub Desktop.
| # usage: redfin-images "http://www.redfin.com/WA/Seattle/123-Home-Row-12345/home/1234567" | |
| function redfin-images() { | |
| wget -O - $1 | grep "full:" | awk -F \" '{print $4}' | xargs wget - | |
| } |
| wget -O - http://www.redfin.com/WA/Seattle/123-Home-Row-12345/home/1234567 | grep "full:" | awk -F \" '{print $4}' | xargs wget - |
@punjabdhaputar Thank you for sharing. It works!
Was able to understand how it works by first saving wget output, then found the format that egrep is looking for in that document, and ascii2uni is changing the format back to something that wget can use. Good stuff and thanks again for sharing!
@punjabdhaputar worked for me too, thanks!!
Any idea how to download the photos that are visible after signing in?
@gauravchak: After using a browser to log in, you might be able to change wget to present session cookies from the browser. For that, look into the --load-cookies option. You'd need to manually create the cookie file.
Assuming you're just saving images from a handful of listings for personal use (which is what this script was intended for), one of these methods might be easier than adding cookie support:
- In Firefox, choose Tools -> Page Info, select the Media tab, highlight multiple image URLs in the listing, and click "Save as." This probably won't show high-res images that are only shown in an interactive gallery (lightbox), but it will at least show the average-size images. If you need high-res images, you can probably find a different real estate site that does show all of the high-res images in one page and use the same technique.
- Use a "Save all images" browser extensions (example: https://github.com/belaviyo/save-images - I haven't personally used it). Browser extensions are risky, so look for a trusted one with lots of users and comments (and ideally, public source code), and uninstall it as soon as you're done.
It was working well earlier this year, but now it just downloads a single image. Does anyone know how to adjust it to download all images? Thanks
@punjabdhaputar ...this still works! thanks!
@punjabdhaputar just tried this and was able to save myself several right click-save trouble...thanks!
It still works.
@gauravchak managed to make it work for listings that require signing in, by using the method outlined here: How do I use wget/curl to download from a site I am logged into?.
- Logged into Redfin in Firefox.
- Open "Network" tab of Web Developer tool: Ctrl-Shift-E
- I took the very first request that was sent when I refreshed the screen on Firefox.
- Pasted it in Sublime, and saw a large amount of cookie values in there. To figure out where cookies started and stopped, I just searched for "-H" in the file and took what was there for cookies only. I took everything that was in between
'Cookie: key1=value1; key2=value2; [....]; keyn=valuen' - Recreated the wget command as such:
wget --no-cookies --header "Cookie: key1=value1; key2=value2; [....]; keyn=valuen" --user-agent="Mozilla" -O - <RedFinURL> | egrep -o "https:\\\\u002F\\\\u002Fssl.cdn-redfin.com\\\\u002Fphoto\\\\u002F\d*\\\\u002Fbigphoto\\\\u002F\d*\\\\u002F[A-Z0-9_]*.jpg" | ascii2uni -Z '\u%04X' | xargs wget --user-agent="Mozilla"
And that did the trick. Hope that helps.
@punjabdhaputar Thanks, it works for me.
Redfin seems to be blocking this now, getting 403 Forbiden
I created a lil Go program to do this https://github.com/timendez/go-redfin-archiver
Clone repo, and just run e.g. go run archive.go https://www.redfin.com/CA/San-Jose/206-Grayson-Ter-95126/home/2122534
@timendez I just tried your Go program and it worked great. Nice work!
For anyone else who encounters this gist: Strongly consider using @timendez's program instead: https://github.com/timendez/go-redfin-archiver
Following worked for me on my mac
1)
brew install uni2ascii2)
wget --user-agent="Mozilla" -O - <RedFinURL> | egrep -o "https:\\\\u002F\\\\u002Fssl.cdn-redfin.com\\\\u002Fphoto\\\\u002F\d*\\\\u002Fbigphoto\\\\u002F\d*\\\\u002F[A-Z0-9_]*.jpg" | ascii2uni -Z '\u%04X' | xargs wget --user-agent="Mozilla"