-
-
Save thoop/8072354 to your computer and use it in GitHub Desktop.
# Change YOUR_TOKEN to your prerender token | |
# Change http://example.com (at the end of the last RewriteRule) to your website url | |
<IfModule mod_headers.c> | |
RequestHeader set X-Prerender-Token "YOUR_TOKEN" | |
</IfModule> | |
<IfModule mod_rewrite.c> | |
RewriteEngine On | |
<IfModule mod_proxy_http.c> | |
RewriteCond %{HTTP_USER_AGENT} googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator [NC,OR] | |
RewriteCond %{QUERY_STRING} _escaped_fragment_ | |
# Only proxy the request to Prerender if it's a request for HTML | |
RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent|\.ttf|\.woff|\.svg))(.*) http://service.prerender.io/http://example.com/$2 [P,L] | |
</IfModule> | |
</IfModule> |
I don't get notified when someone comments on this gist. Send us an email at [email protected] if anyone has any questions or issues.
it doesn't work on my site
RewriteEngine On
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR]
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d
RewriteRule ^ - [L]
RequestHeader set X-Prerender-Token "mytoken"
RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Yandex(Bot|Images|Video|Media) [NC,OR]
RewriteCond %{QUERY_STRING} _escaped_fragment_
RewriteRule ^(.*)$ http://service.prerender.io/http://%{HTTP_HOST}$1 [P,L]
# Allow Robots.txt to pass through
RewriteRule ^robots.txt - [L]
RewriteRule ^ /index.html [L]
please help me to check config
If you have prerender working and running for all pages except the root(home) page.
Your .htaccess might have the URL set to /index.php or /index.shtml for the homepage. If so, you could change your rewrite rule to this:
RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent|\.ttf|\.woff))(index\.php)?(.*) http://service.prerender.io/http://www.example.com$3 [P,L]
I added (index.php or index.shtml)? to capture that out if found and changed the $2 to $3. Don't forget to change the http://www.example.com to your domain.
Hi i have used prerender for my angular project i did everything for activate prerender however not success
My web site url is:http://ugurdogrusoy.com/preparing/
firstly addred
Then added prerender token inside .htaccess you can see rule and token below
RewriteEngine On # If requested resource exists as a file or directory # (REQUEST_FILENAME is only relative in virtualhost context, so not usable) RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR] RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d # Go to it as is RewriteRule ^ - [L]# If non existent
# If path ends with / and is not just a single /, redirect to without the trailing /
RewriteCond %{REQUEST_URI} ^.*/$
RewriteCond %{REQUEST_URI} !^/$
RewriteRule ^(.*)/$ $1 [R,QSA,L]
# Handle Prerender.io
RequestHeader set X-Prerender-Token "MY_TOKEN"
RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR]
RewriteCond %{QUERY_STRING} _escaped_fragment_
# Proxy the request
RewriteRule ^(.*)$ http://service.prerender.io/http://%{HTTP_HOST}/preparing/$1 [P,L]
# If non existent
# Accept everything on index.html
RewriteRule ^ /index.html
i can't escape We haven't seen a request with your prerender token yet
where have i done wrong you can help me ?
Thanks
Any Idea how to integrate that with tomcat? seems P flag in Tomcat doesn't work.
If anybody else is having a problem with always returning index file if it's not a crawler, so your SPA router can catch 404, I created a small gist -
https://gist.github.com/markokosir/77e8df3b7cf0b00023178c907f5c998e
A warning to anyone using vector files. You will need to add a |\.svg
capture group to the proxy rule.
If anyone needs to redirect .html files only (skipping extensions .xhtml,.shtml,.phtml,.jp,.cgi,.ppl,.pl,.perl,.php)
Here is my fork: https://gist.github.com/BorisPocatko/77f8f3812d05b934556854267a48150f
.htaccess
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR] RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d # Go to it as is RewriteRule ^ - [L] RewriteCond %{REQUEST_URI} ^.*/$ RewriteCond %{REQUEST_URI} !^/$ RewriteRule ^(.*)/$ $1 [R,QSA,L] RequestHeader set X-Prerender-Token "%MYTOKEN%" RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR] RewriteCond %{HTTP_USER_AGENT} Yandex(Bot|Images|Video|Media) [NC,OR] RewriteCond %{QUERY_STRING} _escaped_fragment_ RewriteRule ^(.*)$ http://service.prerender.io/http://%{HTTP_HOST}$1 [P,L] # Accept everything on index.html RewriteRule ^ /index.html`
in index.html
But robots can't see my site.
On prerender.io i have 67 cached pages.Could you please help me.
@Dizzyspb Did you fix it?
Hey all, I'm running into a problem when trying to Rewrite part of the URL being cached in prerender.io. I've tried removing that part of the URL using:
Rewrite ^root-directory/parent-directory/dist/(.*)$ $1 [R,L]
Doing this inside of:
<IfModule mod_proxy_http.c>
Rewrite ^root-directory/parent-directory/dist/(.*)$ $1 [R,L]
RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator [NC,OR]
RewriteCond %{QUERY_STRING} _escaped_fragment_
# Only proxy the request to Prerender if it's a request for HTML
RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent|\.ttf|\.woff))(.*) http://service.prerender.io/https://www.example.com/$2 [P,L]
</IfModule>
Doing this on the live server results in TOO_MANY_REDIRECTS
. Is there anyway to solve this? I've changed the https://www.example.com/$2
to represent my sites URL
Hi i have used prerender for my angular project i did everything for activate prerender however not success
My web site url is:http://ugurdogrusoy.com/preparing/
firstly addred
Then added prerender token inside .htaccess you can see rule and token below
RewriteEngine On # If requested resource exists as a file or directory # (REQUEST_FILENAME is only relative in virtualhost context, so not usable) RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR] RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d # Go to it as is RewriteRule ^ - [L]
# If non existent # If path ends with / and is not just a single /, redirect to without the trailing / RewriteCond %{REQUEST_URI} ^.*/$ RewriteCond %{REQUEST_URI} !^/$ RewriteRule ^(.*)/$ $1 [R,QSA,L] # Handle Prerender.io RequestHeader set X-Prerender-Token "MY_TOKEN" RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR] RewriteCond %{QUERY_STRING} _escaped_fragment_ # Proxy the request RewriteRule ^(.*)$ http://service.prerender.io/http://%{HTTP_HOST}/preparing/$1 [P,L] # If non existent # Accept everything on index.html RewriteRule ^ /index.html
i can't escape We haven't seen a request with your prerender token yet
where have i done wrong you can help me ?
Thanks
I have had the same issue, our site is using react, and we even tried to shift the config file further up to the root. Still the same issue. Any luck with this ?
you are awesome!! i almost wept when i finally saw the token not used message go away. thanks!!
Just adding 2 cents:
On my hosting provider (Dreamhost), the [NC,OR] yielded strange results when just copy/pasting. On RewriteCond with only one line, I had to write it as
RewriteCond … googlebot|yandex|…|…|… [NC]
(re-writing the RewriteCond with one per line also worked, but not with [OR] on the first line. This would work:)
RewriteCond … googlebot [NC]
RewriteCond … yandex [NC,OR]
RewriteCond … WhatsApp [NC,OR]
note the seemingly missing OR on the first line
On the other hand, my second cent is that last WhatsApp entry - turns out WhatsApp does the scraping directly from inside the app (at least it did on my android phone today ;) So my full line is now
RewriteCond %{HTTP_USER_AGENT} googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|WhatsApp [NC]
(And my full htaccess
RewriteEngine On
# https://stackoverflow.com/questions/18406156/redirect-all-to-index-php-using-htaccess
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{HTTP_USER_AGENT} googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|WhatsApp [NC]
# RewriteCond %{HTTP_USER_AGENT} facebookexternalhit|googlebot [NC] MUST BE WRITTEN WITHOUT OR
# RewriteCond %{HTTP_USER_AGENT} googlebot [NC]
# RewriteCond %{HTTP_USER_AGENT} facebookexternalhit [NC,OR] 'OR' IS FOR SECOND LINE (AND THIRD AND FOURTH ETC. WON'T WORK ON FIRST LINE)
RewriteRule ^(.*)$ opengraph.php?q=$1 [NC,L,QSA]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ redir.php?orig_path=$1 [NC,L,QSA]
</IfModule>
Angular 7 + Apache2: work for me. Ping me if you have any issue
.htaccess: https://gist.github.com/tyluudinh/2149d6cc62219c141f619682b5378c47
.apache2.conf: https://gist.github.com/tyluudinh/bc257a1aa2a4b8826ff28916238cc27d
@tyluudinh This worked for me, thank you 👍
interesting note I found re: iMessage User Agent. Hope this helps someone!
iMessage User Agent