Web based system to purge multiple Varnish cache servers

Posted by: Admin  :  Category: Web Hosting

Linux Cpanel shared hosting: 600 GB disk space, 6 TB bandwidth, free domain, unlimited databases and ftp accounts, web hosting cheap and pro at Hostony

Hello!

We have been working with varnish for quite a while. And there is quite a lot of documentation out there already for the different methods for purging cache remotely via Curl, the varnish admin tool sets and other related methods.

We deal with varnish in the Amazon Cloud as well as on dedicated servers. In many cases varnish sits in a pool of servers in the web stack before the web services such as Nginx and Apache. Sometimes purging specific cache urls can be cumbersome when you’re dealing with multiple cache servers.

Depending on the CMS you are using, there is some modules / plugins that are available that offer the ability to purge Varnish caches straight from the CMS, such as the Drupal Purge module.

We have decided to put out a secure, web accessible method for purging Varnish cached objects across multiple varnish servers. As always, take the word “secure” with a grain of salt. The recommended way to publish a web accessible method on apache or nginx that gives the end-user the ability to request cache pages be purged would be to take these fundamentals into consideration :

– Make the web accessible page available only to specific source IPs or subnets
– Make the web accessible page password protected with strong passwords and non-standard usernames
– Make the web accessible page fully available via SSL encryption

On the varnish configuration side of things, with security still in mind, you would have to set up the following items in your config :

ACL

Set up an access control list in varnish that only allows specific source IPs to send the PURGE request. Here is an example of one :

# ACL For purging cache
acl purgers {
        "127.0.0.1";
        "192.168.0.1"/24;
}

vcl_recv / vcl_hit / vcl_miss / vcl_pass

Linux Cpanel shared hosting: 600 GB disk space, 6 TB bandwidth, free domain, unlimited databases and ftp accounts, web hosting cheap and pro at Hostony

This is self explanatory (I hope). Obviously you would be integrating the following logic into your existing varnish configuration.

sub vcl_recv {
        if (req.request == "PURGE") {
                if (!client.ip ~ purgers) {
                        error 405 "Method not allowed";
                }
                return (lookup);
        }
}

sub vcl_hit {
        if (req.request == "PURGE") {
                purge;
                error 200 "Purged";
        }
}
sub vcl_miss {
        if (req.request == "PURGE") {
                purge;
                error 404 "Not in cache";
        }
}
sub vcl_pass {
        if (req.request == "PURGE") {
                error 502 "PURGE on a passed object";
        }
}

The code itself is available on our GitHub Project page. Feel free to contribute and add any additional functionality.

It should be important to note that what differentiates our solution among the existing ones out there is that our script will manipulate the host headers of the Curl request in order to submit the same hostname / url request across the array of varnish servers. That way the identical request can be received by multiple varnish servers with no local host file editing or anything like that.

There is lots of room for input sanity checks, better input logic and other options to perhaps integrate with varnish more intuitively. Remember this is a starting point, but hopefully it is useful for you!

The post Web based system to purge multiple Varnish cache servers appeared first on Managed WordPress Hosting | Managed VPS Hosting | Stack Star.

Managed WordPress Hosting | Managed VPS Hosting | Stack Star

Linux Cpanel shared hosting: 600 GB disk space, 6 TB bandwidth, free domain, unlimited databases and ftp accounts, web hosting cheap and pro at Hostony

Use Varnish and Nginx to follow, hide and cache 301 / 302 redirects

Posted by: Admin  :  Category: Web Hosting

Hello!

Varnish is a robust, open source and stable caching solution that has been employed on many different high traffic environments with a significant amount of success.

One of the things that we have come across, specifically with environments such as Amazon Web Services is that websites tend to spread their web stack across multiple services. For example static media such as JS, CSS and image files may be hosted on Amazon S3 storage. This would require either implementing additional CNAMES for your domain (i.e. static.yourdomain.com) that point to the S3 URL, or have your CMS redirect requests for static media to the S3 backend.

Remember with S3, you have to generate the static files and copy them over to S3, so these URLs may need to be generated and maintained by the CMS often times with a redirect (301 or 302) that rewrites the URL to the S3 backend destination.

When Varnish is caching a website, and it comes across a request that is rewritten in a 301/302 redirect by the backend response (“beresp”), varnish typically will simply cache the 301/302 redirect as a response, saving that minuscule amount of processing power that needed to happen to process the request and send the rewrite. Some may argue that that is simply negligible!

Wouldn’t it be nice to actually cache the content after the redirect happens? There’s two ways one could go about doing this.

Cache 301 / 302 Redirects with Varnish 4

In simpler setups, Varnish can simply process the rewrite/redirect and change the url and issue a return(restart) to restart the request with the new URL. This means that Varnish is processing the rewrite and returning the new url so that it can be re-requested and ultimately cached. So in vcl_deliver you can add the following :

# Cache 301/302 redirects
        if ((resp.status == 301) || (resp.status == 302)) {
               set req.url = regsub(resp.http.Location,"^http://[^/]+(.*)","");
               return(restart);
        }

The above should work for you if, lets say, you are using Varnish in front of a simple apache/nginx server that is then processing the request. If Varnish is sending traffic to another proxy (i.e. nginx + proxy_pass), then this above directive may not work for you. The reason why one may want to proxy traffic from varnish to another proxy like nginx may be in a scenario where you want to do some fancy redirection of traffic + DNS resolution. Still confused?

Lets say varnish is at the edge of a network, caching a complicated website. Requests to varnish need to go to another load balancer (i.e. an Amazon ELB). ELB endpoints are never going to be a static IP address and Varnish (as of v4) cannot do DNS resolution of hostnames on a per request basis, so you would need to proxy the request to Nginx which would handle the reverse proxy over to ELB which would then load balance the backend fetch to the CMS.

If your scenario sounds more like the aforementioned one, then you could try following the 301/302 redirect with nginx instead of varnish.

Cache 301 / 302 Redirects with Nginx

Nginx and Varnish seem to go hand in hand. They’re great together! In this scenario you are using Varnish as your edge cache and sending all backend requests to an nginx proxy_pass directive. In order to tell Nginx to follow a redirect before sending any response to Varnish (and ultimately the end-user), you can tell varnish to simply save the redirect location and return the response after redirecting back to Varnish so it can simply cache the response!

location / {
                proxy_pass  http://backend-server.com;
                proxy_set_header Host $  http_host;
                proxy_intercept_errors on;
                error_page 301 302 307 = @handle_redirects;
        }
        location @handle_redirects {
                set $  saved_redirect_location '$  upstream_http_location';
                proxy_pass $  saved_redirect_location;
        }

You can see that the proxy_pass directive is configured normally. In the event of any 301, 302 or 307, process it with the @handle_redirects location directive. Then simply proxy pass the $ saved_redirect_location as if it were the backend server! This means that even if the proxy_pass location is not even in your Varnish configuration as a valid hostname (i.e. random S3 url) Varnish will still cache it, thinking it is backend-server.com.

Hopefully this will help someone out there!

The post Use Varnish and Nginx to follow, hide and cache 301 / 302 redirects appeared first on Managed WordPress Hosting | Managed VPS Hosting | Stack Star.

Managed WordPress Hosting | Managed VPS Hosting | Stack Star

Yahoo! Cache = Web Forgery???

Posted by: Admin  :  Category: Domain Names & DNS

Every once in a while, I  like to check the cache of the major search engines, to see how current the snapshot of my site is.  Was checking Yahoo the other day, and was shocked to see this message pop up:

Yahoo! cache is a web forgery?  Yikes!  What a mess!

The IP address, 74.6.238.254, points to Inktomi Corporation, which was acquired by Yahoo! back in 2002.

Are you paying attention Ms. Mayer?
Domain Blog