9月 062012

Amazon S3 REST-API is necessary to generate signature.
vmod-awsrest generate to Authorization and Date header for Amazon S3.

How to use

VCL

import awsrest;
backend default {
 .host = "s3.amazonaws.com";
 .port = "80";
}
sub vcl_recv{
 awsrest.s3_generic(
 "accessKey", //AWSAccessKeyId
 "secretKey", //SecretAccessKeyID
 req.request, //HTTP-Verb
 req.http.content-md5, //Content-MD5
 req.http.content-type, //Content-Type
 "", //canonicalizedAmzHeaders
 req.url, //canonicalizedResource
 now //Date
 );
}

Output

15 TxHeader b Date: 2012年7月03日 16:21:47 +0000
15 TxHeader b Authorization: AWS accessKey:XUfSbQDuOWL24PTR1qavWSr6vjM=

This module set to req.http.Authorization and req.http.Date, bereq is not use.
I recommend call in the vcl_recv.
And, be careful to default settings.
If req.http.Authorization contains, it is not caching. (default setting)

download here.
libvmod-awsrest


Reference site
PHP で Amazon S3 の REST API を使用 #1
Authenticating REST Requests

日本語はこっち
VarnishでAmazon S3の認証ヘッダを作るVMODを作ってみた


6月 132012

Difficult to access the POST/GET/Cookie value in Varnish.
I want more easily to access it.
Therefore, I tried to make a VMOD to parse.

Feature

  • Support POST/GET request and Cookie header.
  • Support application/x-www-form-urlencoded and multipart/form-data Content-type in POST request.
  • Can be getting a list of key.
  • Support Varnish 3.0.1, 3.0.2 , 3.0.3-rc1

How to use

For example, how to set the response header from the POST key of hoge.
It’s a simple.

import parsereq;
vcl_recv{
 //please write "parsereq.init();" to 1st line in vcl_recv.
 parsereq.init();
}
vcl_deliver{
 set resp.http.hoge = parsereq.post_header("hoge");
}

Future plans

  • urlencode/decode
  • change the value and set to bereq
  • refactoring

I hope that this code is of help to you.

download here.
vmod-parsereq


5月 262012

VMOD processing want to before/after VCL function processing.
But, I do not want to write extra line.
How to do it?


Can be realized by the replace the pointer to sp->vcl->XXXX_func (VCL_XXXX)


Can be added to the processing by replacing pointer

static vcl_func_f *vmod_redirect_Hook_vcl_error = NULL;
static pthread_mutex_t vmod_redirect_mutex = PTHREAD_MUTEX_INITIALIZER;
static unsigned hook_done = 0;
static int vmod_Hook_vcl_error(struct sess *sp){
 .... //write the processing
 //call original vcl_error
 return(vmod_redirect_Hook_vcl_error(sp));
}
int
vmod_location(struct sess *sp, int status, const char*p,...)
{
 //vcl reload detection.
 if(hook_done == 1
 && sp->vcl->error_func != vmod_redirect_Hook_vcl_error ) hook_done = 0;
 if(hook_done == 0){
 //lock to prevent another thread write
 AZ(pthread_mutex_lock(&vmod_redirect_mutex));
 if(hook_done == 0)
 {
 //store the original vcl_error pointer
 vmod_redirect_Hook_vcl_error = sp->vcl->error_func;
 //hook
 sp->vcl->error_func = vmod_Hook_vcl_error;
 hook_done = 1;
 }
 //unlock
 AZ(pthread_mutex_unlock(&vmod_redirect_mutex));
 }
 ....
 return (status);
}

(via:vmod_redirect)

Hook to vcl_error at vmod processing in first time.
because there is no way to write-access the pointer at vmod_Init.


also, has locked using the mutex, because to prevent the loop by thread concurrent access.

I hope that this code is of help to you.

ATTENTION
don’t use varnishadm’s command “vcl.use” and “vcl.discard” . because to the segfault or call to other vcl function.


modify(2012年08月01日)
when you vcl reloaded, hook method be off.


4月 132012

How do ACL match to req.http.X-Forwarded-For? (this is string!!)
I tried to make a vmod.

inet_pton(BOOL ipv6 , STRING str , STRING defaultstr)

example 1

import campur_xcir;
set resp.http.v6 = campur_xcir.inet_pton(true,"2001:0db8:bd05:01d2:288a:1fc0:0001:10ee","1982:db8:20:3:1000:100:20:3");
set resp.http.v4 = campur_xcir.inet_pton(false,"1.1.1.1","2.2.2.2");
set resp.http.v6ng = campur_xcir.inet_pton(true,"2001:0db8:bd05:01d2:288a:1fc0:0001:10eeHOGE","1982:db8:20:3:1000:100:20:3");//NG pattern
set resp.http.v4ng = campur_xcir.inet_pton(false,"1.1.1.1HOGE","2.2.2.2");//NG pattern
//result
v6: 2001:db8:bd05:1d2:288a:1fc0:1:10ee
v4: 1.1.1.1
v6ng: 1982:db8:20:3:1000:100:20:3
v4ng: 2.2.2.2

example 2

import campur_xcir;
//acl
acl local {
 "192.168.1.0"/24;
 !"0.0.0.0";
}
sub vcl_recv{
 if(campur_xcir.inet_pton(false , req.http.X-Forwarded-For , "0.0.0.0") ~ local){
 //acl ok
 ...
 }
}

I hope that this code is of help to you.
libvmod-campur_xcir

this module is my motley function.
Others, get varnish generated hash etc…

this vmod’s function is increase at times. 🙂


3月 202012

To save the split log for every host([hostname].access_log), Apache is easy.
I want the same action in varnishncsa. What should I do?

Use the options -m [tag:regex], -w [file], -a and -D.

-m perform a regex match to the tag’s log entry.
-w write log to a file.
-a append log. Will be overwritten if you do not specify.
-D Daemonize.

exec varnishncsa(host is a.example.net and b.example.net)

varnishncsa -m "RxHeader:^Host: a.example.net$" -a -w /var/log/varnish/a.example.net.access_log -D
varnishncsa -m "RxHeader:^Host: b.example.net$" -a -w /var/log/varnish/b.example.net.access_log -D

after request

cat a.example.net.access_log
192.168.1.199 - - [20/Mar/2012:12:51:50 +0900] "GET http://a.example.net/a HTTP/1.0" 200 280 "-" "Wget/1.12 (linux-gnu)"
cat b.example.net.access_log
192.168.1.199 - - [20/Mar/2012:12:51:59 +0900] "GET http://b.example.net/a HTTP/1.0" 200 280 "-" "Wget/1.12 (linux-gnu)"

work as expected.

If you want to log rotate, please send the SIGHUP.


3月 122012

varnishlogtrans version up(mapping from vcl_trace to vcl source)

Past Articles:convert varnishlog output to easy

Changes

  • Feature: mapping from vcl_trace to vcl source(line).(request from @perbu)
  • Feature: show backend status.(request from @dai_yamashita)
  • Change: Add run option.
  • Experimental: mapping from vcl_trace to vcl source(all).
  • mapping from vcl_trace to vcl source(line).

    if you set to -f option, replace vcl_trace to VCL code
    before.

     | trace | vrt_count:1 vcl_line:55 vcl_pos:1
    

    after(use -f and -e option)

     | trace | set req.http.X-TEST1=campur_xcir.gethash();
     | | ^ (vrt_count:1 vcl_line:55 vcl_pos:1 src_name:input)
    

    show backend status.

    show backend status.
    past status is window 4times.

    backend status
    ------------------------------------------------------------
    name | status | past status
    default | healthy | YY
    

    Add run option.

    -f [vcl file]

    if you set, you can mapping from vcl_trace to vcl source.

    -e [varnishd]

    if you set, you can mapping from vcl_trace to vcl source.

    -cc_command [cc_command]

    if you set in the -f option and varnish’s cc_command option ,please setting.

    –(backend|action|variable) [on|off]

    if you want to disable (backend|action|variable) info , set to off. (default is on)

    mapping from vcl_trace to vcl source(all).

    this function is unstable.
    if you want use this function.
    it need change value and set option(–src=on -f=[vclfile] -e=[varnishd]).

    before

    define('experimental' , false);
    

    after

    define('experimental' , true);
    

    output

    ############################################################
    vcl trace
    [exec] /* "input"*/
    [exec] /*
    ...
    [exec] sub vcl_recv {
    [exec] if (req.restarts == 0) {
    [exec] if (req.http.x-forwarded-for) {
     set req.http.X-Forwarded-For =
     req.http.X-Forwarded-For + ", " + client.ip;
    [exec] } else {
    [exec] set req.http.X-Forwarded-For = client.ip;
    [exec] }
    [exec] }
    [exec] if (req.request != "GET" &&
    [exec] req.request != "HEAD" &&
    [exec] req.request != "PUT" &&
    [exec] req.request != "POST" &&
    ...
    

    I hope that this code is of help to you.

    download here.
    varnishlogTrans


    2月 092012

    Do you use a Varnish? I’ve been using.
    I think to using varnishlog for VCL debugging.

    For example, the output of the varnishlog that include ESI request.

    varnishlog(include ESI)

     13 BackendOpen b default 192.168.1.199 36011 192.168.1.199 81 ★Open backend here.
     13 TxRequest b GET
     13 TxURL b /test2.html
     13 TxProtocol b HTTP/1.1
     13 TxHeader b User-Agent: Wget/1.12 (linux-gnu)
     13 TxHeader b Accept: */*
     13 TxHeader b Host: 192.168.1.199:6081
     13 TxHeader b X-Varnish: 171551254
     13 TxHeader b Accept-Encoding: gzip
     13 RxProtocol b HTTP/1.1
     13 RxStatus b 200
     13 RxResponse b OK
     13 RxHeader b Date: 2012年1月28日 05:05:59 GMT
     13 RxHeader b Server: Apache/2.2.15 (Scientific Linux)
     13 RxHeader b Last-Modified: 2011年7月23日 08:09:05 GMT
     13 RxHeader b ETag: "5a28-51-4a8b8187f8177"
     13 RxHeader b Accept-Ranges: bytes
     13 RxHeader b Content-Length: 81
     13 RxHeader b Connection: close
     13 RxHeader b Content-Type: text/html; charset=UTF-8
     13 Fetch_Body b 4(length) cls 0 mklen 1
     13 Length b 81
     13 BackendClose b default ★Close backend here.
     13 BackendOpen b default 192.168.1.199 36012 192.168.1.199 81 ★Open backend for ESI request here.
     13 TxRequest b GET
     13 TxURL b /test3.html
     13 TxProtocol b HTTP/1.1
     13 TxHeader b User-Agent: Wget/1.12 (linux-gnu)
     13 TxHeader b Accept: */*
     13 TxHeader b Host: 192.168.1.199:6081
     13 TxHeader b X-Varnish: 171551254
     13 TxHeader b Accept-Encoding: gzip
     13 RxProtocol b HTTP/1.1
     13 RxStatus b 200
     13 RxResponse b OK
     13 RxHeader b Date: 2012年1月28日 05:05:59 GMT
     13 RxHeader b Server: Apache/2.2.15 (Scientific Linux)
     13 RxHeader b Last-Modified: 2011年7月23日 08:08:59 GMT
     13 RxHeader b ETag: "5a29-14-4a8b81829c835"
     13 RxHeader b Accept-Ranges: bytes
     13 RxHeader b Content-Length: 20
     13 RxHeader b Connection: close
     13 RxHeader b Content-Type: text/html; charset=UTF-8
     13 Fetch_Body b 4(length) cls 0 mklen 1
     13 Length b 20
     13 BackendClose b default ★Close backend here.
     12 SessionOpen c 192.168.1.199 55115 :6081
     12 ReqStart c 192.168.1.199 55115 171551254 ★Start Request (/test2.html)
     12 RxRequest c GET
     12 RxURL c /test2.html
     12 RxProtocol c HTTP/1.0
     12 RxHeader c User-Agent: Wget/1.12 (linux-gnu)
     12 RxHeader c Accept: */*
     12 RxHeader c Host: 192.168.1.199:6081
     12 RxHeader c Connection: Keep-Alive
     12 VCL_call c recv 1 7.1
     12 VCL_Log c a
     12 VCL_trace c 2 9.1
     12 VCL_Log c b
     12 VCL_trace c 3 11.1
     12 VCL_Log c a
     12 VCL_return c lookup
     12 VCL_call c hash 18 85.5
     12 Hash c /test2.html
     12 VCL_trace c 19 87.9
     12 Hash c 192.168.1.199:6081
     12 VCL_return c hash
     12 VCL_call c miss 22 99.5 fetch
     12 Backend c 13 default default ★Call backend.
     12 TTL c 171551254 RFC 120 -1 -1 1327727160 0 1327727159 0 0
     12 VCL_call c fetch 4 18.1 deliver
     12 ObjProtocol c HTTP/1.1
     12 ObjResponse c OK
     12 ObjHeader c Date: 2012年1月28日 05:05:59 GMT
     12 ObjHeader c Server: Apache/2.2.15 (Scientific Linux)
     12 ObjHeader c Last-Modified: 2011年7月23日 08:09:05 GMT
     12 ObjHeader c ETag: "5a28-51-4a8b8187f8177"
     12 ObjHeader c Content-Type: text/html; charset=UTF-8
     12 VCL_call c deliver 26 116.5 deliver
     12 TxProtocol c HTTP/1.1
     12 TxStatus c 200
     12 TxResponse c OK
     12 TxHeader c Server: Apache/2.2.15 (Scientific Linux)
     12 TxHeader c Last-Modified: 2011年7月23日 08:09:05 GMT
     12 TxHeader c ETag: "5a28-51-4a8b8187f8177"
     12 TxHeader c Content-Type: text/html; charset=UTF-8
     12 TxHeader c Date: 2012年1月28日 05:05:59 GMT
     12 TxHeader c X-Varnish: 171551254
     12 TxHeader c Age: 0
     12 TxHeader c Via: 1.1 varnish
     12 TxHeader c Connection: close
     12 VCL_call c recv 1 7.1 ★Start ESI log.
     12 VCL_Log c a
     12 VCL_trace c 2 9.1
     12 VCL_Log c b
     12 VCL_trace c 3 11.1
     12 VCL_Log c a
     12 VCL_return c lookup
     12 VCL_call c hash 18 85.5
     12 Hash c /test3.html
     12 VCL_trace c 19 87.9
     12 Hash c 192.168.1.199:6081
     12 VCL_return c hash
     12 VCL_call c miss 22 99.5 fetch
     12 Backend c 13 default default
     12 TTL c 171551254 RFC 120 -1 -1 1327727160 0 1327727159 0 0
     12 VCL_call c fetch 4 18.1 deliver
     12 ObjProtocol c HTTP/1.1
     12 ObjResponse c OK
     12 ObjHeader c Date: 2012年1月28日 05:05:59 GMT
     12 ObjHeader c Server: Apache/2.2.15 (Scientific Linux)
     12 ObjHeader c Last-Modified: 2011年7月23日 08:08:59 GMT
     12 ObjHeader c ETag: "5a29-14-4a8b81829c835"
     12 ObjHeader c Content-Type: text/html; charset=UTF-8
     12 VCL_call c deliver 26 116.5 deliver
     12 Length c 20
     12 ReqEnd c 171551254 1327727159.889132023 1327727159.890880346 -0.001736164 nan nan ★Request end.
     12 SessionClose c EOF mode
     12 StatSess c 192.168.1.199 55115 0 1 1 0 0 2 282 20
     0 CLI - Rd ping
     0 CLI - Wr 200 19 PONG 1327727160 1.0
    

    little too difficult for me.
    Therefore, I tried to make a tool to format and display.
    Continue reading »


    2月 052012

    Varnish redirect is pain.

    sub vcl_recv {
     if (req.http.user-agent ~ "iP(hone|od)") {
     error 750 "Moved Temporarily";
     }
    }
    sub vcl_error {
     if (obj.status == 750) {
     set obj.http.Location = "http://www.example.com/iphoneversion/";
     set obj.status = 302;
     return(deliver);
     }
    }
    

    (via:Redirecting using VCL)
    I just want to redirect. but should write code in two action.
    This is it lack of maintenance, and I do not like.
    I tried to make a vmod_redirect can be easily redirect.
    Continue reading »


    7月 132011

    If you manage many objects with high traffic , Varnish get a swap (although it rare)

    Varnish 3 have new parameter “shortlived”.
    This parameter description is

    Objects created with TTL shorter than this are always put in transient storage.

    transient storage is malloc storage.

    storage.list
    200
    Storage devices:
     storage.Transient =*** malloc ***
     storage.s0 = malloc
    

    Continue reading »


    7ads6x98y

    AltStyle によって変換されたページ (->オリジナル) /