2月 092017
 

(vmod-vsthrottle)
使
DLα

vmod-tcp()


vmod-tcptc-fqSO_MAX_PACING_RATE
3.13UbuntuTrustyCentOS(73.10)


varnish-modules

tc-fq


tc-fq使
tc(Linux)

qdisc

xcir@gw01:~$ tc qdisc show dev eth0
qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1


eth0()
pfifo_fast

fq

xcir@gw01:~$ sudo tc qdisc add dev eth0 root handle 1: fq
xcir@gw01:~$ tc qdisc show dev eth0
qdisc fq 1: root refcnt 2 limit 10000p flow_limit 100p buckets 1024 quantum 3028 initial_quantum 15140


qdiscfq
vmod-tcp

使



dat

sub vcl_recv {
  if(req.url ~"\.dat(\?.*)?$"){
    // 1000KB/s(8Mbps)
    tcp.set_socket_pace(1000);
  }
}






xcir@gw01:~$ curl http://*****:6081/10MB.dat  > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10.0M  100 10.0M    0     0   944k      0  0:00:10  0:00:10 --:--:--  943k

xcir@gw01:~$ curl http://*****:6081/10MB.dat2  > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10.0M  100 10.0M    0     0   260M      0 --:--:-- --:--:-- --:--:--  263M


10MB943KiB/s8Mbps(10MB.dat)
263MiB/s


使

keep-alive
tc-fq
keep-alive


xcir@gw01:~$ wget http://*****:6081/10MB.dat  http://*****:6081/10MB.dat2
--2017-02-09 20:02:10--  http://*****:6081/10MB.dat
Connecting to *****:6081... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10485760 (10M) [application/x-ns-proxy-autoconfig]
Saving to: '10MB.dat'

100%[============================================================>] 10,485,760   943KB/s   in 11s

2017-02-09 20:02:20 (945 KB/s) - '10MB.dat' saved [10485760/10485760]

--2017-02-09 20:02:20--  http://*****:6081/10MB.dat2
Reusing existing connection to *****:6081.
HTTP request sent, awaiting response... 200 OK
Length: 10485760 (10M)
Saving to: '10MB.dat2'

100%[============================================================>] 10,485,760   942KB/s   in 11s

2017-02-09 20:02:31 (944 KB/s) - '10MB.dat2' saved [10485760/10485760]

FINISHED --2017-02-09 20:02:31--
Total wall clock time: 22s
Downloaded: 2 files, 20M in 22s (945 KB/s)
xcir@gw01:~$


10MB.dat2

dat(8Mbps)8Mbps
0vcl_recv

sub vcl_recv {
//8Gbps
tcp.set_socket_pace(1000000);
  if(req.url ~"\.dat(\?.*)?$"){
    // 1000KB/s(8Mbps)
    tcp.set_socket_pace(1000);
  }
}




xcir@gw01:~$ wget http://*****:6081/10MB.dat  http://*****:6081/10MB.dat2
--2017-02-09 20:03:40--  http://*****:6081/10MB.dat
Connecting to *****:6081... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10485760 (10M) [application/x-ns-proxy-autoconfig]
Saving to: '10MB.dat'

100%[============================================================>] 10,485,760   943KB/s   in 11s

2017-02-09 20:03:51 (945 KB/s) - '10MB.dat' saved [10485760/10485760]

--2017-02-09 20:03:51--  http://*****:6081/10MB.dat2
Reusing existing connection to *****:6081.
HTTP request sent, awaiting response... 200 OK
Length: 10485760 (10M)
Saving to: '10MB.dat2'

100%[============================================================>] 10,485,760  47.7MB/s   in 0.2s

2017-02-09 20:03:51 (47.7 MB/s) - '10MB.dat2' saved [10485760/10485760]

FINISHED --2017-02-09 20:03:51--
Total wall clock time: 11s
Downloaded: 2 files, 20M in 11s (1.81 MB/s)




TLSMWlocal
VarnishhttpsTLSnginxhaproxyhitch
lotc-fq

http/2
http/21
DL使



10月 032016
 

[] []
HTTP/2DirectorConsistent Hash
HTTP/2使

Hitch1.4.0+Varnish5HTTP/2使


VarnishHTTPS
HTTP/2
VarnishHTTP/2TLS
VarnishSoftwareHitch
HTTP/21.4.0
Ubuntu 16.04build

Hitch


sudo apt-get install libev-dev libssl-dev automake python-docutils flex bison pkg-config
git clone https://github.com/varnish/hitch.git
cd hitch
./bootstrap
./configure --prefix=/opt/hitch/
make
sudo make install


(prefix
HTTP/2使NPNALPN
NPNOpenSSL1.0.1ALPN1.0.2Chrome6NPNHTTP/2ALPN
HTTP/2ALPN
FireFox49NPN1.0.1

Hitch

HTTPS:443           HTTP:80
   ↓               ↓
Hitch:443--(localhost:81)-->Varnish:80
                 ↓
                Origin



Hitch

#/etc/hitch/hitch.conf
frontend       = "[*]:443"
ciphers        = "ECDHE+RSAGCM:ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:EECDH+HIGH:EDH+HIGH:HIGH:+3DES:!RC4:!MD5:!aNULL:!eNULL:!LOW:!EXP:!PSK:!SRP:!DSS:!KRB5:!DH"
backend        = "[127.0.0.1]:81"
write-proxy-v2 = on
workers        = 2
backlog        = 1024
keepalive      = 3600
syslog         = on
user           = "nobody"
daemon         = on
alpn-protos    = "h2,http/1.1"
pem-file       = "/etc/hitch/******.pem"


ciphers
hitchpem-file
 +  + CA + dhparam

Lets Encrypt
privkey.pem + fullchain.pem + dhparam

vhostspem-file

Hitch

/opt/hitch/sbin/hitch --config=/etc/hitch/hitch.conf


Varnish
Varnish
Listen
80HTTP81PROXY


-a :80 \
-a 127.0.0.1:81,PROXY \


HTTP/2
featurehttp2
varnishadm

root@proxy02:~# varnishadm param.set feature +http2

root@proxy02:~# varnishadm param.show feature
feature
        Value is: +esi_disable_xml_check,+http2
        Default is: none

        Enable/Disable various minor features.
           none                       Disable all features.

        Use +/- prefix to enable/disable individual feature:
           short_panic                Short panic message.
           wait_silo                  Wait for persistent silo.
           no_coredump                No coredumps.
           esi_ignore_https           Treat HTTPS as HTTP in
                                      ESI:includes
           esi_disable_xml_check      Don't check of body looks like
                                      XML
           esi_ignore_other_elements  Ignore non-esi XML-elements
           esi_remove_bom             Remove UTF-8 BOM
           https_scheme               Also split https URIs
           http2                      Support HTTP/2 protocol





-p feature=+http2 \







HTTP/2



2.1->3.0


VMODVCL


3.0->4.0


Client/Backend

VCL


4.0->4.1


PROXY




4.1->5.0


HTTP/2

shard director(consistent-hash)

PROXY

VCL使label


4.0HTTP/2
5.03.0->4.0VCL
VCL
replace
stats
LOGHTTP/2
VMOD
4.1使
10/0320
3.0->4.0
HTTP/2使4.0->4.1

HTTP/2(experimental)


HTTP/2(Very Experimental)
varnishadmfeature+http2
111(


vmod-directorshared


consistent-hash
UPLEXvslp
consistent-hash
backend使


VOID shard.set_rampup(DURATION duration=0)
0healthy100%(duration)(slow start)

BOOL shard.add_backend(BACKEND backend, STRING ident=0, DURATION rampup=973279260)

ident使
使weight使
rampup(973279260magic valueset_rampup使

BOOL shard.remove_backend(BACKEND backend=0, STRING ident=0)

backend

BOOL shard.clear(PRIV_TASK)


VOID shard.set_warmup(REAL probability=0.0)
(0.5=50%)

warm



rampupwarmup

BOOL shard.reconfigure(INT replicas=67, ENUM {CRC32,SHA256,RS} alg=SHA256)
consistent-hash使
replicas1consistent-hash
alg使
shard使
commit

BACKEND shard.backend(ENUM {HASH,URL,KEY,BLOB} by=HASH, INT key=0, BLOB key_blob=0, INT alt=0, REAL warmup=-1, BOOL rampup=1, ENUM {CHOSEN,IGNORE,ALL} healthy=CHOSEN)


by=HASH()
vcl_hashhash使

by=URL
req.url/bereq.url使

by=KEY
keyINT使
cookie

shard.backend(by=KEY,key=shard.key(bereq.http.cookie))




by=BLOB
key_blobblob使
使
blob使4使

key
by=key

key_blob
by=blob使blob

alt
n
使(restart/retries使
alt=req.restart or bereq.retries/
rampup/warmupalt01alt

warmup
(-1,0~1)
-1set_warmup使

rampup
rampup使

healthy=CHOSEN()
healthy
alt0healthyn
then return the next healthy backend.althealthy

healthy=IGNORE
helathy
althealthy
rampup/warmup

healthy=ALL
CHOSENalt0


healthy alt==0 alt>0
CHOSEN ヘルスチェックを行う ヘルスチェックを行わない
IGNORE ヘルスチェックを行わない ヘルスチェックを行わない
ALL ヘルスチェックを行う ヘルスチェックを行う

 

INT shard.key(STRING string, ENUM {CRC32,SHA256,RS} alg=SHA256)
INT
SHA256256bit32bit
shard.backendby=KEY使使

使


sub vcl_init {
  new ws_cache = directors.shard();
  ws_cache.set_rampup(10s);
  ws_cache.add_backend(ws01);
  ws_cache.add_backend(ws02);
  ws_cache.add_backend(ws03);
  ws_cache.add_backend(ws04);
  ws_cache.reconfigure();
}

sub vcl_recv {
  //x-varnish-hashにはユーザIDとかそんなのが入ってる
  ...
  set req.backend_hint = ws_cache.backend(by=KEY, key=ws_cache.key(req.url + ":" + req.http.host + ":" + req.http.x-varnish-hash));
}


 

INT,BACKENDBOOL




sub vcl_recv {
  if(1){
    set resp.http.foo = "1";
  }
  if(req.backend_hint){
    set resp.http.bar = "1";
  }
}


INT0false
BACKENDfalse

VCL使label


VCLinclude
includeVCL
labelVCL
使

VCLload

varnishadm vcl.load a_example_net /etc/varnish/a_examplet_net.vcl
varnishadm vcl.load b_example_net /etc/varnish/b_examplet_net.vcl
varnishadm vcl.load c_example_net /etc/varnish/c_examplet_net.vcl

varnishadm vcl.label vcl_a_example_net a_example_net
varnishadm vcl.label vcl_b_example_net b_example_net
varnishadm vcl.label vcl_c_example_net c_example_net


default.vcl

vcl 4.0;
//最低限1つはバックエンドがないと動かないのでダミー
backend default { .host = "127.0.0.1"; }

sub vcl_recv {
  if(req.http.host ~ "a\.example\.net$"){
    return(vcl(vcl_a_example_net));
  }
  if(req.http.host ~ "b\.example\.net$"){
    return(vcl(vcl_b_example_net));
  }
  if(req.http.host ~ "c\.example\.net$"){
    return(vcl(vcl_c_example_net));
  }
  return (synth(403));
}


a.example.netvcl_a_example_netVCL(/etc/varnish/a_examplet_net.vcl)使
使CDNMWVarnish使VCL
VCL使
vcl_a_example_netb.examplet.net
req.http.host

vcl.loadvcl.label

varnishadm vcl.load a_example_net_1 /etc/varnish/a_examplet_net.vcl
varnishadm vcl.label vcl_a_example_net a_example_net_1


vcl.list

# varnishadm vcl.list
active      auto/warm          0 boot
available   auto/warm          0 a_example_net
available  label/warm          0 vcl_a_example_net -> a_example_net_1
available   auto/warm          6 a_example_net_1 (1 label)


vcl_a_example_neta_example_net_1
reloadVCLreload

PROXY


.proxy_header=[];proxy使

backend ws01 {.host = "192.168.1.1";.proxy_header = 1;}//PROXYプロトコルv1
backend ws02 {.host = "192.168.1.2";.proxy_header = 2;}//PROXYプロトコルv2

varnishtest


HTTP/2proxy
resp.msgresp.reason
semabarrier

std.integerREAL


1.21E+2
INT

std.logvcl_init


 

varnishhisty


 

varnishstat-fglob(7)使


 

POST


VCLPOST
bodyGETbody使POST
GETbodybuiltin.vcl

sub vcl_backend_fetch {
    if (bereq.method == "GET") {
        unset bereq.body;
    }
    return (fetch);
}


VCLbuiltin.vclreturn)unset bereq.body
POSTbuiltin.vclpass
hashbody
bodyvmod使


statsn_waitinglist

VCL
vcl使
[A-Za-z][A-Za-z0-9_-]*

VCL使
backendacl


pass(uncacheable)cache-control/ExpiresTTL
passTTL

-T/-Snone


ban_lurker_holdoff
banban.listban_lurkerlookupsleep
0.1sban_lurker_sleep

session_max
session_max
4.0


vcl_dir->vcl_path / vmod_dir->vmod_path
使_dir
:path
vcl_path/etc/varnish:/usr/share/varnish/vcl
includeVCL/etc/varnish
/usr/share/varnish/vcl
/usr/share/varnish/vcldevicedetect.vcl

builtin.vcl


GETbereq.bodyunset
POST

hit-for-pass
return(pass)hit-for-pass(HFP)
2passHFP
4.15.0diff

 sub vcl_backend_response {
-    if (beresp.ttl <= 0s ||
+    if (bereq.uncacheable) {
+        return (deliver);
+    } else if (beresp.ttl <= 0s ||
       beresp.http.Set-Cookie ||
       beresp.http.Surrogate-control ~ "no-store" ||
       (!beresp.http.Surrogate-Control &&
         beresp.http.Cache-Control ~ "no-cache|no-store|private") ||
       beresp.http.Vary == "*") {
-        /*
-        * Mark as "Hit-For-Pass" for the next 2 minutes
-        */
+        # Mark as "Hit-For-Pass" for the next 2 minutes
         set beresp.ttl = 120s;
         set beresp.uncacheable = true;
     }

VCL


backends.proxy_header


vcl_recv:returnvcl(label)


vcl_recv:rollback

vcl_hit:return(fetch)
4.1fetch->miss

vcl_backend_*:remote.ip, client.ip, local.ip, server.ip

vcl_backend_fetch:bereq.body


vcl_backend_error:synthetic()beresp.body


    set beresp.body = {"<!DOCTYPE html>
<html>
  <head>
    <title>"} + beresp.status + " " + beresp.reason + {"</title>
  </head>
  <body>
    <h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>
    <p>"} + beresp.reason + {"</p>
    <h3>Guru Meditation:</h3>
    <p>XID: "} + bereq.xid + {"</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>
"};


vcl_synth:synthetic()resp.body

vcl_deliver: obj.ttl, obj.age, obj.grace, obj.keep(read)

matchBOOL

set req.http.foo = req.http.bar ~ "bar";








Varnish

libs

libs-dev

docs


4


Varnish+lib+docs

dev


2


VMODABI
VMODABI使VMOD

reload
systemd
VCL()reload
Xenial(systemd)LOGNAMEuuid
uuidreloadVCL
reload_

-    vcl_label="${LOGNAME}${LOGNAME:+_}${uuid}"
+    vcl_label="reload_${LOGNAME}${LOGNAME:+_}${uuid}"



RHEL/CentOSRHEL(:使)