⛳️ 💖 👩🏿‍🔧 كيف قمنا في Dropbox بالتبديل من Nginx إلى Envoy 💃 🙂 🚶🏼

في هذه المقالة ، سنتحدث عن بنيتنا التحتية القديمة المستندة إلى Nginx ، وتقرحاتها ، والفوائد التي حصلنا عليها بعد الانتقال إلى Envoy . سنقارن بين Nginx و Envoy بطرق مختلفة. سنتطرق أيضًا بإيجاز إلى عملية الترحيل ، والوضع الحالي ، بالإضافة إلى المشكلات التي تمت مواجهتها أثناء الانتقال.

عندما قمنا بتحويل معظم حركة المرور إلى Envoy ، تمكنا من ترحيل نظام يعالج عشرات الملايين من الاتصالات المفتوحة ، وملايين الطلبات في الثانية ، وعرض النطاق الترددي الذي يبلغ تيرابايت بسلاسة. في الواقع ، لقد أصبحنا أحد أكبر مستخدمي Envoy في العالم.

إخلاء المسؤولية: نحاول أن نظل موضوعيين ، فالكثير من المقارنات تنطبق فقط على Dropbox ومبادئ تطوير البرمجيات لدينا: نحن نراهن على Bazel و gRPC و C ++ و Golang.

, Nginx , .

, Nginx

Nginx , Python2, Jinja YAML. . , upstream , Lua. , Go. , Nginx.

Nginx . :

() API REST gRPC, .
Protocol buffers de facto
, , Bazel.
.

Nginx :

YAML, Jinja2 Python.
Lua, ,
, .
, : syslog, logrotate , , .

Nginx.

Bandaid?

, Bandaid, -, Go. Dropbox, Go: , , .. Nginx, , :

Golang , C++. Edge, " " .
- (GC), HTTP TLS, , BoringSSL, Nginx\Envoy.
- "goroutine-per-request" GC , .
FIPS Golang TLS
Bandaid Dropbox, , .

Envoy.

, Envoy

, , , Envoy , Nginx Envoy.

Nginx . SO_REUSEPORT, EPOLLEXCLUSIVE, . , , , , , , ( aio, aio_write, thread pools). , .

Envoy , . SO_REUSEPORT ( BPF) libevent ( — epoll(2), EPOLLEXCLUSIVE). Envoy - -. , .

, Nginx Envoy . — , Nginx Envoy. , , , .

: RPS, , gRPC . . Nginx , . Envoy , envoy-perf, , , . "hulk", "" .

Nginx . -, SO_REUSEPORT, .
Nginx Envoy, Lua Nginx RPS . , lua_shared_dict, mutex. , . - counter(9) FreeBSD, : , , . , Nginx ( , ), , .

Envoy , 60% , Nginx.

, , . , , .

Nginx "stub status", :

Active connections: 291
server accepts handled requests
 16630948 16630948 31070465
Reading: 6 Writing: 179 Waiting: 106

, log_by_lua, , , Lua: , , .. , :

function _M.cache_hit_stats(stat)
    if _var.upstream_cache_status then
        if _var.upstream_cache_status == "HIT" then
            stat:add("upstream_cache_hit")
        else
            stat:add("upstream_cache_miss")
        end
    end
end

, , error.log, upstream, http, Lua TLS.

Nginx: , , RSS\VMS, TLS .

Envoy ( Prometheus), , :

$ curl -s http://localhost:3990/stats/prometheus | wc -l
14819

\ upstream\ , .
: TCP\HTTP\TLS
, , .

Envoy. /certs, /clusters config_dump, :

/logging, .
/cpuprofiler, /heapprofiler, /contention,
/runtime_modify, , ..

Envoy . , , , Edge . Nginx OpenTracing, .

Envoy gRPC, syslog-to-hive (, .. Envoy syslog . ). ( !) gRPC, TCP\UDP.

Envoy, , gRP: Access Log Service (ALS). Envoy data plane , .

Nginx "". . ( , , \ ..) ( syslog, HTTP). , Nginx shell-. .

Envoy , data plane control plane, , , . protobuf gRPC, API, xDS. Envoy ( ) xDS. Envoy, UDPA (universal data plane interface) : "de facto" L4\L7. — . ORCA (Open Request Cost Agregation) UDPA , Envoy, Katran, eBPF\XDP L4.

Dropbox, API gRPC. xDS control plane, Envoy , , . Dropbox RPC — , , , , gRPC.

xDS, Nginx, , :

Access Log Service (ALS), , , . log_format access_log Nginx
Endpoint discovery service (EDS), . upstream server ( Lua balancer_by_lua_block) Nginx. .
Secret discovery service (SDS), TLS, Nginx ssl_* ( ssl_*_by_lua_block ). .
Runtime discovery service (RTDS), runtime . Nginx , Lua. . Envoy -, RTDS API , (, , sysctl), .
Route discovery service (RDS): . Nginx location c set_header```proxy_set_header\proxy_pass```. .

Envoy . control plane Envoy, Istio go-control-plane. Envoy API xDS. gRPC . Golang Envoy API xDS. , , cron\logrotate\syslog\ ..

Nginx . , . — Python2, Jinja2 YAML. , erb, pug, Text::Template m4 ( ! . ):

{% for server in servers %}
server {
    {% for error_page in server.error_pages %}
    error_page {{ error_page.statuses|join(' ') }} {{ error_page.file }};
    {% endfor %}
    ...
    {% for route in service.routes %}
    {% if route.regex or route.prefix or route.exact_path %}
    location {% if route.regex %}~ {{route.regex}}{%
            elif route.exact_path %}= {{ route.exact_path }}{%
            else %}{{ route.prefix }}{% endif %} {
        {% if route.brotli_level %}
        brotli on;
        brotli_comp_level {{ route.brotli_level }};
        {% endif %}
        ...

Nginx : \ . YAML anchors, Jinja2 — , , Python . . , :

. — .
C. , , . nginx -t.

Envoy : Protocol Buffers. , . , , protobuf , \ — .

Envoy protobuf Python3. proto, Python. :

from dropbox.proto.envoy.extensions.filters.http.gzip.v3.gzip_pb2 import Gzip
from dropbox.proto.envoy.extensions.filters.http.compressor.v3.compressor_pb2 import Compressor

def default_gzip_config(
    compression_level: Gzip.CompressionLevel.Enum = Gzip.CompressionLevel.DEFAULT,
    ) -> Gzip:
        return Gzip(
            # Envoy's default is 6 (Z_DEFAULT_COMPRESSION).
            compression_level=compression_level,
            # Envoy's default is 4k (12 bits). Nginx uses 32k (MAX_WBITS, 15 bits).
            window_bits=UInt32Value(value=12),
            # Envoy's default is 5. Nginx uses 8 (MAX_MEM_LEVEL - 1).
            memory_level=UInt32Value(value=5),
            compressor=Compressor(
                content_length=UInt32Value(value=1024),
                remove_accept_encoding_header=True,
                content_type=default_compressible_mime_types(),
            ),
        )

Python3 ! mypy-protobuf . IDE . , protobuf . window_bits Gzip 9 15. protoc-gen-validate:

google.protobuf.UInt32Value window_bits = 9 [(validate.rules).uint32 = {lte: 15 gte: 9}];

— , :

// Value from 1 to 9 that controls the amount of internal memory used by zlib. Higher values.
// use more memory, but are faster and produce better compression results. The default value is 5.
google.protobuf.UInt32Value memory_level = 1 [(validate.rules).uint32 = {lte: 9 gte: 1}];

, protobuf , , , Harvey Tuch, Envoy.

Nginx - , , C. . , . Nginx. , , : hash-, , - , ( RAII), HTTP. , pcre, zlib, openssl libc.

Nginx Perl Javascript. , .

lua-nginx-module OpenResty. . log_by_lua balancer_by_lua backend.

Nginx C++, , . - . , .

Envoy — C++. Nginx, . - , :

. — .
C++14. , -, . C++14 Golang — Python. (! . )
C++14 . abseil, C++, mutex , , \ , .

Envoy Vortex2 ( framework ) 200 stats.

Envoy Lua moonjit, LuaJIT c Lua 5.2. Lua Nginx , Lua Envoy - , . , Lua, , Lua C++ Envoy.

Envoy , WebAssembly (WASM) — , . WASM . . Envoy WebAssembly for Proxies ( SDK C++ Rust), WASM L4\L7. , WASM . , proxy-wasm xDS, A\B . Kubecon'19 ( , ?) WASM Envoy . 60-70% C++.

WASM . , , proxy-wasm ABI. , WebAssembly. , C++.

Istio WebAssembly, WebAssemblyHub . .

Dropbox WebAssembly, , proxy-wasm SDK Go.

Nginx shell-, make. , Bazel , , . Google Nginx Bazel, Nginx, BoringSSL, PCRE, ZLIB Brotli.

, Nginx unit-.

Lua Python:

class ProtocolCountersTest(NginxTestCase):
    @classmethod
    def setUpClass(cls):
        super(ProtocolCountersTest, cls).setUpClass()
        cls.nginx_a = cls.add_nginx(
            nginx_CONFIG_PATH, endpoint=["in"], upstream=["out"],
        )
        cls.start_nginxes()

    @assert_delta(lambda d: d == 0, get_stat("request_protocol_http2"))
    @assert_delta(lambda d: d == 1, get_stat("request_protocol_http1"))
    def test_http(self):
        r = requests.get(self.nginx_a.endpoint["in"].url("/"))
        assert r.status_code == requests.codes.ok

( ip- 127.0.0.1/8, ..) nginx -c.

Envoy, Bazel, : Bazel . copybara protobuf Envoy, UDPA. , .

Envoy unit- ( gtest\gmock) , , . , .

Envoy 100% unit-. CI Azure .

google\benchmark:

$ bazel run --compilation_mode=opt test/common/upstream:load_balancer_benchmark -- --benchmark_filter=".*LeastRequestLoadBalancerChooseHost.*"
BM_LeastRequestLoadBalancerChooseHost/100/1/1000000          848 ms          449 ms            2 mean_hits=10k relative_stddev_hits=0.0102051 stddev_hits=102.051
...

Envoy unit- :

TEST_F(CourierClientIdFilterTest, IdentityParsing) {
  struct TestCase {
    std::vector<std::string> uris;
    Identity expected;
  };
  std::vector<TestCase> tests = {
    {{"spiffe://prod.dropbox.com/service/foo"}, {"spiffe://prod.dropbox.com/service/foo", "foo"}},
    {{"spiffe://prod.dropbox.com/user/boo"}, {"spiffe://prod.dropbox.com/user/boo", "user.boo"}},
    {{"spiffe://prod.dropbox.com/host/strange"}, {"spiffe://prod.dropbox.com/host/strange", "host.strange"}},
    {{"spiffe://corp.dropbox.com/user/bad-prefix"}, {"", ""}},
  };
  for (auto& test : tests) {
    EXPECT_CALL(*ssl_, uriSanPeerCertificate()).WillOnce(testing::Return(test.uris));
    EXPECT_EQ(GetIdentity(ssl_), test.expected);
  }
}

. . unit- , Envoy.

Bazel — , - . , : , , / ..

Bazel , . . , , , ..

Nginx , . : zlib ( ), - TLS PCRE. Nginx , . , libc.

Nginx , OpenBSD. OpenBSD httpd. BSDCon.

, Nginx 30 11 .

Envoy , , C++ , C, Nginx. . , , . .

Envoy . AddressSanitizer, ThreadSanitizer MemorySanitizer. fuzzing.

, IT, OSS-Fuzz, fuzzing. .

, , . 22 .

Envoy , . Envoy Google's Vulnerability Reward Program (VRP). Google , , .

, , CVE-2019–18801

Ubuntu Debian, hardened , Edge. ASLR, :

build:hardened --force_pic
build:hardened --copt=-fstack-clash-protection
build:hardened --copt=-fstack-protector-strong
build:hardened --linkopt=-Wl,-z,relro,-z,now

fork, Nginx, , - , — 1000 . Envoy, , .

, . BoringSSL FIPS, . ASAN Edge.

, .

Nginx , . : , ( ), .

Nginx , . HTTP/2 , gRPC , . gRPC. " " , . , , , .

Envoy ingress\egress , gRPC . : , , . Nginx, Envoy upstream .

Envoy , Envoy S3 . eCache, HTTP Envoy.

Envoy , gRPC:

gRPC, , gRPC ( Dropbox )
HTTP/2 , TCP , .
gRPC->HTTP ( ), HTTP/1 gRPC.
gRPC->WEB, gRPC , (firewall, IDS ) HTTP/2.
gRPC JSON transcoder, , API Dropbox, REST gRPC.

Envoy , :

Egress , Envoy HTTP CONNECT — Squid . Squid Envoy, , data plane ( ).
: Courier gRPC Envoy service mesh. Envoy , . Envoy . Hadoop . Superset airflow, presto hive. Grafan MySQL.

Nginx , . , bug tracker. #nginx IRC FreeNode, .

Envoy : \ GitHub, ( Zoom , . ). Slack, .

, HTTP/3.

QUIC HTTP/3 Nginx F5. , . , Cloudflare . Nginx ( — , ! . )

Envoy , quiche. , , , , " ".

, . , Envoy, , gRPC .

Nginx Envoy , DNS. Envoy :

Ingress . Dropbox gRPC Envoy. Envoy Edge.
Ingress RPS. , gRPC, - .
. , HTTP ( ). gRPC long-polling.
RPS. API ( ). API gRPC. API REST Edge.
Egress, . — AWS, S3. Squid , L4\L7 data plane.

, , www.dropbox.com. Edge Nginx. .

,

. - . API. Dropbox API, curl\wget HTTP/1.0 HTTP. Nginx "de facto", , . Nginx Envoy, API, Envoy . , .

/ RF :

URL'. — , Nginx , Envoy . , , merge_slashes.
. Nginx Host : example.com example.com:port. , . ( ), Envoy: strip_matching_host_port.
. API - Transfer-Encoding: Chunked ( C). , RFC7230 , Transfer-Encoding/TE . .
, Content-Length Transfer-Encoding: chunked. Nginx, Envoy. RFC7230 , , , "". , Content-Length . http-parse, , Envoy.

, :

circuit-breaking. Envoy , HTTP/1 HTTP/2, circuit breakers . , , Envoy mesh .
. Nginx , , HTTP/1.0, chunked. Nginx Content-Length, . Envoy Buffer, , , .

Envoy — . , .

?

HTTP/3. . Envoy . Linux UDP Edge.
xDS . Load Reporting service (LRS) Endpoint discovery service (EDS) Envoy, gRPC.
Envoy WASM. Golang proxy-wasm SDK — Envoy Go, Golang.
Bandaid. Dropbox data plane — . , Bandaid ( ) Envoy. , .
Envoy mobile. , Envoy . (HTTP/3, gRPC, TLS 1.3 ..) .

. Traffic Runtime, : Agata Cieplik, Jeffrey Gensler, Konstantin Belyalov, Louis Opter, Naphat Sanguansin, Nikita V. Shirokov, Utsav Shah, Yi-Shu Tai, Envoy, .

Runtime Ruslan Nigmatullin, Envoy, Envoy MVP, .

كيف قمنا في Dropbox بالتبديل من Nginx إلى Envoy

, Nginx

Bandaid?

, Envoy

,

?

More articles: