AHdark Blog

The overseas transit links of Source Global CDN often require reverse proxying for multiple domains. The previous solution was to create a separate vhost for each domain. However, in the reality of growing links and increasing servers, this solution has become increasingly difficult to maintain.

As the main operations personnel of Source Global CDN, I naturally took on this task.

Problem#

With the increase in servers used for services and the growth of links, we face the following issues:

1. When configuration changes occur, content modifications are needed for all servers, which is time-consuming and labor-intensive.
2. If new services are added, multiple operations need to be performed on all servers, and relevant testing is required.
3. Management of SSL certificates and vhost conf files is chaotic.

After careful consideration, we roughly came up with the following solutions:

1. Use wildcard domain resolution to improve cohesion, allowing all servers to automatically cooperate with addition, deletion, and modification operations without separate configuration.
2. Use a private DNS service to manage links on a single platform without needing modifications on the servers.

The highly cohesive wildcard domain resolution solution significantly reduces the number of vhost files and SSL certificates, while the private DNS avoids the issue of unmodified data within the servers.

Implementation#

Wildcard Domain Resolution#

After consideration, we formulated the following plan:

1. A SaaS task is responsible for using acme.sh to operate DNS and apply for wildcard SSL certificates.
2. Store the certificates in a bucket and update the Content-MD5 meta through another SaaS task.
3. The server's Crontab automatically retrieves MD5 from the bucket every hour and compares it. If there are changes, it updates the local SSL and reloads Nginx.

The acme.sh operation can refer to Using acme.sh to Automatically Configure Certificates for IP/Domain with slight modifications.

Nginx Dynamic Reverse Proxy#

After completing the above simple tasks, we began to consider Nginx's dynamic reverse proxy.

The reason for not considering the use of Lua scripts is that the time cost of reinstalling OpenResty on all servers is too great.

Most people are likely already familiar with Nginx's reverse proxy, which typically presents the following pattern:

upstream backend {
  server 127.0.0.1:8080 weight=1 fail_timeout=5s max_fails=3;
  server 127.0.0.1:8081 weight=1 fail_timeout=5s max_fails=3;
  server 127.0.0.1:8082 weight=1 fail_timeout=5s max_fails=3 backup;
}

server {
  listen       80;
  server_name  www.example.com;
  index  index.html index.htm index.php;

  location / {
    proxy_pass http://backend;
  }
}

In this pattern, the reverse proxy always fetches resources from 127.0.0.1:8080, which is static.

Our requirements are as follows:

• When $host = gh.sourcegcdn.com, reverse proxy to https://gh.origin.sgcdn
• When $host = wp.sourcegcdn.com, reverse proxy to https://wp.origin.sgcdn
• When $host = <subdomain>.sourcegcdn.com, reverse proxy to https://<subdomain>.origin.sgcdn

That is, seek the prefix based on the subdomain of the request header and complete the suffix before performing the reverse proxy.

Our final implementation is as follows:

server {
  listen 80;
  listen [::]:80;
  listen 443 ssl http2;
  listen [::]:443 ssl http2;

  resolver 8.8.8.8 valid=10s;

  ssl_certificate /usr/local/nginx/conf/ssl/_.sourcegcdn.com.crt;
  ssl_certificate_key /usr/local/nginx/conf/ssl/_.sourcegcdn.com.key;
  ssl_protocols TLSv1.1 TLSv1.2 TLSv1.3;
  ssl_ecdh_curve X25519:prime256v1:secp384r1:secp521r1;
  ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256;
  ssl_conf_command Ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256;
  ssl_conf_command Options PrioritizeChaCha;
  ssl_prefer_server_ciphers on;
  ssl_session_timeout 10m;
  ssl_session_cache shared:SSL:10m;
  ssl_buffer_size 2k;
  add_header Strict-Transport-Security max-age=15768000;
  ssl_stapling on;
  ssl_stapling_verify on;

  server_name ~^(?<subdomain>.+)\.sourcegcdn\.com$;
  access_log /data/wwwlogs/_.sourcegcdn.com.nginx.log combined;
  error_log /data/wwwlogs/_.sourcegcdn.com.nginx.error.log;

  add_header X-Powered-By "Source Global CDN Global Accurate";

  index index.html;
  root /data/wwwroot/*.sourcegcdn.com;

  #error_page 404 /404.html;
  #error_page 502 /502.html;

  add_header "X-Subdomain-Deliver" $subdomain;

  location @proxy {
    # Back source HOST
    proxy_set_header  Host              $subdomain.sourcegcdn.com;
    proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
    # Configure backend server
    proxy_pass https://$subdomain.origin.sgcdn;
    proxy_redirect off;
  }

  location / {
    try_files $uri @proxy;
  }

  location ~ .*\.(gif|jpg|jpeg|png|bmp|swf|flv|mp4|ico)$ {
    expires 365d;
    access_log off;
    try_files $uri @proxy;
  }
  location ~ .*\.(js|css)?$ {
    expires 30d;
    access_log off;
    try_files $uri @proxy;
  }

  location ~ /(\.user\.ini|\.ht|\.git|\.svn|\.project|LICENSE|README\.md) {
    deny all;
  }
  location /.well-known {
    allow all;
  }
}

DNS Resolution#

With this configuration, we only need to set up the CDN to configure the unified node *.sourcegcdn.com, and then add DNS resolution as needed to use it normally.

*.sourcegcdn.com 1 IN A 127.0.0.1

References#

• Nginx Dynamic Reverse Proxy Based on Subdomain Exposing Intranet HTTP Service - iVampireSP's Story
• https://nginx.org/en/docs/http/ngx_http_proxy_module.html
• http://nginx.org/en/docs/http/server_names.html