云原生基础

[Plan 需求]      Jira / Linear / 飞书项目        ← 工程师拆卡，写 AC
    ↓
[Code 编码]      Git + GitHub/GitLab             ← feature 分支，commit message 关联 issue
    ↓
[Build 构建]     Docker buildx + BuildKit        ← 多阶段镜像构建，推 Harbor
    ↓
[Test 测试]      pytest / go test + SonarQube    ← 单测+覆盖率+静态扫描
    ↓
[Scan 安全]      Trivy + Grype                   ← 扫镜像 CVE，扫 IaC 配置
    ↓
[Deploy 部署]    ArgoCD GitOps                   ← 监听 manifest repo，自动 sync
    ↓
[Verify 验证]    Prometheus + Blackbox Exporter  ← 部署后 smoke test + SLO 检查
    ↓
[Monitor 监控]   Grafana + Loki + Tempo          ← 指标/日志/链路三件套
    ↓
[Feedback 反馈]  告警 → On-call → 复盘 → Jira    ← 失败案例回灌成新的需求

# .gitlab-ci.yml —— 应用代码 repo 里的配置
stages:
  - build
  - test
  - publish
  - deploy

variables:
  IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA  # 用 commit 短 SHA 做镜像 tag，可追溯

# 构建阶段：用 BuildKit 跑多阶段构建
build:
  stage: build
  image: docker:24
  services:
    - docker:24-dind
  script:
    - docker buildx build -t $IMAGE --push .
  only:
    - main

# 测试阶段：Go 单测 + 覆盖率
test:
  stage: test
  image: golang:1.22
  script:
    - go test -race -coverprofile=coverage.out ./...
    - go tool cover -func=coverage.out
  only:
    - main

# 部署阶段：更新 manifest repo 里的镜像 tag，ArgoCD 监听到变化自动 sync
deploy:
  stage: deploy
  image: alpine/git
  script:
    - git clone https://gitlab-ci-token:$CI_JOB_TOKEN@gitlab.com/team/manifests.git
    - cd manifests
    - sed -i "s|image:.*|image: $IMAGE|" apps/myapp/deployment.yaml
    - git commit -am "chore(myapp): bump to $CI_COMMIT_SHORT_SHA"
    - git push
  only:
    - main

# sloth.yaml —— Sloth 的 SLO 声明
version: "prometheus/v1"
service: "api-gateway"
labels:
  owner: "team-platform"
slos:
  - name: "requests-availability"
    objective: 99.9                    # 99.9% 成功率
    description: "API 网关 5xx 比例 SLO"
    sli:
      events:
        error_query: sum(rate(http_requests_total{job="api-gateway",code=~"5.."}[{{.window}}]))
        total_query: sum(rate(http_requests_total{job="api-gateway"}[{{.window}}]))
    alerting:
      name: ApiGatewayAvailabilityBurnRate
      labels:
        category: availability
      annotations:
        summary: "API 网关可用性错误预算消耗过快"
      page_alert:
        labels:
          severity: pageteam
      ticket_alert:
        labels:
          severity: slack

sloth generate -i sloth.yaml -o prometheus-slo-rules.yaml

# prometheus-slo-rules.yaml —— 纯手写版（精简）
groups:
  - name: slo-requests-availability
    rules:
      # 5 分钟窗口的 SLO 完成度
      - record: slo:sli_error:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{job="api-gateway",code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="api-gateway"}[5m]))          
      # 错误预算消耗速率
      - record: slo:error_budget:burn_rate5m
        expr: slo:sli_error:ratio_rate5m / (1 - 0.999)
      # 告警：5 分钟 burn rate > 14.4 倍（约 2% 预算/小时）
      - alert: SLOBurnRateHigh
        expr: slo:error_budget:burn_rate5m > 14.4
        for: 2m
        labels:
          severity: pageteam
        annotations:
          summary: "SLO 错误预算消耗速率过高"

# certificate.yaml —— Let's Encrypt 自动签发
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-gateway-tls
  namespace: prod
spec:
  secretName: api-gateway-tls        # 证书会写到这个 Secret 里
  duration: 2160h                    # 90 天
  renewBefore: 360h                  # 提前 15 天续期，留足失败重试时间
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - api.example.com

你是一名资深 SRE，擅长编写 Prometheus PromQL 查询语句。

# 可用指标清单（严格遵守，禁止臆造指标名）
- http_requests_total{job, method, code, path}
- http_request_duration_seconds_bucket{job, le}
- container_cpu_usage_seconds_total{pod, namespace}
- container_memory_working_set_bytes{pod, namespace}
- kube_pod_status_phase{namespace, phase}
- node_cpu_seconds_total{node, mode}

# 用户问题
{user_question}

# 输出要求
1. 只输出一条 PromQL 语句，不要解释
2. 涉及速率统一用 rate()，区间用 [5m]
3. 直方图分位数用 histogram_quantile(0.95, sum by (le)(rate(...[5m])))
4. 如果用户问题涉及不存在的指标，直接回复 "UNSUPPORTED"

                    ┌─────────────────────────────┐
   运维同学          │  前端：飞书机器人 / Web UI   │
   "订单服务 500"     └──────────────┬──────────────┘
                                     │ 自然语言问题
                                     ▼
                    ┌─────────────────────────────┐
                    │  Query Rewriting            │
                    │  把口语改成检索友好的关键词    │
                    └──────────────┬──────────────┘
                                     │ 改写后的 query
              ┌──────────────────────┼──────────────────────┐
              ▼                      ▼                      ▼
   ┌──────────────────┐  ┌────────────────────┐  ┌─────────────────────┐
   │ Embedding Model  │  │ 向量库 Milvus       │  │ 关键词库 ES          │
   │ bge-large-zh     │  │ Top-K 语义召回       │  │ BM25 召回            │
   └──────────────────┘  └────────────────────┘  └─────────────────────┘
              │                      │                      │
              └──────────────────────┼──────────────────────┘
                                     ▼
                       ┌──────────────────────────┐
                       │ Reranker（bge-reranker）│
                       │ 精排，取 Top 3           │
                       └──────────────┬───────────┘
                                      ▼
                       ┌──────────────────────────┐
                       │ LLM 生成回答              │
                       │ 注入：召回文档 + 告警上下文 │
                       └──────────────┬───────────┘
                                      │
                                      ▼
                       ┌──────────────────────────┐
                       │ 输出：根因分析 + 处置步骤   │
                       │ + 引用来源链接            │
                       └──────────────────────────┘

# main.tf —— AWS EKS 最小可用集群
terraform {
  required_version = ">= 1.5"

  # 远端 backend：state 存到 S3，加 DynamoDB 锁
  # 这里有个坑：bucket 和 table 必须先手动建好，terraform 自己管不了 backend 资源
  backend "s3" {
    bucket         = "my-tf-state-prod"
    key            = "eks/prod/terraform.tfstate"
    region         = "ap-northeast-1"
    dynamodb_table = "tf-locks"
    encrypt        = true
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "ap-northeast-1"
}

# VPC：用官方 module，开内网 + 公网子网
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "prod-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["ap-northeast-1a", "ap-northeast-1c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true  # 省钱：只开一个 NAT，生产里要 HA 就别用 single
  enable_dns_hostnames = true  # EKS 必须开，否则 CoreDNS 解析有问题
}

# EKS 集群
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = "prod-cluster"
  cluster_version = "1.30"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  enable_irsa = true  # 启用 IRSA，让 Pod 用 IAM Role 而不是 node IAM

  eks_managed_node_groups = {
    general = {
      instance_types = ["t3.large"]
      desired_size   = 2
      min_size       = 1
      max_size       = 4
      disk_size      = 50
    }
  }

  # 坑：cluster_endpoint_public_access 默认是 false，kubectl 连不上
  cluster_endpoint_public_access = true
}

output "cluster_endpoint" {
  value = module.eks.cluster_endpoint
}

terraform init      # 拉 provider + 初始化 backend
terraform plan      # 先 dry-run 看变更，重点 review destroy 那一栏
terraform apply     # 确认无误再 apply

# .gitignore
*.tfstate
*.tfstate.*
crash.log
.terraform/
.terraform.lock.hcl
*.tfvars      # 含密钥的 tfvars 也不进 git，用 *.tfvars.example 做模板

# 检测漂移
terraform plan -detailed-exitcode    # exit code 2 = 有漂移

# 同步真实状态（不修改云资源，只改 state）
terraform refresh                    # 老命令，新版已废弃
terraform apply -refresh-only        # 新写法，只刷新 state

# 实在拉不回来，手动改 state
terraform state list                 # 看 state 里有哪些资源
terraform state rm aws_xxx.foo       # 从 state 里移除（资源留在云上）
terraform import aws_xxx.foo id      # 把云上已有资源导入 state

terraform force-unlock <LOCK_ID>     # 强制释放锁

# composition.yaml —— 平台团队定义：标准 S3 桶
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: standard-bucket
  labels:
    provider: aws
spec:
  compositeTypeRef:
    apiVersion: platform.example.com/v1alpha1
    kind: XBucket
  resources:
    - name: bucket
      base:
        apiVersion: s3.aws.crossplane.io/v1beta1
        kind: Bucket
        spec:
          forProvider:
            region: ap-northeast-1
            acl: private
            versioningConfiguration:
              status: Enabled        # 强制开版本控制，防误删
            lifecycleConfiguration:
              rules:
                - id: expire-logs
                  status: Enabled
                  expirationInDays: 30   # 30 天后自动清理
      patches:
        - fromFieldPath: "spec.bucketName"
          toFieldPath: "metadata.annotations[crossplane.io/external-name]"
        - fromFieldPath: "spec.region"
          toFieldPath: "spec.forProvider.region"

# claim.yaml —— 业务团队申请一个桶
apiVersion: platform.example.com/v1alpha1
kind: BucketClaim
metadata:
  name: order-service-logs
  namespace: prod
spec:
  bucketName: order-service-logs-prod-2025
  region: ap-northeast-1

kubectl apply -f claim.yaml
kubectl get bucketclaim order-service-logs -n prod -w