Lead Software Engineer

Company Overview

What You'll Do

Operating a reliable global service requires robust observability and automation. The Observability team builds the real‑time telemetry and AI/ML capabilities that power how engineers measure, visualize, investigate, and improve customer experience—at planetary scale.

This position is an individual contributor role reporting to the Engineering Manger for Observability.

• Own end‑to‑end pipelines—from ingestion and feature engineering to training and real‑time/batch inference—for operational time‑series use cases (metrics, logs, events)

• Design and operate anomaly‑detection & forecasting services (statistical/ML) that improve SLAs, reduce alert fatigue, and accelerate incident triage

• Build internal SDKs, metadata catalogs, and reusable ingestion frameworks that standardize access, enforce governance, and accelerate adoption across product and platform teams

• Harden data quality (sanity scoring, validation, drift checks), backfills, and replay strategies; define SLOs for data and models

• Partner with Applied Scientists to productionize models with pragmatic algorithms (e.g., tree‑based methods, classical TS) and selectively introduce deep learning where it pays off

• Implement CI/CD for data & models (tests, canaries/shadowing, monitoring/evals, safe rollback)

• Optimize time‑series storage and compute (e.g., ClickHouse, Postgres/TimescaleDB, columnar stores): partitioning, rollups, retention, and cost controls

• Integrate with the observability stack (OTel for signals; dashboards/alerts) and collaborate with SRE/infra to ensure multi‑tenant performance and resilience

• Explore LLM‑assisted on‑call (RAG over logs/runbooks) to improve diagnosis and guidance; manage prompt safety, evals, and latency budgets

Job Designation

What You Bring

• 12+ years of software engineering, with deep Python and SQL in production
• Hands‑on building real‑time and batch pipelines with stringent SLAs and data quality controls
• Proven experience in time‑series analysis, anomaly detection, and forecasting for operational systems
• Experience deploying containerized services (Docker), Linux fundamentals, and CI/CD
• Proficiency with at least one time‑series/analytical store (ClickHouse, Postgres/TimescaleDB, columnar stores), plus caching/NoSQL where appropriate
• Experience with workflow orchestration (Prefect, Airflow, or Dagster)

• Streaming platforms (Kafka/Redpanda/Pulsar) or equivalent messaging; schema management and idempotent/exactly‑once strategies
• Model serving monitoring (custom FastAPI/Flask services, MLflow, KServe/Seldon); drift detection; shadow/canary rollouts
• Familiarity with observability tooling (OpenTelemetry, Prometheus, Grafana) and alerting best practices (SLOs, MTTR/MTTA)
• Exposure to LLMs and Hugging Face; interest in applying LLMs to ops guidance (RAG over telemetry/runbooks)
• Distributed systems fundamentals; cloud experience (GCP/AWS/Azure) and IaC
• Kubernetes experience or willingness to ramp quickly

Life At Docusign

Our global benefits