Overview
Pipelines
Data Quality
Resources
Logs
All primary pipelines running normally. Next scheduled maintenance: Jan 28, 2026 02:00 UTC
Active Pipelines
24
All healthy
Jobs (24h)
1,847
+12.3% vs yesterday
Records Processed
48.2M
Avg 2.01M/hr
Error Rate
0.12%
SLA Target: <0.5%
Avg Latency
342ms
P99: 1.2s
Pipeline Throughput (24h)
Error Distribution
Success Rate Trend
Latency Distribution
Data Volume by Source
Recent Pipeline Executions
Job ID Pipeline Status Records Duration Started Trigger
Primary ETL Pipeline - DAG View
Source Ingest
Running
Validate
Running
Transform
Running
Enrich
Running
Deduplicate
Processing
Load (DWH)
Waiting
Index & Cache
Waiting
All Pipelines
Name Status Schedule Last Run Next Run
etl_main_v3 Running */5 * * * * 2 min ago 3 min
user_events_stream Running realtime streaming -
daily_aggregation Completed 0 2 * * * 5h ago 19h
ml_feature_store Running 0 */2 * * * 45 min ago 1h 15m
clickstream_etl Degraded */10 * * * * 8 min ago 2 min
data_catalog_sync Completed 0 6 * * * 1h ago 23h
payment_reconcile Running */15 * * * * 12 min ago 3 min
gdpr_anonymize Failed 0 3 * * 0 2d ago 5d
Execution History (7 Days)
Dependency Graph
SLA Compliance
Completeness
99.4%
Uniqueness
98.7%
Validity
97.2%
Freshness
4.2min
Avg data lag across all sources
Data Quality Score Trend (30d)
Quality Dimensions
Anomaly Detection
Column Health Matrix
Validation Rules
Rule Table Column Type Pass Rate Last Check Status
not_null_check orders user_id Completeness 100% 2 min ago Pass
email_format users email Validity 99.8% 5 min ago Pass
unique_txn_id transactions txn_id Uniqueness 99.97% 3 min ago Pass
amount_range transactions amount Validity 96.3% 3 min ago Warn
ref_integrity order_items product_id Consistency 99.9% 8 min ago Pass
freshness_check events created_at Freshness 84.1% 1 min ago Fail
Worker Node 1
Healthy
CPU
67%8 cores
Memory
54%32 GB
Disk I/O
38%240 MB/s
Worker Node 2
Healthy
CPU
82%16 cores
Memory
71%64 GB
Disk I/O
62%480 MB/s
Worker Node 3
High Load
CPU
91%16 cores
Memory
86%64 GB
Disk I/O
74%560 MB/s
Worker Node 4
Healthy
CPU
45%8 cores
Memory
39%32 GB
Disk I/O
22%170 MB/s
CPU Usage Timeline (All Workers)
Memory Usage Timeline
Network I/O
Disk Usage
Queue Depth
Filter:
Showing 0 entries
pipeline-ops-logs Live
Log Distribution
Log Rate (per min)
Active Incidents & Alerts
Severity Source Message Triggered Acknowledged
CRITICAL gdpr_anonymize Pipeline failed: timeout on anonymization batch 47/120 2d ago Yes
WARNING worker-3 CPU usage exceeded 90% threshold for 15 minutes 23 min ago No
WARNING clickstream_etl Data freshness degraded: avg lag increased to 8.4min 1h ago Yes
INFO etl_main_v3 Schema drift detected: new column "promo_code" in orders table 3h ago Yes
INFO auto-scaler Scaling down worker-5: idle for 30 minutes 4h ago Yes