diff --git a/apps/metrics/backend/services/starrocks_metrics_service.py b/apps/metrics/backend/services/starrocks_metrics_service.py index 329815b..648dd74 100644 --- a/apps/metrics/backend/services/starrocks_metrics_service.py +++ b/apps/metrics/backend/services/starrocks_metrics_service.py @@ -25,7 +25,7 @@ class StarRocksMetricsService: updated_at FROM dws_daily_registered_users WHERE date_id >= %s - AND date_id <= %s + AND date_id < %s AND product_id = %s ORDER BY date_id ASC """, diff --git a/apps/metrics/docs/design.md b/apps/metrics/docs/design.md index 796b5ae..ea9fa46 100644 --- a/apps/metrics/docs/design.md +++ b/apps/metrics/docs/design.md @@ -8,6 +8,159 @@ We support two ways to query metrics: We can implement StarRocks Metric queries similar to Prometheus Metric queries. The only difference is replacing PromQL with SQL and querying through StarRocks API. +## 2.1.Metrics Config +Currently, metrics are configured in code through the `StarRocksMetricsService.METRIC_SQL_MAP` dictionary. In the future, they will be configured through database or other methods. +Organization structure: Product ID -> Metric Name -> SQL Query +```python +METRIC_SQL_MAP: Dict[str, Dict[str, str]] = { + "freeleaps": { + "daily_registered_users": """ + SELECT + date_id, + product_id, + registered_cnt, + updated_at + FROM dws_daily_registered_users + WHERE date_id >= %s + AND date_id <= %s + AND product_id = %s + ORDER BY date_id ASC + """, + }, + "magicleaps": { + # Future metrics can be added here + } +} +``` + +## 2.2.API Design + +### 2.2.1.Query Metrics by Product ID + +API: `/api/metrics/starrocks/product/{product_id}/available-metrics` + +Method: GET +Request: +``` +product_id=freeleaps +``` +Response: + +```json +{ + "product_id": "freeleaps", + "available_metrics": [ + "daily_registered_users" + ], + "total_count": 1, + "description": "List of StarRocks-backed metrics for product 'freeleaps'" +} +``` + +### 2.2.2.Query Metric Info +API: `/api/metrics/starrocks/product/{product_id}/metric/{metric_name}/info` + +Method: GET +Request: +``` +product_id=freeleaps +metric_name=daily_registered_users +``` +Response: + +```json +{ + "metric_info": { + "product_id": "freeleaps", + "metric_name": "daily_registered_users", + "sql_query": "SELECT date_id, product_id, registered_cnt, updated_at FROM dws_daily_registered_users WHERE date_id >= %s AND date_id <= %s AND product_id = %s ORDER BY date_id ASC", + "description": "Daily registered users count from StarRocks table dws_daily_registered_users" + }, + "description": "Information about StarRocks metric 'daily_registered_users' in product 'freeleaps'" +} +``` + +### 2.2.3.Query Metric Data +API: `/api/metrics/starrocks/metrics_query` + +Method: POST +Request: +```json +{ + "product_id": "freeleaps", + "metric_name": "daily_registered_users", + "start_date": "2024-09-10", + "end_date": "2024-09-20" +} +``` +Response: + +```json +{ + "metric_name": "daily_registered_users", + "data_points": [ + { + "date": "2024-09-10", + "value": 45, + "labels": { + "product_id": "freeleaps", + "metric_type": "daily_registered_users" + } + }, + { + "date": "2024-09-11", + "value": 52, + "labels": { + "product_id": "freeleaps", + "metric_type": "daily_registered_users" + } + }, + { + "date": "2024-09-12", + "value": 38, + "labels": { + "product_id": "freeleaps", + "metric_type": "daily_registered_users" + } + }, + ... + { + "date": "2024-09-19", + "value": 67, + "labels": { + "product_id": "freeleaps", + "metric_type": "daily_registered_users" + } + } + ], + "total_points": 10, + "time_range": { + "start": "2024-09-10", + "end": "2024-09-19" + } +} +``` + +## 2.3.Technical Implementation + +### 2.3.1.StarRocks Client +- Uses PyMySQL to connect to StarRocks database +- Supports parameterized queries for security +- Automatic connection management with context manager +- Error handling and logging + +### 2.3.2.Data Format +- Date format: `YYYY-MM-DD` +- Values are returned as integers or floats +- Labels include product_id and metric_type for debugging +- Results are sorted by date in ascending order + +### 2.3.3.Validation +- Date range validation (start_date < end_date) +- Maximum date range limit (1 year) +- Product ID and metric name validation against available mappings +- Input format validation for date strings + # 3.Prometheus Metric ## 3.1.Metrics Config