## Overview **Error Rate** measures the percentage of requests that result in errors (4xx or 5xx HTTP status codes, exceptions, or application-level errors). It's a critical reliability metric that indicates system health, impacts user experience, and serves as an early warning system for incidents. ## Why It Matters - **User experience**: Errors frustrate users and cause churn - **Revenue impact**: Errors in checkout/payment flows lose money - **Early warning**: Spikes indicate incidents before they escalate - **Quality indicator**: Reflects code quality and testing effectiveness - **SLA compliance**: Error rate thresholds in service agreements - **Debugging**: Helps identify and prioritize issues ## Types of Errors ### By HTTP Status Code **Client Errors (4xx):** - `400 Bad Request`: Invalid request from client - `401 Unauthorized`: Authentication required - `403 Forbidden`: Insufficient permissions - `404 Not Found`: Resource doesn't exist - `429 Too Many Requests`: Rate limit exceeded **Server Errors (5xx):** - `500 Internal Server Error`: Generic server error - `502 Bad Gateway`: Upstream service failure - `503 Service Unavailable`: Server overloaded/maintenance - `504 Gateway Timeout`: Upstream service timeout ### By Impact **User-Facing Errors:** - Failed page loads - Failed form submissions - Broken features - Payment failures **Background Errors:** - Cron job failures - Queue processing errors - Webhook delivery failures - Scheduled task errors ## How to Measure ### Calculation ``` Error Rate = (Error Count ÷ Total Requests) × 100 Example (1-hour window): Total requests: 1,000,000 Failed requests: 1,250 Error Rate: 0.125% ``` ### What Counts as an Error? **Include:** - 5xx server errors (all) - 4xx client errors (except 404s, sometimes) - Unhandled exceptions - Timeouts - Failed database queries **Exclude (sometimes):** - 404 Not Found (may be user error, not system error) - 401 Unauthorized (expected for unauthenticated requests) - 429 Rate Limited (intentional throttling) ### Segmentation Track error rate by: - **Endpoint**: Which APIs are failing? - **Status code**: What types of errors? - **Geography**: Regional issues? - **Client**: Mobile vs web vs API - **User**: Specific accounts having issues? ## Recommended Visualizations ### 1. Error Rate Trend **Best for**: Real-time monitoring ### 2. Error Rate Gauge **Best for**: Current status ### 3. Error Types Breakdown **Best for**: Identifying error patterns ## Target Ranges ### By Service Type | Service Type | Target | Acceptable | Critical | |--------------|--------|-----------|----------| | **Critical APIs** | < 0.01% | < 0.1% | > 1% | | **Standard APIs** | < 0.1% | < 0.5% | > 2% | | **Background jobs** | < 1% | < 5% | > 10% | | **Webhooks** | < 5% | < 10% | > 20% | ### By Error Type | Error Type | Target | Notes | |-----------|--------|-------| | **5xx errors** | < 0.01% | Server-side, completely unacceptable | | **4xx errors** | < 1% | Client-side, some expected | | **Timeouts** | < 0.1% | Infrastructure issue | | **Exceptions** | 0% | Code bugs, should not happen | ### Industry Standards **E-commerce:** - Checkout: < 0.01% (every error = lost revenue) - Product pages: < 0.1% - Search: < 0.5% **SaaS:** - Core features: < 0.1% - Auth/login: < 0.01% - API: < 0.5% **FinTech:** - Payments: < 0.001% (three zeros!) - Trading: < 0.01% - Account management: < 0.1% ## How to Improve ### 1. Implement Proper Error Handling ```python # Bad: No error handling def process_payment(amount): charge = stripe.charge.create(amount=amount) return charge # Good: Comprehensive error handling def process_payment(amount): try: charge = stripe.charge.create(amount=amount) return {'success': True, 'charge': charge} except stripe.CardError as e: # Card was declined log.warning(f'Card declined: {e}') return {'success': False, 'error': 'card_declined'} except stripe.APIConnectionError as e: # Network error log.error(f'Stripe connection error: {e}') return {'success': False, 'error': 'service_unavailable'} except Exception as e: # Unknown error log.error(f'Payment failed: {e}', exc_info=True) return {'success': False, 'error': 'internal_error'} ``` ### 2. Input Validation ```python from pydantic import BaseModel, validator class CreateUserRequest(BaseModel): email: str age: int username: str @validator('email') def validate_email(cls, v): if '@' not in v: raise ValueError('Invalid email') return v @validator('age') def validate_age(cls, v): if v < 0 or v > 150: raise ValueError('Invalid age') return v # Returns 400 Bad Request before hitting database # Prevents 500 errors from invalid data ``` ### 3. Retry Logic with Exponential Backoff ```python import time from functools import wraps def retry(max_attempts=3, backoff=2): def decorator(func): @wraps(func) def wrapper(*args, **kwargs): for attempt in range(max_attempts): try: return func(*args, **kwargs) except Exception as e: if attempt == max_attempts - 1: raise wait_time = backoff ** attempt time.sleep(wait_time) return None return wrapper return decorator @retry(max_attempts=3) def call_external_api(): return requests.get('https://api.example.com') ``` ### 4. Circuit Breakers ```python from circuitbreaker import circuit @circuit(failure_threshold=5, recovery_timeout=60) def call_payment_api(): response = requests.post('https://payments.example.com') if response.status_code >= 500: raise Exception('Service error') return response # After 5 failures, circuit opens # Returns error immediately for 60 seconds # Prevents cascading failures ``` ### 5. Graceful Degradation ```python def get_user_profile(user_id): try: profile = cache.get(f'profile:{user_id}') if profile: return profile except RedisError: # Cache failed, continue anyway log.warning('Cache unavailable, fetching from DB') try: profile = db.query(User).get(user_id) return profile except DatabaseError as e: # Database failed, return minimal data log.error(f'DB error: {e}') return {'id': user_id, 'name': 'Unknown', 'status': 'error'} # Service stays up even if cache and DB fail ``` ### 6. Monitoring and Alerting ```python # DataDog metric statsd.increment('api.request.error', tags=[ f'endpoint:{request.path}', f'status:{response.status_code}', f'error_type:{error.__class__.__name__}' ]) # Alert when error rate > 1% for 5 minutes ``` ### 7. Error Budgets ``` Error Budget = (1 - SLA) × Total Requests Example: SLA: 99.9% uptime Error Budget: 0.1% For 1M requests/day: Error Budget = 1,000 errors/day If you exceed budget: Freeze feature releases, focus on reliability ``` ## Common Pitfalls ### ❌ Alerting on Every Error **Problem**: Alert fatigue, important errors missed **Solution**: Alert on error rate > threshold, not individual errors ### ❌ Not Categorizing Errors **Problem**: Client errors (4xx) mixed with server errors (5xx) **Solution**: Track separately, different targets for each ### ❌ Ignoring Context **Problem**: 1% error rate on low-traffic endpoint vs high-traffic **Solution**: Consider absolute error count and user impact ### ❌ No Error Tracking for Background Jobs **Problem**: Silent failures go unnoticed **Solution**: Track and alert on background job failures ### ❌ Treating All 4xx as Errors **Problem**: 404s and 401s inflate error rate **Solution**: Exclude expected 4xx codes from alerts ## Implementation Guide ### Week 1: Instrumentation ```javascript // Express.js error tracking app.use((err, req, res, next) => { // Log error console.error(err.stack); // Send to error tracking (Sentry) Sentry.captureException(err); // Track metric statsd.increment('api.error', { endpoint: req.path, status: err.status || 500 }); // Return error response res.status(err.status || 500).json({ error: process.env.NODE_ENV === 'production' ? 'Internal server error' : err.message }); }); ``` ### Week 2: Dashboards ```yaml # DataDog monitor name: "High Error Rate" type: metric alert query: "sum(last_5m):sum:api.error{*}.as_count() / sum:api.request{*}.as_count() > 0.01" message: | Error rate exceeds 1% for 5 minutes Current rate: {{value}}% notify: - "@slack-eng" - "@pagerduty" ``` ### Week 3: Error Investigation Process 1. **Alert fires** → Error rate > 1% 2. **Check dashboard** → Which endpoints? 3. **Check logs** → What errors? 4. **Check recent changes** → New deployment? 5. **Mitigate** → Rollback or hotfix 6. **Post-mortem** → Root cause, prevention ### Week 4: Continuous Improvement - Review top errors weekly - Fix highest-frequency errors - Add tests to prevent regressions - Update error handling patterns ## Dashboard Example ### Operations View ``` ┌──────────────────────────────────────────────┐ │ Error Rate │ │ Current: 0.08% ✓ Good │ │ ████████████████████████████░░ Under 0.1% │ │ │ │ Last Hour: │ │ • Total Requests: 1,250,000 │ │ • Errors: 1,000 │ │ • Error Rate: 0.08% │ │ │ │ Top Errors: │ │ • 500 /api/reports 450 (45%) │ │ • 503 /api/search 280 (28%) │ │ • 504 /api/webhooks 150 (15%) │ │ • 400 /api/users 120 (12%) │ └──────────────────────────────────────────────┘ ``` ### Detailed Breakdown ``` Error Distribution (Last 24 Hours) ──────────────────────────────────────────── Status Count % Endpoint ──────────────────────────────────────────── 500 8,450 (42%) /api/reports 503 5,220 (26%) /api/search 504 2,890 (14%) /api/webhooks 400 2,150 (11%) /api/users 502 1,420 (7%) /api/payments ──────────────────────────────────────────── Total 20,130 (0.11%) Trend: ↑ +15% vs. yesterday Alert: ⚠️ /api/reports needs investigation ``` ## Related Metrics - **API Response Time**: Slow requests often become errors - **System Uptime**: High error rate indicates downtime - **Change Failure Rate**: Deployments causing errors - **MTTR**: How fast you fix errors - **User Satisfaction**: Errors correlate with NPS ## Tools & Integrations ### Error Tracking - **Sentry**: Real-time error tracking and alerting - **Rollbar**: Error monitoring for all platforms - **Bugsnag**: Error monitoring with release tracking - **Raygun**: Error tracking + RUM ### APM with Error Tracking - **DataDog**: Full-stack monitoring - **New Relic**: Application performance - **Dynatrace**: AI-powered monitoring - **AppDynamics**: Business transaction monitoring ### Log Aggregation - **ELK Stack**: Elasticsearch + Logstash + Kibana - **Splunk**: Enterprise log management - **Sumo Logic**: Cloud-native log analytics ## Questions to Ask ### For Operations - What's causing the most errors? - Are errors concentrated in specific endpoints? - Is error rate trending up or down? - Do errors correlate with deployments? ### For Engineering - Are we handling errors gracefully? - Do we have proper input validation? - Are we monitoring all error types? - Do we retry transient failures? ### For Leadership - Is error rate impacting customer satisfaction? - Do we need to invest in reliability? - Are we meeting our SLAs? - What's the business impact of errors? ## Success Stories ### SaaS Platform - **Before**: 2.5% error rate, customer complaints - **After**: 0.05% error rate, NPS +30 points - **Changes**: - Added comprehensive input validation - Implemented circuit breakers for external services - Proper error handling in all endpoints - Automated error alerting and response - **Impact**: 98% error reduction, customer satisfaction dramatically improved ### E-commerce Site - **Before**: 0.8% error rate in checkout, $100K/month lost - **After**: 0.02% error rate, revenue recovered - **Changes**: - Retry logic for payment API - Better error messages to users - Fallback payment methods - Real-time error monitoring - **Impact**: 97.5% error reduction, $96K monthly revenue recovered ## Conclusion Error Rate is a critical reliability metric that directly impacts user experience and revenue. Target < 0.1% for production APIs, < 0.01% for critical services. Reduce errors through proper error handling, input validation, retry logic, circuit breakers, and graceful degradation. Monitor error rate in real-time, alert on spikes, and investigate root causes promptly. Remember: every error is a potential lost customer—invest in reliability. **Quick Start:** 1. Instrument error tracking in all endpoints 2. Set up error rate dashboard 3. Create alerts for > 1% error rate 4. Categorize errors by type and endpoint 5. Establish error investigation process 6. Fix top errors systematically

Alerting

Featured

Error Rate

All Alerting

Want to contribute?