Logging Debugging

When your codebase is tiny, print() and random try/except blocks "work". Once you have APIs, background workers, cron jobs, queues, and multiple servers, that approach falls apart. Learn professional-grade debugging + logging + error-handling systems used by real engineering teams.

Learn Python, JavaScript, Java and more with free interactive lessons, real projects and AI-powered help. Beginner-friendly.

Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

This lesson takes you from basic debugging → production-grade logging systems used by companies handling thousands of users, multiple environments, and distributed services.

🔥 1. Why Logging Beats print() in Real Systems

When you have thousands of users, multiple environments (dev/staging/prod), background jobs, schedulers, and queues — print() falls apart.

Rule: print() is only for quick experiments. Real services use logging everywhere.

⚙️ 2. Basic Logging Setup (Per-Module Loggers)

You should never use the root logger directly in big projects. Instead, use one logger per module:

Then in your entry point (e.g. main.py or app.py):

This gives you timestamp, log level, logger name (payments.service, users.api), and message. You can raise log level in dev: DEBUG, staging: INFO, prod: WARNING.

🧱 3. Logging Levels: A Contract for Your Team

Use logging levels consistently across your team:

When everyone uses the same level rules, you can alert only on ERROR/CRITICAL, filter out noisy DEBUG in production, and quickly see timeline of events with INFO logs.

🧩 4. Structured Logging (So Logs Are Actually Searchable)

For small scripts, string messages are fine. At scale, you need structured logs so tools like Datadog/Loki/ELK/CloudWatch can filter and aggregate.

Now your logging backend can filter by user_id, count logins from each IP, and group by event.

🧲 5. Handlers: Sending Logs to Multiple Destinations

A handler decides where logs go. Common ones:

Now dev uses console, ops can inspect app.log, and files don't grow forever. In a SaaS/microservice setup, you'd usually log to stdout and let Docker/Kubernetes send logs to a central system.

🧠 6. Error Handling Strategy (Don't Just "try/except Everything")

Good pattern: Catch specific exceptions, log with traceback, decide whether to handle or re-raise

Key ideas: Use domain-level exceptions (PaymentError), wrap low-level exceptions, pass exc_info=True to capture traceback, use raise ... from e to keep the error chain.

🧨 7. Logging Tracebacks Correctly

This logs message, full traceback, and error type.

Both approaches are valid. logger.exception() is just a shortcut for logger.error(..., exc_info=True) inside an except block.

🪤 8. Global Exception Hooks (Last-Resort Safety Net)

For CLI/worker processes, you can register a global handler:

Now any uncaught error in the main thread is logged at CRITICAL. You still let the process crash (which is often correct in prod). In frameworks: Django has middleware for logging unhandled errors, FastAPI lets you add global exception handlers, Celery workers log errors from tasks.

🧪 9. Debugging Strategy: Logs + Debugger + Assertions

✅ Use assertions in dev: assert total >= 0, "Total should never be negative"

In production, you can turn off certain assertions or keep only the critical ones.

🎯 10. Principles for "At Scale" Error Handling

Always log unexpected exceptions. Prefer fail loud and fast to silent corruption.

HTTP layer (FastAPI/Django view), Message queue consumer, CLI/worker entry point

DomainError → ValidationError, PaymentError, ExternalServiceError. This makes handling & logging more deliberate.

4. Keep logs human-readable AND machine-parseable

Too much DEBUG in prod = noisy. Too few logs in prod = blind.

Part 1 covered local logging and error handling. Now we move into real production engineering — the systems used by FastAPI/Django backends, microservices, SaaS platforms, and distributed workers.

🔥 1. Centralised Log Aggregation (The Real World Standard)

As soon as you have multiple servers, background workers, containers, or functions-as-a-service — you cannot inspect logs locally anymore. Real systems send logs to a central place.

✔ ELK Stack (Elasticsearch + Logstash + Kibana) — Most customizable, open-source, works at huge scale

✔ Loki + Grafana — Insanely fast, cheap, streams logs from Docker/Kubernetes

✔ AWS CloudWatch / GCP Logging / Azure Monitor — Great if you already host on those platforms

✔ Datadog / Sentry / NewRelic — Expensive, but world-class dashboards + alerts

⚙️ 2. Logging to STDOUT in Containers (Best Practice)

In Docker/Kubernetes, you never write log files inside the container. You output logs to STDOUT:

Kubernetes will automatically: ✔ capture your stdout, ✔ send it to your log system, ✔ attach metadata (pod, namespace, service). This makes logs fully searchable across the entire cluster.

🧠 3. The Importance of Request IDs / Correlation IDs

Imagine debugging: User logs in → Their request hits API A → API A calls API B → API B queries the database → A background worker processes a message → Something fails.

Without correlation IDs, logs look like a mess. Solution: Generate a unique ID per user request.

Pass the ID through HTTP headers, background jobs, microservice calls, and log contexts. Now you can filter your log system for request_id = "1f0cd133-f9ab-4bd9-a6cd-92d3a002a415" and see EVERYTHING that happened.

This alone can save hours per week in debugging.

🧩 4. Logging Context Automatically (ContextVars)

Python provides a way to attach values (like request IDs) to all logs inside async or threaded code.

Now every log line automatically includes request_id, user_id (if added), job_id, trace_id. This is essential in async web apps like FastAPI.

🚀 5. Distributed Tracing (How Big Systems Debug)

Used by Uber, Netflix, Stripe, Google-scale systems. Distributed tracing tracks: "This request flowed through: API → Worker → DB → Cache → Queue → Another Worker"

Tools: OpenTelemetry (industry standard), Jaeger, Zipkin, Datadog APM

Now logs + traces appear linked. This gives timings for each step, slow points, bottlenecks, failed spans, and dependency maps. This is how modern SaaS teams debug performance problems.

🧵 6. Logging in Async Python (Trickier Than It Looks)

Async apps (FastAPI, aiohttp) handle hundreds/thousands of concurrent tasks. Debugging them needs extra care.

Problem 1: Interleaved logs — Two tasks printing logs at the same time → scrambled output

Practice quiz

• Browse all free courses
• Cheat sheets & quizzes
• Coding dictionary