python-performance-optimization

Python Performance Optimization

Skill Overview

Analyze and optimize Python code performance using cProfile, memory profilers, and performance best practices.

Suitable Scenarios

1. Performance Bottleneck Troubleshooting

When a Python application runs slowly, has long response times, or shows high CPU usage, use this skill to locate performance bottlenecks in the code. Identify hotspot functions by using a CPU profiler and pinpoint the longest time-consuming code sections.

2. Memory Issue Diagnosis

Resolve problems such as excessive memory usage, memory leaks, or stuttering caused by frequent GC in Python programs. Use memory analysis tools to trace object allocations and identify the source of memory. leaks.

3. Accelerating Data Processing

Optimize large-scale data processing pipelines, batch jobs, and compute-intensive operations. Improve throughput through algorithm optimization, data structure selection, and parallel processing.

Core Capabilities

1. CPU Performance Profiling

Use tools like cProfile to perform CPU-level performance analysis on code, generate function call statistics reports, and identify hotspot functions and call paths. Supports statistical analysis, visualization, and bottleneck identification.

2. Memory Profiling and Optimization

Use a memory profiler to track object creation, destruction, and reference relationships; detect memory leaks and unnecessary memory usage. Provide memory usage reports and optimization recommendations.

3. Performance Optimization Best Practices

Offer practical guidance on performance optimization based on Python language features, including tips on data structure selection, algorithm optimization, I/O optimization, and database query optimization.

Common Questions

How can I troubleshoot when Python code runs too slowly?

Use cProfile for CPU performance profiling as the first step in diagnosing slow code. Run python -m cProfile -o output.prof your_script.py to generate profiling data, then use pstats or visualization tools such as snakeviz to inspect hotspot functions. Focus on functions with the longest cumulative time (cumtime), which are typically the primary targets for optimization.

How do I detect memory leaks in a Python program?

Use tools such as memory_profiler or tracemalloc to track memory usage. After installing memory_profiler, use the @profile decorator to mark functions that need analysis, then run python -m memory_profiler your_script.py to view per-line memory usage changes. Memory leaks usually manifest as object references not being freed; compare memory snapshots before and after execution to locate the source.

What are some best practices for Python performance optimization?

Prefer built-in data structures and functions (e.g., list comprehensions, dictionary lookups) and avoid repeated computations inside loops. Use generators to handle large datasets and reduce memory usage. For CPU-intensive tasks, consider using multiprocessing for parallel processing. Cache function results with functools.lru_cache. For database queries, use ORM features like select_related and prefetch_related to eliminate N+1 query problems.