pandas

What is pandas?

pandas is a Python library for working with tabular data. It gives you two core structures: Series, which is a labelled one-dimensional array, and DataFrame, which is a table with rows, columns, labels, and data types.

With pandas, you can read CSV, Excel, JSON, Parquet, SQL tables, and many other formats; clean and reshape the data; join it to other tables; group and aggregate it; then write the result back out.

The easiest mental model is Excel with a programming language around it. Instead of changing one cell at a time, you write an expression that applies to a whole column, group, or table. The steps are reproducible and can live in a notebook, script, or pipeline.

Why pandas became so common

It fits the analyst workflow. Read data, inspect it, clean it, calculate, plot, export. pandas covers that loop in one library.

It works with the Python ecosystem. A DataFrame can feed NumPy, scikit-learn, matplotlib, Plotly, SQLAlchemy, statsmodels, and many other tools.

It is easy to learn from examples. Nearly every data task has a pandas example somewhere, from date parsing to pivots to window calculations.

It is excellent in notebooks. pandas displays tables directly, which makes it a natural companion to Jupyter, Fabric notebooks, Databricks notebooks, and quick exploratory analysis.

It is good enough for a lot of real work. If the dataset fits comfortably in memory, pandas is often the fastest way to answer the question.

pandas in Fabric and Databricks

Microsoft Fabric notebooks include Python support and can use pandas for smaller data work. You can read from lakehouse files or tables, inspect a sample, prepare a small dataset, and write results that other Fabric items can use.

Databricks notebooks also support pandas, but the usual split is clear: pandas for work that fits on the driver machine, Spark DataFrames for distributed processing. Databricks and Spark also expose pandas-like APIs for some distributed workloads, although the behaviour is not always identical to local pandas.

The practical rule: start with pandas when the data fits on one machine and the work is exploratory. Move to Spark, SQL, Polars, or DuckDB when size, speed, or production needs push beyond that.

pandas, Polars, DuckDB, and Spark

pandas. Best for familiar Python analysis on small and medium data, especially in notebooks.

Polars. A DataFrame library written in Rust, designed for speed, lazy execution, and multi-core processing. It is often a better fit for larger local datasets and analytical transformations.

DuckDB. An embedded analytical database. You write SQL and query CSV, Parquet, pandas, or Polars data directly. It is a strong choice when the person doing the work thinks in SQL or when the data lives in files.

Apache Spark. A distributed processing engine for data that is too large for one machine or needs cluster-scale transformations. It has more overhead, so it is not automatically better for small data.

These tools can work together. A common pattern is DuckDB or Spark reading Parquet, pandas handling a sample or final result, and Python controlling the workflow.

Common pandas tasks

• Load CSV, Excel, JSON, Parquet, or SQL data into a DataFrame.

• Filter rows, select columns, rename fields, and fix data types.

• Join customers, orders, products, and lookup tables.

• Group by date, region, product, or customer segment and calculate totals.

• Pivot data into a report-friendly shape.

• Prepare features for a machine learning model.

• Write cleaned data back to files, tables, or another process.

What to watch out for with pandas

Everything lives in memory. A file that looks modest on disk can become much larger in RAM. If your laptop starts swapping, use Polars, DuckDB, Spark, or a database engine.

Chained assignment is confusing. The famous SettingWithCopyWarning appears when pandas is unsure whether you are changing a view or a copy. Use explicit .loc or .copy() when assigning values.

Data types can surprise you. Missing integers may become floats, mixed dates may become objects, and strings may stay larger than expected. Check .dtypes after loading and set types explicitly for important columns.

Notebook output can hide state. pandas is often used in notebooks where cells may run out of order. Restart and run all before trusting results.

Production needs version control. Pin pandas versions for pipelines. Major releases can change defaults or remove deprecated behaviour.