Notebook (data)

What is a notebook?

A notebook is an interactive work document for data. It is made of cells: some cells contain code, some contain explanation, and some show the result of the previous step as a table, chart, text output, or error message.

That makes a notebook different from a normal script. A script usually runs from top to bottom. A notebook lets you work step by step while you are still learning the data. You filter a table, run the cell, inspect the row count, write down what you learned, and continue.

Jupyter is the best-known notebook environment. A Jupyter notebook is stored as an .ipynb file, which is JSON under the hood and can contain code, Markdown text, metadata, and saved outputs. Open a colleague's notebook and you may see the results from the last run without executing it yourself.

When to use a notebook

• Exploring a new dataset. Check columns, missing values, duplicates, date ranges, odd categories, and example rows before building a pipeline.

• One-off analysis. Answer a specific business question and keep the reasoning next to the code, so someone can review it later.

• Machine learning experiments. Train several model variants, compare metrics, inspect errors, and record why one approach was kept.

• Executable documentation. Explain a dataset or calculation with code that can be rerun instead of screenshots that go stale.

Notebooks are strongest while the question is still changing. They are less ideal when the process is stable, business-critical, and needs formal testing.

Notebooks in Microsoft Fabric

In Microsoft Fabric, a notebook is an item inside a workspace. It runs in the browser, can attach to a lakehouse, and can work with Spark, Python, SQL, Scala, or R depending on the runtime and kernel.

That makes notebooks useful for Fabric data engineering. You can read tables from a lakehouse, transform them with PySpark or Spark SQL, write results back to Delta tables, and then use those tables in Power BI or another Fabric item.

Fabric notebooks can also be scheduled or used as activities inside a data pipeline. That is useful when a piece of logic starts as exploration and later becomes a recurring transformation. Parameters let the same notebook run for different dates, files, or entities.

Databricks, JupyterLab, Google Colab, Azure Machine Learning, and many other tools use the same general notebook pattern, even though the surrounding platform differs.

Notebook, dataflow, script, or pipeline?

Use a dataflow when the transformation is mostly cleaning, filtering, merging, and reshaping that business users can manage through Power Query. Dataflow Gen2 is often the quickest route in Fabric when low-code is enough.

Use a notebook when you need code and visibility into each step. It is a good fit for exploration, custom calculations, machine learning, API experiments, and transformations that are awkward in clicks.

Use a script when the logic is stable and should be tested like software. A plain Python file or Spark job definition is easier to review, lint, test, and version than a notebook full of saved outputs.

Use a pipeline to orchestrate several steps: copy data, run a notebook, refresh a semantic model, send an alert, or branch on success and failure.

Work often moves through these shapes. Explore in a notebook. Keep the reusable parts. Move stable work to a script, dataflow, or pipeline when it becomes operational.

What to watch out for with notebooks

Cell order can lie. You can run cell 12 before cell 3, keep variables from deleted cells, or accidentally rely on memory from an earlier run. Before sharing or scheduling, restart the session and run everything from top to bottom.

Outputs can leak data. An .ipynb file can store result tables, charts, logs, and errors. Clear outputs before sending a notebook around or pushing it to Git if those outputs contain customer data, secrets, or internal paths.

Secrets do not belong in cells. Use a vault, managed identity, environment variable, or platform connection instead of pasting passwords and API keys into code.

Production needs discipline. Scheduled notebooks can run real processes, but they still need ownership, review, monitoring, parameters, failure handling, and version control.

Large work needs the right engine. pandas on one machine is excellent for small and medium data. For large tables, use Spark, SQL, or a warehouse/lakehouse engine that can handle the size.