cheat sheets / big data
pandas, Spark & SQL Cheat Sheets for Data Work
Three references for the tools data engineers actually live in, plus the two ideas that separate slow pipelines from fast ones: why loops kill pandas, and why shuffles dominate Spark.
Working with data at scale means living in a few core tools: pandas for analysis that fits in memory, Spark when the data outgrows one machine, and SQL as the common language across both. The references below are the official or best-maintained sheet for each. The more valuable thing to carry alongside them is a sense of what makes each tool slow, because in data work the gap between a naive implementation and a good one is often not ten percent but two orders of magnitude.
01 · THE REFERENCES
Wrangle & scale
From a single DataFrame to a distributed cluster. The pandas sheet is the project's own one-page PDF, which is genuinely worth printing; the Spark guide is the official documentation for its DataFrame and SQL APIs.
- pandas Cheat Sheet (official PDF) ↗The pandas project's own one-page reference — filtering, grouping, reshaping, and merging DataFrames. Print it out.pandas.pydata.org
- Apache Spark SQL Guide (official) ↗The authoritative guide to Spark's DataFrame and SQL APIs — the workhorse for large-scale data processing.spark.apache.org
- SQL Cheat Sheet ↗The common query language under every data warehouse and analytics engine — JOINs, aggregates, and window basics.quickref.me
02 · PANDAS
Think in columns, not rows
The most common performance mistake in pandas is writing a Python loop over rows. It feels natural if you come from general programming, and it is dramatically slower than the alternative, because each iteration pays Python interpreter overhead on a single value. pandas operations are backed by NumPy arrays and run their work in compiled code across a whole column at once. Rewriting a row loop as a vectorized column operation routinely turns minutes into seconds.
The rule that follows: whenever you find yourself iterating, look for the column-wise expression that does the same job. Filtering with a boolean mask, arithmetic across whole columns, grouping and aggregating, and merging are all designed for this. Reaching for a loop usually means the tool is being used against its grain.
Views, copies, and the warning everybody ignores
pandas sometimes returns a view onto existing data and sometimes an independent copy, and it is not always obvious which. Chained indexing, where you select twice in sequence and then assign, is where this bites: the assignment may land on a temporary object and silently fail to change anything. This is what the well-known warning about setting a value on a copy is trying to tell you. Selecting explicitly by label or position in a single step avoids the ambiguity entirely.
Watch the data types
A column of numbers read from a messy file can quietly come back as generic objects rather than a numeric type, which costs both memory and speed and produces confusing results in comparisons. Checking types after loading is a two-second habit that prevents a lot of downstream mystery.
03 · SPARK
Laziness is the point, shuffles are the cost
Spark separates operations into transformations, which describe work, and actions, which trigger it. Transformations build a plan and execute nothing. Only when an action asks for a result does Spark optimize the accumulated plan and run it. This surprises people who add a transformation, see it return instantly, and conclude the cluster is fast. It has not done anything yet.
The practical consequence is that where you place actions determines performance. Repeatedly triggering computation on the same intermediate result recomputes it from the source each time unless you explicitly cache it. Conversely, caching everything wastes memory on data used once.
The other dominant cost is the shuffle: any operation that needs to move rows between machines so that related records end up together. Grouping, joining, and repartitioning all shuffle. Shuffles cross the network and write to disk, so they are orders of magnitude more expensive than work done locally on a partition. Most Spark tuning is some version of shuffling less, shuffling smaller data, or filtering before the shuffle rather than after it.
04 · CHOOSING
Which tool for which size
A distributed cluster has real overhead: scheduling, serialization, and network traffic that a single process never pays. For data that fits comfortably in memory on one machine, pandas will usually beat Spark outright, and it is far easier to debug. Reaching for a cluster because the word "big" appeared in the requirements is a common and expensive mistake.
Spark earns its overhead when the data genuinely exceeds one machine, when the work is already sitting in a distributed store, or when you need the fault tolerance of a cluster for long jobs. SQL sits underneath both, and it is worth knowing well regardless: the same query concepts apply whether the engine is a database, a warehouse, or Spark's SQL layer, and window functions in particular replace a great deal of awkward procedural code.
05 · FAQ
Frequently asked questions
Are these data engineering cheat sheets free?
Yes. The pandas one-page PDF is published by the pandas project, the Spark SQL guide is Apache's official documentation, and the SQL reference is a free page on QuickRef. None require an account.
Why is my pandas code so slow?
Most often because it loops over rows in Python. Each iteration pays interpreter overhead on a single value, while column-wise operations run in compiled code across the whole array at once. Rewriting a row loop as a vectorized expression frequently turns minutes into seconds.
Do I need Spark, or is pandas enough?
If the data fits comfortably in memory on one machine, pandas is usually faster and much easier to debug, because a cluster adds scheduling, serialization, and network overhead. Spark earns that overhead when the data genuinely exceeds a single machine or already lives in a distributed store.
What is a shuffle in Spark and why does it matter?
A shuffle is any operation that moves rows between machines so related records end up together, which grouping, joining, and repartitioning all require. Shuffles cross the network and write to disk, making them far more expensive than work done locally, so reducing data before a shuffle is the main tuning lever.
Do I still need SQL if I know pandas and Spark?
Yes. SQL is the common language across databases, warehouses, and Spark's own SQL layer, so it transfers everywhere. It is also the interface most analytics tools and teams expect, which makes it the most portable skill of the three.