The Apache Spark community has improved support for Python to such a great degree over the past few years that Python is now a “first-class” language, and no longer a “clunky” add-on as it once was, ...

Likewise, with Python, Databricks supports Python 2 and Python 3 clusters, but runs Python packages in a virtual environment.

With Apache Spark Declarative Pipelines, engineers describe what their pipeline should do using SQL or Python, and Apache Spark handles the execution.